TASK DEFINITION: In this task, you are given a part of an article. Your task is to generate headline (title) for this text. Preferred headlines are under fifteen words.
PROBLEM: A famous biologically inspired hierarchical model firstly proposed by Riesenhuber and Poggio has been successfully applied to multiple visual recognition tasks. The model is able to achieve a set of positionand scale-tolerant recognition, which is a central problem in pattern recognition. In this paper, based on some other biological experimental results, we introduce the Memory and Association Mechanisms into the above biologically inspired model. The main motivations of the work are (a) to mimic the active memory and association mechanism and add the ’top down’ adjustment to the above biologically inspired hierarchical model and (b) to build up an algorithm which can save the space and keep a good recognition performance. More details of the work could be explained as follows: (1) In objects memorizing process: Our proposed model mimics some characteristics of human’s memory mechanism as follows: (a) In our model, one object is memorized by semantic attributes and special image patches (corresponding to episodic memory). The semantic attributes describe each part of the object with clear physical meaning, for example, if eyes and mouths of faces are ’big’ or ’small’ and so on. One special patch is selected if the value of the corresponding semantic feature is far from average one. The patch should be the most prominent part of the object. (b) In our model, different features (semantic attributes and special patches) of one object are stored in distributed places and the common feature of different objects is saved aggregately, which can learn to classify the difference of similar features of different objects. The similarity thresholds to each object can be learnt when new objects are learnt. (2) In object recognition process: In biological process, the associated recognition including familiarity discrimination and recollective matching. In our proposed model, firstly mimicking familiarity discrimination (’knowing’ though the episode), we compare the special patches of candidates with that of saved objects using above mentioned biologically inspired hierarchical model, where the candidates and saved objects have the same prominent semantic features. Then mimicking recollective matching, the comparison results of special patches are combined with semantic feature comparison. The new model is also applied to object recognition processes. The primary experimental results show that our method is efficient with much less memory requirement.

SOLUTION: Introducing Memory and Association Mechanism Into a Biologically Inspired Visual Model

PROBLEM: We start with an overview of a class of submodular functions called SCMMs (sums of concave composed with non-negative modular functions plus a final arbitrary modular). We then define a new class of submodular functions we call deep submodular functions or DSFs. We show that DSFs are a flexible parametric family of submodular functions that share many of the properties and advantages of deep neural networks (DNNs), including many-layered hierarchical topologies, representation learning, distributed representations, opportunities and strategies for training, and suitability to GPU-based matrix/vector computing. DSFs can be motivated by considering a hierarchy of descriptive concepts over ground elements and where one wishes to allow submodular interaction throughout this hierarchy. In machine learning and data science applications, where there is often either a natural or an automatically learnt hierarchy of concepts over data, DSFs therefore naturally apply. Results in this paper show that DSFs constitute a strictly larger class of submodular functions than SCMMs, thus justifying their mathematical and practical utility. Moreover, we show that, for any integer k > 0, there are k-layer DSFs that cannot be represented by a k′-layer DSF for any k′ < k. This implies that, like DNNs, there is a utility to depth, but unlike DNNs (which can be universally approximated by shallow networks), the family of DSFs strictly increase with depth. Despite this property, however, we show that DSFs, even with arbitrarily large k, do not comprise all submodular functions. We show this using a technique that “backpropagates” certain requirements if it was the case that DSFs comprised all submodular functions. In offering the above results, we also define the notion of an antitone superdifferential of a concave function and show how this relates to submodular functions (in general), DSFs (in particular), negative second-order partial derivatives, continuous submodularity, and concave extensions. To further motivate our analysis, we provide various special case results from matroid theory, comparing DSFs with forms of matroid rank, in particular the laminar matroid. Lastly, we discuss strategies to learn DSFs, and define the classes of deep supermodular functions, deep difference of submodular functions, and deep multivariate submodular functions, and discuss where these can be useful in applications.

SOLUTION: Deep Submodular Functions

PROBLEM: We describe Swapout, a new stochastic training method, that outperforms ResNets of identical network structure yielding impressive results on CIFAR-10 and CIFAR100. Swapout samples from a rich set of architectures including dropout [17], stochastic depth [6] and residual architectures [4, 5] as special cases. When viewed as a regularization method swapout not only inhibits co-adaptation of units in a layer, similar to dropout, but also across network layers. We conjecture that swapout achieves strong regularization by implicitly tying the parameters across layers. When viewed as an ensemble training method, it samples a much richer set of architectures than existing methods such as dropout or stochastic depth. We propose a parameterization that reveals connections to exiting architectures and suggests a much richer set of architectures to be explored. We show that our formulation suggests an efficient training method and validate our conclusions on CIFAR-10 and CIFAR-100 matching state of the art accuracy. Remarkably, our 32 layer wider model performs similar to a 1001 layer ResNet model.

SOLUTION:
Swapout: Learning an ensemble of deep architectures