Few-shot Learning by Focusing on Differences

1 INTRODUCTION . Progress in artificial intelligence ( AI ) has been rapid . AI agents have been outperforming humans in an increasing variety of tasks , such as in recognizing images on ImageNet ( He et al. , 2016 ) and in the ancient game of Go ( Silver et al. , 2016 ) . However , challenges remain – systems that outperform humans usually require learning from very large-scale data . In contrast , humans only require few examples to be able to rapidly adapt to a novel task ; humans are still better learners . Few-shot learning methods – which learn classes from few labeled examples – aim to bridge this gap . In learning a classification algorithm from a few labeled examples , one may train the algorithm with a different set of abundantly labeled data ( base set ) ; before adapting it to the unknown examples . However , it might be the case that the available labeled examples are of different granularity level . For example , it is possible that it is only trained to differentiate between cats and dogs , but is tested on differentiating different breeds of dogs . The set of labeled examples is also very limited ; in the extreme case , only one labeled example is provided for each class ( called one-shot classification ) . A few-shot learning algorithm can learn to identify features that are important for doing well on the base set – these can be adapted to classify the few labeled examples as long as the domain remains similar . But with so few examples provided and possible differences in the task granularity , few-shot learning algorithms need to be very biased to perform well . The question is then : what kind of bias is reasonable ? Our work . In this paper , we propose a model that performs classification in a novel task by focusing on the differences between closely related classes of its support set . Our bias is loosely inspired by how scientists often work ( Mill ’ s method of difference ) : in looking for potential causes of a phenomenon , a scientist would often focus on the differences in the circumstances ( features ) in the instance in which the phenomenon occurred and the circumstances in instances for which the phenomenon did not occur ( Mill , 1875 ) . Our method focuses on the differences by removing components in each class representative that are shared with closely related classes . Our contributions . 1 . We introduce a method which incorporates a simple yet effective inductive bias : focusing on differences , and show that it works well on classifying closely related classes . Our results show that the method achieves better performance on a standard fine-grained classification benchmark ( CUB dataset ) and on our proposed benchmark consisting of a mix of fine and coarse-grained classification tasks , CIFAR-Hard . On other commonly used benchmark datasets , CIFAR-FS , mini-ImageNet , and Omniglot , its performance is competitive with existing methods . 2 . We propose a methodology to build harder few-shot learning datasets without requiring very large hierarchically labeled datasets . Using this methodology , we build CIFAR-Hard – a mix of fine-grained and coarse-grained few-shot classification dataset derived from CIFAR-100 . Our empirical evaluation shows that the currently existing methods are not well equipped to handle this scenario . 2 DISSIMILARITY NETWORK . 2.1 FEW-SHOT LEARNING . In few-shot learning , we are given a base set B and a novel set N. The base set contains labeled examples from a large number of classes while novel set contains classes not found in the base set . The objective of few-shot learning is to train a classification algorithm P on the training set Xtrain = B , in such a way that it generalizes to the elements of the novel set N. Some methods may also train on the small number of labeled examples from the novel set . In that case , the training set becomes Xtrain = B ∪ Nlabeled , with labeled novel set Nlabeled ⊂ N. In one-shot learning , the novel set only contains one labeled example for each class , while for k-shot learning , the novel set contains k examples for each class . We use the episodic training proposed by Vinyals et al . ( 2016 ) to make sure that the training and test condition match . At every step , we sample some examples to form an episode T ⊂ Xtrain to train the classification algorithm P . Each episode T consists of a small set of N -labeled examples ( called support set ) S = { ( xi , yi ) } Ni=1 and a set of M examples to be labeled ( called query set ) Q = { xi } Mi=1 , where xi ∈ RD is a D-dimensional feature vector with label yi ( for computing the loss on the query set ) , simulating the conditions for learning the novel set N. We denote the set of labeled examples from the support set with class k ∈ { 1 , ... , K } in an episode as Sk . Our method adopts the approach of similarity learning . Instead of learning a distance or similarity function , we learn a space ( embedding ) that works well with a fixed similarity-based classifier in that space . Specifically , our model learns to construct a space that is optimized to separate data that belong to different classes for a classifier that uses dot-product as its similarity function . We define two levels of embedding based on how the embedding utilizes task information : Global task embedding learns an embedding function that is optimized for all episodes that it is trained upon . The assumption is that the produced embedding will learn a meaningful and general representation that is sufficient to separate data points on the novel task without knowing what classes appear in it . Task-aware embedding removes the assumption that the learned global embedding is sufficient for the novel task . Instead , it takes the possible classes of the novel task into account . On the novel task , it will embed the query set conditioned on the support set that is aware of its member , giving the full context to the prediction . Our model builds an explicit task-aware embedding that separate a class from the weighted average of its closely-related classes . It is explicit in the sense that our model explicitly encodes such an inductive bias into its function ( architecture ) . In contrast , implicit task-aware embedding only relies upon its loss function to adjust its function as to induce separation between classes . 2.2 MODEL . For each class , our model – which we call the Dissimilarity Network – computes a class representative , called a prototype , that is used to classify a new instance of data by comparing its similarity to the prototype , through an inner product . For 1-shot classification , each training example in the novel task is transformed into the prototype , while for k-shot classification , the mean of the k instance representations that belong to each class is transformed into the prototype . The Dissimilarity Network computes prototypes that are dissimilar to one another , in the sense that the components of the class prototypes are orthogonal to the direction of the weighted average of closely-related classes . This lifts the burden of the classifier as the classes are more easily distinguishable – since what remains are their differences . The model works by iteratively enhancing the representation of the prototypes through a set of learned transformations ( embedding functions ) . The first transformation builds a global task embedding through a learned dimensionality-reduction function . The global embedding retains features that are useful for the training tasks but does not take into account the classes that are present in the novel task . The second transformation transforms the prototypes into a task-aware embedding using self-attention networks , taking into account other classes that are present in the task . Finally , the last transformation computes the class prototypes that are dissimilar , by locally orthogonalizing the representations to the weighted average of other closely related class representations . When presented with a new point to classify , it computes the global embedding for that point , transform it using the task-aware embedding , locally orthogonalize the point for each possible class , and select the most similar prototype ’ s class as the class of the new point . Figure 1 illustrates how the model constructs class representations as well as predict the new point . 2.2.1 GLOBAL EMBEDDING . We learn a feature extraction function ff : RD → RH for the images to reduce their representation to a H-dimensional vector . We used deep convolutional neural networks ( Krizhevsky et al. , 2012 ) which captures the local interaction of neighboring pixels and builds a hierarchical representation of them . This feature extractor constructs our first level of embedding . It is trained to extract information that is useful for all the training episodes but is not specialized to a particular novel task . We use the global embedding to compute the class mean ( or prototype ) ck of the H-dimensional representation of the support points , ck = 1 |Sk| ∑ ( xi , yi ) ∈Sk ff ( xi ) . ( 1 ) 2.2.2 SELF-ATTENTION . By seeing the prototypes as a set of vector : C = { ci } Ki=1 , we can learn a set-to-set function that conditions the member of the set to every other member within the set . This gives task-awareness to our prototypes ; each member is now aware of the other class representatives of the given task . Our set-to-set operation is based on the self-attention mechanism introduced by Vaswani et al . ( 2017 ) . Given a query , an attention mechanism learns to ” attend ” – by means of weighted aver- age – to different parts of the set depending on how relevant it is to the query . In self-attention , the query is the member of the set itself . Our self attention uses embedding functions for query hQ , key hK , and value hV . Each embedding function is parameterized by a neural network that computes a mapping RK×H → RK×H . For simplicity we assume that for any input in the form of a set of vector A , it will automatically be cast into matrix A ∈ RK×H , whereas the output will be cast back into a set . The self-attention mechanism self -attn : RK×H → RK×H is formulated as follows . self -attn ( C ) = softmax ( hQ ( C ) hK ( C ) T√ H ) hV ( C ) ( 2 ) Intuitively , the self-attention computes a weighted average of the element of the input matrix C ∈ RK×H , representing a set of prototypes C. This operation has the effect of averaging out noisy components from global embedding that may be relevant for other tasks but are irrelevant to the set of classes in the current novel task . The weights are obtained using the learned attention function , which in this case , is parameterized by hQ and hK . In our case , our prototype ck learns to be aware of the other class representatives C \ ck by incorporating some of their components . We use bidirectional LSTM ( BLSTM ) ( Hochreiter & Schmidhuber , 1997 ) for our attention embedding function for query hQ and key hK , while using either identity function or BLSTM on hV . BLSM computes a concatenation of two sequence of opposing-direction by sequentially applying the element xt ∈ RH of its input x = [ x1 , ... , xT ] into an LSTM , ht , ut = LSTM ( xt−1 , ht−1 , ut−1 ) . The computation yields a sequence of vector that each are conditioned on its neighboring elements . Our BLSTM uses context sharing between key attention embedding hK and query attention embedding hQ . We are aware of the sequential nature of BLSTM , which can be counter-intuitive as we are modeling a set-to-set operation which should not have any preference for ordering . However , we found empirically that this setup offers more performance gain compared to the use of traditional linear function as attention embedding function . The BLSTM may learn to ignore the unimportance of set ordering due to the nature of episodic training , which exposes it to many permutations of the possible class-orderings . Moreover , the attention also gives a global context of the member of the set , which could further alleviate the ordering issues ( if any ) .

Summary: The stated contributions of the paper are: (1) a method for performing few-shot learning and (2) an approach for building harder few-shot learning datasets from existing datasets. The authors describe a model for creating a task-aware embedding for different novel sets (for different image classification settings) using a nonlinear self-attention-like mechanism applied to the centroid of the global embeddings for each class. The resulting embeddings are used per class with an additional attention layer applied on the embeddings from the other classes to identify closely-related classes and consider the part of the embedding orthogonal to the attention-weighted-average of these closely-related classes. They compare the accuracy of their model vs others in the 1-shot and 5-shot setting on various datasets, including a derived dataset from CIFAR which they call Hierarchical-CIFAR.