Few-Shot Few-Shot Learning and the role of Spatial Attention

1 INTRODUCTION . The ever improving performance of deep learning models , apart from technical progress , is largely due to the existence of large-scale datasets , fully or weakly labeled by humans ( Russakovsky et al. , 2015 ; Kuznetsova et al. , 2018 ; Mahajan et al. , 2018 ) . At the same time , reducing the need for supervision is becoming increasingly important , e.g . by taking advantage of prior learning ( Bengio et al. , 2011 ; Mallya et al. , 2018 ; Li & Hoiem , 2018 ) or exploiting unlabeled data ( Caron et al. , 2018 ; Radosavovic et al. , 2018 ) . An extreme situation is few-shot learning ( Koch et al. , 2015 ; Vinyals et al. , 2016 ; Snell et al. , 2017 ; Hariharan & Girshick , 2017 ) , where the problem is to learn novel ( previously unseen ) classes using only a very small labeled training set , typically not more than 10 examples per class . Here not only the annotation but even the raw data are not available . This problem is often motivated by the ability of humans to learn new tasks from few examples ( Lake et al. , 2011 ; 2015 ) , which has given rise to meta-learning ( Santoro et al. , 2016 ; Mishra et al. , 2017 ; Ravi & Larochelle , 2017 ; Bertinetto et al. , 2018 ; Han et al. , 2018 ) , or learning to learn . In this scenario , a training set is treated as a collection of smaller sets where every class has few labeled examples . However , there is a huge gap between the motivating example of humans learning new tasks and how the few-shot classification task is set up . On one hand , for the sake of simplicity in experiments , the base class datasets where the representation is learned from scratch , contain a few dozen or hundred classes with a few hundred examples each . This is by no means comparable to datasets available to date ( Kuznetsova et al. , 2018 ; Mahajan et al. , 2018 ) , let alone the amount of prior knowledge that a human may have accumulated before learning a new task . On the other hand , for a given domain of novel classes , e.g . bird species ( Welinder et al. , 2010 ) , base class data of such size in the same domain may not exist . In this work , we depart from the standard few-shot classification scenario in two directions . First , we allow the representation to be learned from a large-scale dataset in a domain different than the base and novel class domain . In particular , we model prior knowledge by a classifier that is pre-trained on such a dataset , having no access to its training process . We thus maintain the difficulty of domain gap and the simplicity of experiments ( by not training from scratch ) , while allowing a powerful representation . Second , we assume only few or zero examples per base class . Hence , the role of base classes is to adapt the representation to the domain at hand rather than learn from scratch . This scenario can be seen as few-shot version of few-shot learning . We treat this problem as a two-stage adaptation process , first on the few base class examples if available and second on the even fewer novel class examples . Because of the limited amount of data , it is not appropriate to apply e.g . transfer learning ( Bengio et al. , 2011 ) or domain adaptation ( Ganin & Lempitsky , 2014 ) , in either of the two stages . Because the network is pre-trained , and we do not have access to its training process or data , meta-learning is not an option either . We thus resort to few steps of fine-tuning as in the meta-testing stage of Finn et al . ( 2017 ) and Ravi & Larochelle ( 2017 ) . Focusing on image classification , we then investigate the role of spatial attention in the new problem . With large base class datasets , the network can implicitly learn the relevant parts of the images where to focus . In our setup , base class data are few , so our motivation is that a spatial attention mechanism may help the classifier in focusing on objects , suppressing background clutter . We observe that although the prior classes of the pre-trained classifier may be irrelevant to a new task , uncertainty over a large number of such classes may express anything unknown like background . This is a class-agnostic property and can apply to new tasks . In particular , given an input image , we measure the entropy-based certainty of the pre-trained classifier in its prediction on the prior classes at every spatial location and we use it to construct a spatial attention map . This map can be utilized in a variety of ways , for instance weighted spatial pooling or weighted loss per location ; and at different situations like the two adaptation stages or at inference . By exploring different alternatives , we show that a pre-trained network may be easily adapted to novel classes , without meta-learning . In the following , we provide a detailed problem formulation and related background in section 2 , then we describe our spatial attention mechanism in section 3 and its use in few-shot classification in section 4 . We provide experimental results in section 5 , and we conclude in section 6 . 2 PROBLEM , BACKGROUND , RELATED WORK AND CONTRIBUTION . Few-shot classification . We are given a set of training examples X : = { xi } ni=1 ⊂ X , and corresponding labels y : = ( yi ) ni=1 ⊂ Cn where C : = [ c ] : = { 1 , . . . , c } is a set of base classes . The objective is to learn a representation on these data , a process that we call base training , such that we can solve new tasks . A new task comprises a set of support examples X ′ : = { x′i } n ′ i=1 ⊂ X and labels y′ : = ( y′i ) n′ i=1 ⊂ ( C ′ ) n ′ , where n′ n and C ′ : = [ c′ ] is a set of novel classes disjoint from C. The most common setting is k′ examples per novel class , so that n′ = k′c′ , referred to as c′-way , k′-shot classification . The objective now is to learn a classifier on these support data , a process that we call adaptation . This classifier should map a new query example from X to a prediction in C ′ . Few-shot few-shot classification . Few-shot classification assumes there is more data in base than novel classes , and a domain shift between the two , in the sense of no class overlap . Here we consider a modified problem where n can be small or zero , but there is another set C◦ = [ c◦ ] of prior classes with even more data X◦ and labels y◦ with n n◦ : = |X◦| and a greater domain shift to C , C ′ . Again , the most common setting is k examples per base class , so that n = kc . We are using a classifier that is pre-trained on this data but we do not have direct access to either X◦ , y◦ , or its learning process . The objective of base training is now to adapt the representation to the domain of C , C ′ rather than learn it from scratch ; but we still call it base training . In the remaining of this section we present general background on few-shot classification that typically applies to base classes C or novel classes C ′ , but may also apply to both , in which case the symbols c , C and c′ , C ′ may be used interchangeably . Prior classes and the pre-trained network are only considered in the following sections . Classifier . The classifier is a function fθ , W : X → Rc ( resp . Rc ′ ) with learnable parameters θ , W , mapping a new example x ∈ X to a vector of probabilities p : = fθ , W ( x ) over c ( resp . c′ ) base ( resp . novel ) classes . The classifier prediction is the class of maximum probability π ( p ) : = arg max j pj , ( 1 ) where pj is the j-th element of p. The classifier is built on top of an embedding function φθ : X → Rr×d . Given an example x ∈ X , this function yields a r× d feature tensor φθ ( x ) , where r represents the spatial dimensions and d the feature dimensions . For X comprising 2d images for instance , the feature is a w × h× d tensor that is the activation of the last convolutional layer , where r = w × h is the spatial resolution . The embedding is a vector in Rd in the special case r = 1 . The embedding parameters θ may be updated into θ′ in the adaptation process , in which case we have an embedding function φθ′ and classifier fθ′ , W . Again θ , θ′ may be used interchangeably . Cosine classifier . A simple form of classifier that was introduced in few-shot learning independently by Qi et al . ( 2018 ) and Gidaris & Komodakis ( 2018 ) is a parametric linear classifier that consists of a fully-connected layer without bias on top of the embedding function φθ followed by softmax . If W : = ( wj ) c j=1 is the collection of class weights with wj ∈ Rr×d , the classifier is defined by fθ , W ( x ) : = σ ( τ [ s ( φθ ( x ) , wj ) ] c j=1 ) ( 2 ) for x ∈ X , where σ : Rm → Rm is the softmax function σ ( u ) : = ( e u1 , ... , euc ) ∑ j e uj for u ∈ Rc , τ ∈ R+ is a trainable scale parameter and s is cosine similarity1 . We minimize the cost function J ( X , y ; θ , W ) : = n∑ i=1 ` ( fθ , W ( xi ) , yi ) ( 3 ) over θ , W at base training , where ` ( p , y ) : = − log py for p ∈ Rc+ , y ∈ C is the cross-entropy loss . Prototypes . An alternative classifier that is more appropriate during few-shot adaptation or at testing is a prototype classifier proposed by Snell et al . ( 2017 ) and followed by Qi et al . ( 2018 ) and Gidaris & Komodakis ( 2018 ) too . If Sj : = { i ∈ [ n′ ] : y′i = j } denotes the indices of support examples labeled in class j , then the prototype of this class j is given by the average features pj = 1 |Sj | ∑ i∈Sj φθ′ ( x ′ i ) ( 4 ) of those examples for j ∈ C ′ . Then , denoting by P : = ( pj ) c ′ j=1 the collection of prototypes , a query x ∈ X is classified as fθ′ , P ( x ) , as defined by ( 2 ) . Dense classifier . This is a classifier where the loss function applies densely at each spatial location of the feature tensor rather than by global pooling or flattening as implied by ( 2 ) . The classifier can be of any form but a cosine classifier ( Qi et al. , 2018 ; Gidaris & Komodakis , 2018 ) is studied by Lifchitz et al . ( 2019 ) . In particular , the embedding φθ ( x ) is seen as a collection of vectors [ φ ( q ) ( x ) ] rq=1 , where φ ( q ) ( x ) ∈ Rd is an embedding of spatial location q ∈ [ r ] . The classifier ( 2 ) is then generalized to fθ , W : X → Rr×c , now mapping an example to a vector of probabilities per location , defined by fθ , W ( x ) : = [ σ ( τ [ s ( φ ( q ) θ ( x ) , wj ) ] c j=1 ) ] r q=1 ( 5 ) for x ∈ X , while the class weights W are shared over locations with wj ∈ Rd . Cross-entropy applies using the same label yi for each location of example xi , generalizing the cost function ( 3 ) to J ( X , y ; θ , W ) : = n∑ i=1 r∑ q=1 ` ( f ( q ) θ , W ( xi ) , yi ) . ( 6 ) Related work . Prototypical networks ( Snell et al. , 2017 ) use a prototype classifier . At testing , a query is classified to the nearest prototype , while at adaptation , computing a prototype per class ( 4 ) 1For matrices u , v ∈ Rr×d , s ( u , v ) : = 〈u , v〉 / ( ‖u‖ ‖v‖ ) with 〈· , ·〉 and ‖·‖ being the Frobenius inner product and norm respectively . is the only learning to be done . Base training is based on meta-learning : a number of fictitious tasks called episodes are generated by randomly sampling a number of classes from C and then a number of examples in each class from X with their labels from y . These data are assumed to be support examples and queries of novel classes C ′ . Labels are now available for the queries and the objective is that they are classified correctly . In imprinting ( Qi et al. , 2018 ) , a cosine classifier ( 2 ) and standard cross-entropy ( 3 ) are used instead at base training . At adaptation , class prototypes P are computed ( 4 ) and imprinted in the classifier , that is , W is replaced by W ′ : = ( W , P ) . The entire embedding function is then fine-tuned based again on ( 3 ) to make predictions on n + n′ base and novel classes , which requires the entire training data ( X , y ) . Few-shot learning without forgetting ( Gidaris & Komodakis , 2018 ) uses model similar to imprinting , the main difference being that only the weight parameters W of the base classes are stored and not the entire training data . At base training , a cosine classifier is trained by ( 3 ) followed by episodes . At adaptation , prototypes are adapted to W by a class attention mechanism . Model agnostic meta-learning ( MAML ) ( Finn et al. , 2017 ) uses a fully-connected layer as classifier . At adaptation , the entire embedding function is fine-tuned on each new task using ( 3 ) only on the novel class data , but for few steps such that the classifier does not overfit . At base training , episodes are used where the loss function mimics the iterative optimization that normally takes place at adaptation . In implanting ( Lifchitz et al. , 2019 ) , a prototype classifier is trained in episodes , keeping the base embedding function fixed but attaching a parallel implant stream of convolutional layers that learns features useful for each new task . Contribution . The problem we consider is a variant of few-shot learning that has not been studied before . It involves sequential adaptation of a given network in two stages , each comprising a limited amount of data . There are many ways of exploiting prior learning to reduce the required amount data and supervision like transfer learning ( Bengio et al. , 2011 ; Yosinski et al. , 2014 ) , domain adaptation ( Ganin & Lempitsky , 2014 ; Rebuffi et al. , 2017 ; Mallya et al. , 2018 ) , or incremental learning ( Li & Hoiem , 2018 ; Rebuffi et al. , 2016 ; Yoon et al. , 2018 ) . However , none applies to the few-shot domain where just a handful of examples are given . Attention has been studied as a core component of several few-shot learning and meta-learning approaches ( Vinyals et al. , 2016 ; Mishra et al. , 2018 ; Ren et al. , 2018 ; Gidaris & Komodakis , 2018 ) , but it always refers to examples ( e.g . images ) as a unit . Spatial attention on the other hand refers to neurons at different spatial locations . It is ubiquitous in several problems , for instance weakly supervised object detection ( Zhou et al. , 2016 ; Hou et al. , 2018 ; Zhu et al. , 2017 ) and non-local convolution ( Hu et al. , 2018 ; Wang et al. , 2018 ; Chen et al. , 2018 ) but has not been applied to few-shot learning until recently ( Lifchitz et al. , 2019 ; Wertheimer & Hariharan , 2019 ; Li et al. , 2019b ; Liu et al. , 2019 ) . By breaking up one example ( e.g . image ) to several , one at each spatial location , the supervision becomes weak , as in multiple-instance learning ( Andrews et al. , 2003 ) , typical e.g . in weakly-supervised image segmentation ( Jetley et al. , 2018 ) . Our spatial attention mechanism is extremely simple , based on the pre-trained network , without training on the base or novel class data .

Summary: The paper introduces a problem “few-shot few-shot learning” that aims to firstly transfer prior knowledge from one domain to the domain where the base training tasks reside, and then train a few-shot learning model on training tasks and apply it to novel test tasks. The two “few-shot” in the name refers to base training tasks and novel test tasks. In their algorithm, they use a model pre-trained on another dataset as the prior knowledge and fine-tune it on training tasks. During the test, they use the weighted average of samples’ representations per class as the prototype of each class, where the weight is large for samples with more discriminative prediction over pre-trained domain’s classes. Afterward, classification is reduced to finding the nearest neighbor among the class prototypes. Some experiments show that the pre-trained model can improve few-shot classification accuracy.