Few-Shot Few-Shot Learning and the role of Spatial Attention

1 INTRODUCTION . The ever improving performance of deep learning models , apart from technical progress , is largely due to the existence of large-scale datasets , fully or weakly labeled by humans ( Russakovsky et al. , 2015 ; Kuznetsova et al. , 2018 ; Mahajan et al. , 2018 ) . At the same time , reducing the need for supervision is becoming increasingly important , e.g . by taking advantage of prior learning ( Bengio et al. , 2011 ; Mallya et al. , 2018 ; Li & Hoiem , 2018 ) or exploiting unlabeled data ( Caron et al. , 2018 ; Radosavovic et al. , 2018 ) . An extreme situation is few-shot learning ( Koch et al. , 2015 ; Vinyals et al. , 2016 ; Snell et al. , 2017 ; Hariharan & Girshick , 2017 ) , where the problem is to learn novel ( previously unseen ) classes using only a very small labeled training set , typically not more than 10 examples per class . Here not only the annotation but even the raw data are not available . This problem is often motivated by the ability of humans to learn new tasks from few examples ( Lake et al. , 2011 ; 2015 ) , which has given rise to meta-learning ( Santoro et al. , 2016 ; Mishra et al. , 2017 ; Ravi & Larochelle , 2017 ; Bertinetto et al. , 2018 ; Han et al. , 2018 ) , or learning to learn . In this scenario , a training set is treated as a collection of smaller sets where every class has few labeled examples . However , there is a huge gap between the motivating example of humans learning new tasks and how the few-shot classification task is set up . On one hand , for the sake of simplicity in experiments , the base class datasets where the representation is learned from scratch , contain a few dozen or hundred classes with a few hundred examples each . This is by no means comparable to datasets available to date ( Kuznetsova et al. , 2018 ; Mahajan et al. , 2018 ) , let alone the amount of prior knowledge that a human may have accumulated before learning a new task . On the other hand , for a given domain of novel classes , e.g . bird species ( Welinder et al. , 2010 ) , base class data of such size in the same domain may not exist . In this work , we depart from the standard few-shot classification scenario in two directions . First , we allow the representation to be learned from a large-scale dataset in a domain different than the base and novel class domain . In particular , we model prior knowledge by a classifier that is pre-trained on such a dataset , having no access to its training process . We thus maintain the difficulty of domain gap and the simplicity of experiments ( by not training from scratch ) , while allowing a powerful representation . Second , we assume only few or zero examples per base class . Hence , the role of base classes is to adapt the representation to the domain at hand rather than learn from scratch . This scenario can be seen as few-shot version of few-shot learning . We treat this problem as a two-stage adaptation process , first on the few base class examples if available and second on the even fewer novel class examples . Because of the limited amount of data , it is not appropriate to apply e.g . transfer learning ( Bengio et al. , 2011 ) or domain adaptation ( Ganin & Lempitsky , 2014 ) , in either of the two stages . Because the network is pre-trained , and we do not have access to its training process or data , meta-learning is not an option either . We thus resort to few steps of fine-tuning as in the meta-testing stage of Finn et al . ( 2017 ) and Ravi & Larochelle ( 2017 ) . Focusing on image classification , we then investigate the role of spatial attention in the new problem . With large base class datasets , the network can implicitly learn the relevant parts of the images where to focus . In our setup , base class data are few , so our motivation is that a spatial attention mechanism may help the classifier in focusing on objects , suppressing background clutter . We observe that although the prior classes of the pre-trained classifier may be irrelevant to a new task , uncertainty over a large number of such classes may express anything unknown like background . This is a class-agnostic property and can apply to new tasks . In particular , given an input image , we measure the entropy-based certainty of the pre-trained classifier in its prediction on the prior classes at every spatial location and we use it to construct a spatial attention map . This map can be utilized in a variety of ways , for instance weighted spatial pooling or weighted loss per location ; and at different situations like the two adaptation stages or at inference . By exploring different alternatives , we show that a pre-trained network may be easily adapted to novel classes , without meta-learning . In the following , we provide a detailed problem formulation and related background in section 2 , then we describe our spatial attention mechanism in section 3 and its use in few-shot classification in section 4 . We provide experimental results in section 5 , and we conclude in section 6 . 2 PROBLEM , BACKGROUND , RELATED WORK AND CONTRIBUTION . Few-shot classification . We are given a set of training examples X : = { xi } ni=1 ⊂ X , and corresponding labels y : = ( yi ) ni=1 ⊂ Cn where C : = [ c ] : = { 1 , . . . , c } is a set of base classes . The objective is to learn a representation on these data , a process that we call base training , such that we can solve new tasks . A new task comprises a set of support examples X ′ : = { x′i } n ′ i=1 ⊂ X and labels y′ : = ( y′i ) n′ i=1 ⊂ ( C ′ ) n ′ , where n′ n and C ′ : = [ c′ ] is a set of novel classes disjoint from C. The most common setting is k′ examples per novel class , so that n′ = k′c′ , referred to as c′-way , k′-shot classification . The objective now is to learn a classifier on these support data , a process that we call adaptation . This classifier should map a new query example from X to a prediction in C ′ . Few-shot few-shot classification . Few-shot classification assumes there is more data in base than novel classes , and a domain shift between the two , in the sense of no class overlap . Here we consider a modified problem where n can be small or zero , but there is another set C◦ = [ c◦ ] of prior classes with even more data X◦ and labels y◦ with n n◦ : = |X◦| and a greater domain shift to C , C ′ . Again , the most common setting is k examples per base class , so that n = kc . We are using a classifier that is pre-trained on this data but we do not have direct access to either X◦ , y◦ , or its learning process . The objective of base training is now to adapt the representation to the domain of C , C ′ rather than learn it from scratch ; but we still call it base training . In the remaining of this section we present general background on few-shot classification that typically applies to base classes C or novel classes C ′ , but may also apply to both , in which case the symbols c , C and c′ , C ′ may be used interchangeably . Prior classes and the pre-trained network are only considered in the following sections . Classifier . The classifier is a function fθ , W : X → Rc ( resp . Rc ′ ) with learnable parameters θ , W , mapping a new example x ∈ X to a vector of probabilities p : = fθ , W ( x ) over c ( resp . c′ ) base ( resp . novel ) classes . The classifier prediction is the class of maximum probability π ( p ) : = arg max j pj , ( 1 ) where pj is the j-th element of p. The classifier is built on top of an embedding function φθ : X → Rr×d . Given an example x ∈ X , this function yields a r× d feature tensor φθ ( x ) , where r represents the spatial dimensions and d the feature dimensions . For X comprising 2d images for instance , the feature is a w × h× d tensor that is the activation of the last convolutional layer , where r = w × h is the spatial resolution . The embedding is a vector in Rd in the special case r = 1 . The embedding parameters θ may be updated into θ′ in the adaptation process , in which case we have an embedding function φθ′ and classifier fθ′ , W . Again θ , θ′ may be used interchangeably . Cosine classifier . A simple form of classifier that was introduced in few-shot learning independently by Qi et al . ( 2018 ) and Gidaris & Komodakis ( 2018 ) is a parametric linear classifier that consists of a fully-connected layer without bias on top of the embedding function φθ followed by softmax . If W : = ( wj ) c j=1 is the collection of class weights with wj ∈ Rr×d , the classifier is defined by fθ , W ( x ) : = σ ( τ [ s ( φθ ( x ) , wj ) ] c j=1 ) ( 2 ) for x ∈ X , where σ : Rm → Rm is the softmax function σ ( u ) : = ( e u1 , ... , euc ) ∑ j e uj for u ∈ Rc , τ ∈ R+ is a trainable scale parameter and s is cosine similarity1 . We minimize the cost function J ( X , y ; θ , W ) : = n∑ i=1 ` ( fθ , W ( xi ) , yi ) ( 3 ) over θ , W at base training , where ` ( p , y ) : = − log py for p ∈ Rc+ , y ∈ C is the cross-entropy loss . Prototypes . An alternative classifier that is more appropriate during few-shot adaptation or at testing is a prototype classifier proposed by Snell et al . ( 2017 ) and followed by Qi et al . ( 2018 ) and Gidaris & Komodakis ( 2018 ) too . If Sj : = { i ∈ [ n′ ] : y′i = j } denotes the indices of support examples labeled in class j , then the prototype of this class j is given by the average features pj = 1 |Sj | ∑ i∈Sj φθ′ ( x ′ i ) ( 4 ) of those examples for j ∈ C ′ . Then , denoting by P : = ( pj ) c ′ j=1 the collection of prototypes , a query x ∈ X is classified as fθ′ , P ( x ) , as defined by ( 2 ) . Dense classifier . This is a classifier where the loss function applies densely at each spatial location of the feature tensor rather than by global pooling or flattening as implied by ( 2 ) . The classifier can be of any form but a cosine classifier ( Qi et al. , 2018 ; Gidaris & Komodakis , 2018 ) is studied by Lifchitz et al . ( 2019 ) . In particular , the embedding φθ ( x ) is seen as a collection of vectors [ φ ( q ) ( x ) ] rq=1 , where φ ( q ) ( x ) ∈ Rd is an embedding of spatial location q ∈ [ r ] . The classifier ( 2 ) is then generalized to fθ , W : X → Rr×c , now mapping an example to a vector of probabilities per location , defined by fθ , W ( x ) : = [ σ ( τ [ s ( φ ( q ) θ ( x ) , wj ) ] c j=1 ) ] r q=1 ( 5 ) for x ∈ X , while the class weights W are shared over locations with wj ∈ Rd . Cross-entropy applies using the same label yi for each location of example xi , generalizing the cost function ( 3 ) to J ( X , y ; θ , W ) : = n∑ i=1 r∑ q=1 ` ( f ( q ) θ , W ( xi ) , yi ) . ( 6 ) Related work . Prototypical networks ( Snell et al. , 2017 ) use a prototype classifier . At testing , a query is classified to the nearest prototype , while at adaptation , computing a prototype per class ( 4 ) 1For matrices u , v ∈ Rr×d , s ( u , v ) : = 〈u , v〉 / ( ‖u‖ ‖v‖ ) with 〈· , ·〉 and ‖·‖ being the Frobenius inner product and norm respectively . is the only learning to be done . Base training is based on meta-learning : a number of fictitious tasks called episodes are generated by randomly sampling a number of classes from C and then a number of examples in each class from X with their labels from y . These data are assumed to be support examples and queries of novel classes C ′ . Labels are now available for the queries and the objective is that they are classified correctly . In imprinting ( Qi et al. , 2018 ) , a cosine classifier ( 2 ) and standard cross-entropy ( 3 ) are used instead at base training . At adaptation , class prototypes P are computed ( 4 ) and imprinted in the classifier , that is , W is replaced by W ′ : = ( W , P ) . The entire embedding function is then fine-tuned based again on ( 3 ) to make predictions on n + n′ base and novel classes , which requires the entire training data ( X , y ) . Few-shot learning without forgetting ( Gidaris & Komodakis , 2018 ) uses model similar to imprinting , the main difference being that only the weight parameters W of the base classes are stored and not the entire training data . At base training , a cosine classifier is trained by ( 3 ) followed by episodes . At adaptation , prototypes are adapted to W by a class attention mechanism . Model agnostic meta-learning ( MAML ) ( Finn et al. , 2017 ) uses a fully-connected layer as classifier . At adaptation , the entire embedding function is fine-tuned on each new task using ( 3 ) only on the novel class data , but for few steps such that the classifier does not overfit . At base training , episodes are used where the loss function mimics the iterative optimization that normally takes place at adaptation . In implanting ( Lifchitz et al. , 2019 ) , a prototype classifier is trained in episodes , keeping the base embedding function fixed but attaching a parallel implant stream of convolutional layers that learns features useful for each new task . Contribution . The problem we consider is a variant of few-shot learning that has not been studied before . It involves sequential adaptation of a given network in two stages , each comprising a limited amount of data . There are many ways of exploiting prior learning to reduce the required amount data and supervision like transfer learning ( Bengio et al. , 2011 ; Yosinski et al. , 2014 ) , domain adaptation ( Ganin & Lempitsky , 2014 ; Rebuffi et al. , 2017 ; Mallya et al. , 2018 ) , or incremental learning ( Li & Hoiem , 2018 ; Rebuffi et al. , 2016 ; Yoon et al. , 2018 ) . However , none applies to the few-shot domain where just a handful of examples are given . Attention has been studied as a core component of several few-shot learning and meta-learning approaches ( Vinyals et al. , 2016 ; Mishra et al. , 2018 ; Ren et al. , 2018 ; Gidaris & Komodakis , 2018 ) , but it always refers to examples ( e.g . images ) as a unit . Spatial attention on the other hand refers to neurons at different spatial locations . It is ubiquitous in several problems , for instance weakly supervised object detection ( Zhou et al. , 2016 ; Hou et al. , 2018 ; Zhu et al. , 2017 ) and non-local convolution ( Hu et al. , 2018 ; Wang et al. , 2018 ; Chen et al. , 2018 ) but has not been applied to few-shot learning until recently ( Lifchitz et al. , 2019 ; Wertheimer & Hariharan , 2019 ; Li et al. , 2019b ; Liu et al. , 2019 ) . By breaking up one example ( e.g . image ) to several , one at each spatial location , the supervision becomes weak , as in multiple-instance learning ( Andrews et al. , 2003 ) , typical e.g . in weakly-supervised image segmentation ( Jetley et al. , 2018 ) . Our spatial attention mechanism is extremely simple , based on the pre-trained network , without training on the base or novel class data .

Summary: This paper proposed a new realistic setting for few-shot learning that we can obtain representations from a pre-trained model trained on a large-scale dataset, but cannot access its training details. Also, there may be a large domain shift between the dataset of the pre-trained model and our dataset. For the pre-trained model, they will not only use its weights but also use it to generate a spatial attention map and help the model focuses on objects of images. Back to the standard few-shot classification problem, they will first adapt the model with base class samples and then adapt to novel classes.