Attentional Constellation Nets for Few-Shot Learning

1 INTRODUCTION . Tremendous progress has been made in both the development and the applications of the deep convolutional neural networks ( CNNs ) ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2015 ; Szegedy et al. , 2015 ; He et al. , 2016 ; Xie et al. , 2017 ) . Visualization of the internal CNN structure trained on e.g . ImageNet ( Deng et al. , 2009 ) has revealed the increasing level of semantic relevance for the learned convolution kernels/filters to the semantics of the object classes , displaying bar/edge like patterns in the early layers , object parts in the middle layers , and face/object like patterns in the higher layers ( Zeiler & Fergus , 2014 ) . In general , we consider the learned convolution kernels being somewhat implicit about the underlying objects since they represent projections/mappings for the input but without the explicit knowledge about the parts in terms of their numbers , distributions , and spatial configurations . On the other hand , there has been a rich history about explicit object representations starting from deformable templates ( Yuille et al. , 1992 ) , pictorial structure ( Felzenszwalb & Huttenlocher , 2005 ) , constellation models ( Weber et al. , 2000 ; Fergus et al. , 2003 ; Sudderth et al. , 2005 ; Fei-Fei et al. , 2006 ) , and grammar-based model ( Zhu & Mumford , 2007 ) . These part-based models ( Weber et al. , 2000 ; Felzenszwalb & Huttenlocher , 2005 ; Fergus et al. , 2003 ; Sudderth et al. , 2005 ; Zhu & Mumford , 2007 ) share three common properties in the algorithm design : ( 1 ) unsupervised learning , ( 2 ) explicit clustering to obtain the parts , and ( 3 ) modeling to characterize the spatial configuration of the parts . Compared to the CNN architectures , these methods are expressive with explicit part-based representation . They have pointed to a promising direction for object recognition , albeit a lack of strong practice performance on the modern datasets . Another line of object recognition system with the part concept but trained discriminatively includes the discriminative trained part-based model ( DPM ) ( Felzenszwalb et al. , 2009 ) and the spatial pyramid matching method ( SPM ) ( Lazebnik et al. , 2006 ) . In the context of deep learning , efforts exist to bring the explicit part representation into deep hierarchical structures ( Salakhutdinov et al. , 2012 ) . The implicit and explicit feature representations could share mutual benefits , especially in fewshot learning where training data is scarce : CNNs may face difficulty in learning a generalized representation due to lack of sufficient training data , whereas clustering and dictionary learning ∗indicates equal contribution provide a direct means for data abstraction . In general , end-to-end learning of both the implicit and explicit part-based representations is a viable and valuable means in machine learning . We view convolutional features as an implicit part-based representation since they are learned through back-propagation via filtering processes . On the other hand , an explicit representation can be attained by introducing feature clustering that captures the data abstraction/distribution under a mixture model . In this paper , we develop an end-to-end framework to combine the implicit and explicit part-based representations for the few-shot classification task by seamlessly integrating constellation models with convolution operations . In addition to keeping a standard CNN architecture , we also employ a cell feature clustering module to encode the potential object parts . This procedure is similar to the clustering/codebook learning for appearance in the constellation model ( Weber et al. , 2000 ) . The cell feature clustering process generates a dense distance map . We further model the relations for the cells using a self-attention mechanism , resembling the spatial configuration design in the constellation model ( Weber et al. , 2000 ) . Thus , we name our method constellation networks ( ConstellationNet ) . We demonstrate the effectiveness of our approach on standard few-shot benchmarks , including FC100 ( Oreshkin et al. , 2018 ) , CIFAR-FS ( Bertinetto et al. , 2018 ) and mini-ImageNet ( Vinyals et al. , 2016 ) by showing a significant improvement over the existing methods . An ablation study also demonstrates the effectiveness of ConstellationNet is not achieved by simply increasing the model complexity using e.g . more convolution channels or deeper and wider convolution layers ( WRN-28-10 ( Zagoruyko & Komodakis , 2016 ) ) ( see ablation study in Table 3 and Figure 2 ( e ) ) . 2 RELATED WORK . Few-Shot Learning . Recently , few-shot learning attracts much attention in the deep learning community ( Snell et al. , 2017 ; Lee et al. , 2019 ) . Current few-shot learning is typically formulated as a meta-learning problem ( Finn et al. , 2017 ) , in which an effective feature embedding is learned for generalization across novel tasks . We broadly divide the existing few-shot learning approaches into three categories : ( 1 ) Gradient-based methods optimize feature embedding with gradient descent during meta-test stage ( Finn et al. , 2017 ; Bertinetto et al. , 2018 ; Lee et al. , 2019 ) . ( 2 ) Metric-based methods learn a fixed optimal embedding with a distance-based prediction rule ( Vinyals et al. , 2016 ; Snell et al. , 2017 ) . ( 3 ) Model-based methods obtains a conditional feature embedding via a weight predictor ( Mishra et al. , 2017 ; Munkhdalai et al. , 2017 ) . Here we adopt ProtoNet ( Snell et al. , 2017 ) , a popular metric-based framework , in our approach and boost the generalization ability of the feature embeddings with explicit structured representations from the constellation model . Recently , Tokmakov et al . ( 2019 ) proposes a compositional regularization to the image with its attribute annotations , which is different from out unsupervised part-discovery strategy . Part-Based Constellation/Discriminative Models . The constellation model family ( Weber et al. , 2000 ; Felzenszwalb & Huttenlocher , 2005 ; Fergus et al. , 2003 ; Sudderth et al. , 2005 ; Fei-Fei et al. , 2006 ; Zhu & Mumford , 2007 ) is mostly generative/expressive that shares two commonalities in the representation : ( 1 ) clustering/codebook learning in the appearance and ( 2 ) modeling of the spatial configurations . The key difference among these approaches lies in how the spatial configuration is modeled : Gaussian distributions ( Weber et al. , 2000 ) ; pictorial structure ( Felzenszwalb & Huttenlocher , 2005 ) ; joint shape model ( Fergus et al. , 2003 ) ; hierarchical graphical model ( Sudderth et al. , 2005 ) ; grammar-based ( Zhu & Mumford , 2007 ) . These constellation models represent a promising direction for object recognition but are not practical competitive compared with deep learning based approaches . There are also discriminative models : The discriminatively trained part-based model ( DPM ) ( Felzenszwalb et al. , 2009 ) is a typical method in this vein where object parts ( as HOG features ( Dalal & Triggs , 2005 ) ) and their configurations ( a star model ) are learned jointly in a discriminative way . The spatial pyramid matching method ( SPM ) ( Lazebnik et al. , 2006 ) has no explicit parts but instead builds on top of different levels of grids with codebook learned on top of the SIFT features ( Lowe , 2004 ) . DPM and SPM are of practical significance for object detection and recognition . In our approach , we implement the constellation model with cell feature clustering and attention-based cell relation modeling to demonstrate the appearance learning and spatial configuration respectively . Parts models are extensively studied in fine-grained image classifications and object detection to provide spatial guidance for filtering uninformative object proposals ( Simon & Rodner , 2015 ; Peng et al. , 2017 ; Zhu et al. , 2017 ; Ge et al. , 2019 ; Qi et al. , 2019 ) . Related to our work , Neural Activation Constellations ( NAC ) ( Simon & Rodner , 2015 ) introduces the constellation model to perform unsupervised part model discovery with convolutional networks . Our work is different from NAC in three aspects : ( 1 ) The algorithmic mechanisms behind Simon & Rodner ( 2015 ) and ours are different . Simon & Rodner ( 2015 ) implements a traditional Gaussian-based constellation module to model the spatial configuration and part selection on top of a fixed pre-trained CNN . However , in our ConstellationNet , our part representation and spatial configuration are modeled by cell feature clustering and self-attention based cell relation module , which is general-purpose , modularized and recursive . ( 2 ) In Simon & Rodner ( 2015 ) , the constellation module is optimized in an EM-like algorithm , which is separate from the CNN optimization . Our constellation modules are seamlessly integrated into the current CNNs and jointly optimized with them . ( 3 ) Our ConstellationNet uses the dense cell features from the CNN feature maps , which considers all positions from the images as potential parts and models their relation . However , ( Simon et al . 2015 ) extracts sparse part representations ( i.e . it uses at most one part proposal per channel and selects even less parts later ) , which may not fully utilize the rich information from the CNN feature maps . 3 FEW-SHOT LEARNING . In a standard classification problem , we aim to learn a model trained on the dataset Dbase that can generalize its classification ability to unseen test set Dnovel belonging to same categories . In few-shot classification problem , we encourage Dbase and Dnovel to be formed from different categories to emphasize model ’ s generalization ability on novel categories , where we denote training categories as Cbase , test categories as Cnovel , and Cbase ∩ Cnovel = ∅ to ensure the fairness . In the training stage ( a.k.a . meta-train stage ) , metric-based few-shot learning approaches ( Snell et al. , 2017 ; Vinyals et al. , 2016 ; Oreshkin et al. , 2018 ) usually learn a feature extractor φ ( x ) on the dataset Dbase to obtain generic feature embedding by optimizing the loss L ( φ ) : L ( φ ) = E { ( x , y ) } ∼Dbase ` ( { ( φ ( x ) , y ) } ) ( 1 ) where { ( x , y ) } is a sampled mini-batch of data points and ` ( · ) is usually an episodic few-shot loss ( Vinyals et al. , 2016 ) or a standard cross-entropy loss ( Chen et al. , 2020 ) . In the inference stage ( a.k.a . meta-test stage ) , a typical few-shot benchmark evaluates the model on K-way , N -shot classification tasks T drawn from Dnovel , where each task has a support set and a query set , i.e . T = ( T supp , T query ) . The support set T supp contains K classes and each class has N images ( e.g . K = 5 , N ∈ { 1 , 5 } ) . Following Snell et al . ( 2017 ) , the prediction ŷ′ of a query image x′ ∈ T query is given by the label of nearest prototype ck from T supp under a cosine similarity d ( · , · ) : ŷ′ = arg max k d ( φ ( x′ ) , ck ) , ck = 1 N ∑ ( x , y ) ∈T supp , y=k φ ( x ) . ( 2 ) An extended description of the few-shot learning framework can be found from Appendix A.1 . The generalization ability of the feature extractor φ ( x ) is improved in terms of training scheme ( e.g . episodic learning ( Vinyals et al. , 2016 ) ) , network design ( e.g . task condition ( Oreshkin et al. , 2018 ) ) or objective function ( e.g . learnable distance ( Sung et al. , 2018 ) ) . In our method , we propose a novel network design by inserting constellation models into CNNs and strengthen the intermediate features .

Summary: The paper proposes a constellation model that performs feature clustering and encoding dense part representations. The constellation module is placed after convolutional blocks. The module clusters cell features and calculates distance map between each cluster centroids and cell feature. The self-attention mechanism is applied on the distance map and concatenated to the original feature map to complement the feature representation. The resulting feature representation contains part representations. The few-shot experiments on the mini-Imagenet, CIFAR-FS, and FC100 datasets show the effectiveness of the proposed method.