Attentional Constellation Nets for Few-Shot Learning
Authors: Weijian Xu
yifan xu
Huaijin Wang
Zhuowen Tu
Venue: ICLR 2021 Poster (2021)
Keywords: few-shot learning
constellation models

Abstract: The success of deep convolutional neural networks builds on top of the learning of effective convolution operations, capturing a hierarchy of structured features via filtering, activation, and pooling. However, the explicit structured features, e.g. object parts, are not expressive in the existing CNN frameworks. In this paper, we tackle the few-shot learning problem and make an effort to enhance structured features by expanding CNNs with a constellation model, which performs cell feature clustering and encoding with a dense part representation; the relationships among the cell features are further modeled by an attention mechanism. With the additional constellation branch to increase the awareness of object parts, our method is able to attain the advantages of the CNNs while making the overall internal representations more robust in the few-shot learning setting. Our approach attains a significant improvement over the existing methods in few-shot learning on the CIFAR-FS, FC100, and mini-ImageNet benchmarks.

Published as a conference paper at ICLR 2021
ATTENTIONAL CONSTELLATION NETS FOR FEW-SHOT LEARNING
Weijian Xu*!, Yifan Xu"l, Huaijin Wang"| & Zhuowen Tu University of California San Diego', Amazon Web Services? (wex041, yix081, huw011, ztu) @ucsd.edu.
ABSTRACT
The success of deep convolutional neural networks builds on top of the learning of effective convolution operations, capturing a hierarchy of structured features via filtering, activation, and pooling. However, the explicit structured features, e.g- object parts, not expressive in the existing CNN frameworks. In this paper, we tackle the few-shot learning problem and make an effort to enhance structured features by expanding CNNs with a constellation model, which performs cell feature clustering and encoding with a dense part representation; the relationships among the cell features further modeled by attention mechanism. With the additional constellation branch to increase the awareness of object parts, our method is able to attain the advantages of the CNNs while making the overall internal representations more robust in the few-shot learning setting. Our approach attains a significant improvement over the existing methods in few-shot learning on the CIFAR-FS, FC100, and mini- Imageeet benchmarks.
INTRODUCTION
Tremendous progress has been made in both the development and the applications of the deep convolutional neural networks (CNNs) (Krizhevsky et al., 2012: Simonyan & Zisserman 2015 Szegedy et al. 2013 He et al. 2016 Xie et al. 2017). Visual lizatinzaiion of the internal CNN structure trained e.g. ImageNet (Deng al. 2009p has revealed the increasing level of semantic relevance for the learned convolution kernels/filters to the semantics of the object classes, displaying bar/edge like patterns in the early layers, object parts in the middle layers, and face/object like patterns in the higher layers (Zeiler& & Fergus 2014). In general, we consider the learned convolution kernels being somewhat implicit about the underlying objects since they represent projections/mappings for the input but without the explicit knowledge about the parts in terms of their numbers, distributions, and spatial configurations.
On the other hand, there has been a rich history about explicit object representations starting from deformable templates (Yuille et al.
1992). pictorial structure (Felzenszwalb & Huttenlocher] 22005),
constellation models (Weber et al.
2000
Fergus et al. 20033 Sudderth et al. 20055 Fei-Fei et al.
2006), and grammar-based model (Zhu & Mumford 2007). These past-based models (Weber et al.
2000 Felzenszwalb & Huttenlocher 2005 Fergus al. 2003 Sudderth et al. 2005 Zhu & Mumford
2007) share three common properties in the algorithm design: (I) unsupervised learning, (2) explicir
clustering to obtain the parts, and (3) modeling to characterize the spatial configuration of the parts. Compared to the CNN architectures, these methods are expressive with explicit part-based representation. They have pointed promising direction for object recognition, albeit lack of strong practice performance on the modern datasets. Another line of object recognition system with the part concept but trained discriminatively includes the discriminative trained part-based model (DPM) (Felzenszwalb et al. 2009) and the spatial pyramid matching method (SPM) (Lazebnik et al., 2006). In the context of deep learning, efforts exist bring the explicit part representation into deep hierarchical structures (Salakhutdinov et al. 2012).
The implicit and explicit feature representations could share mutual benefits, especially in few- shot learning where training data is scarce: CNNs may face difficulty in learning generalized representation due to lack sufficient training data, whereas clustering and dictionary learning
indicates equal contribution

---

Published as a conference paper at ICLR 2021
provide direct means for data abstraction. In general, end-to-end learning of both the implicit and explicit part-based representations is a viable and valuable means in machine learning. We view convolutional features an implicit part- based representation since they are learned through back- baco-propagation via filtering processes. On the other hand, an explicit representation can be attained by introducing feature clustering that captures the data abstraction/distriutioo under mixture model.
In this paper, we develop an end-to-end framework to combine the implicit and explicit part- bast- representations for the few-shot classification task by seamlessly integrating constellation models with convolution operations. In addition to keeping a standard CNN architecture, we also employ a cell feature clustering module encode the potential object parts. This procedure is similar to the dlustriigg/ooeebok learning for appearance in the constellation model (Weber et al., 2000). The cell feature clustering process generates a dense distance map. We further model the relations for the cells using a self-attention mechanism, resembling the spatial configuration design in the constellation model (Weber et al. 2000). Thus, we name our method constellation networks (ConstellationNNe)) We demonstrate the effectiveness of our approach on standard few-shot benchmarks, including FC100 (Oreshkin et al_ 2018), CIFAR-FS (Bertinetto et al. 2018) and mini-ImageNet (Vinyals et al. 2016) by showing significant improvement over the existing methods. An ablation study also demonstrates the effectiveness of ConstellationNet is not achieved by simply increasing the model complexity using e.g. more convolution channels or deeper and wider convolution layers (WRN-28-10 (Zagoruyko & Komodakis 2016)) (see ablation study in Table and Figure (e))
RELATED WORK
Few-Shot Learning. Recently, few-shot learning attracts much attention in the deep learning community (Snell et al. 2017 Lee et al. (209)). Current few-shot learning is typically formulated as a meta-leurning problem (Finn et al. 2017). in which an effective feature embedding is learned for generalization across novel tasks. We broadly divide the existing few-shot learning approaches into three categories: (1) Gradient basedd methods optimize feature embedding with gradient descent
during meta-test stage (Finn et al. 2017: Bertinetto et al. 20188 Lee al. 2019). (2) Metric-based methods learn a fixed optimal embedding with a distance-based prediction rule (Vinyals et al.) 2016)
Snell et al., (017). (3) Model-based methods obtains conditional feature embedding via a weight predictor MMishra et al.] 20177] Munkhdalai et al. (017).. Here we adopt ProtoNet (Snell et al.,
2017). a popular aetriccbbsee framework, in our approach and boost the generalization ability of the feature embeddings with explicit structured representations from the constellation model. Recently, Tokmakov al.) (2019) proposes a compositional regularization to the image with its attribute annotations, which is different from out unsupervised part-discovery strategy.
Part-Based constellatioo/iseeinnaaiveee Models. The constellation model family (Weber et al.
2000 Felzenszwalb & Huttenlocher] 2005) |Fergus et al.) [0003] (Sudderth et al. 2005 Fei-Fei et al.
2006
Zhu & | 2007) is mostly generative/expressive that shares two commonalities in the
representation: (I) clustering/codebook learning in the appearance and (2) modeling of the spatial configurations. The key difference among these approaches lies in how the spatial configuration is
modeled: Gaussian distributions (Weber et al. 2000); pictorial structure (Felzenszwalb & Hutten locher 2005); joint shape model (Fergus et al. 2003) hicrarchical graphical model (Sudderth et al.
2005); grammar-based (Zhu & Mumford, 2007 These constellation models represent promising direction for object recognition but are not practical competitive compared with deep learning based approaches. There are also discriminative models: The discriminatively trained part-based model (DPM) (Felzenszwalb et al. 2009) is a typical method in this vein where object parts (as HOG features (Dalal & Triggs] (005)) and their configurations (a star model) are learned jointly in a discriminative way. The spatial pyramid matching method (SPM) (Lazebnik et al. (006) has no explicit parts but instead builds on top of different levels of grids with codebook learned on top of the SIFT features (Lowe, 2004). DPM and SPM are of practical significance for object detection and recognition. In our approach, we implement the constellation model with cell feature clustering and attention- based cell relation modeling to demonstrate the appearance learning and spatial configuration respectively.
Parts models are extensively studied in fine-grained image classifications and object detection to provide spatial guidance for filtering uninformative object proposals (Simon & Rodner 2015 Peng et al. 2017; Zhu et al., 2017 Ge et al. 2019; Qi et al. 2019). Related to our work, Neural Activation Constellation
(NAC) (Simon & Rodner 2015) introduces the constellation model to perform unsupervised part model discovery with convolutional networks. Our work is different from
NAC in three aspects: (1) The algorithmic mechanisms behind Simon & Rodner 2015) and ours
2

---

Published as a conference paper at ICLR 2021
Chostering
Modeling
Feature
lla
Ros tlonal Encoding
Value
Concatonation
lla
Key
ReLLU
69
Softmax
Comv 3x3
Query
Convelutiral Mio
Distance Map
Dest Prosuct
(Gall Coce Mani)
Flementt whon Sum
008
X
Support
cee
0.15
X
Constel
Constell
Conv
Constell
Constall
Avg
Query
Embadrings
Illustration of our ConstellationNet pipeline where the bottom part is the network architecture based on Conv-4 backbone, and the top part shows the constellation model. Our proposed Constellation consists of "Constell." modules that perform explicit cell feature clustering with self-attention for joint relation modeling.
Figure 1:
different. /Simon& & Rodner (2015) implements a traditional Gaussian-based constellation module to model the spatial configuration and part selection top of fixed pre-trained CNN. However, in our ConstellatiomNet, our part representation and spatial configuration are modeled by cell feature clustering and self-attention based cell relation module, which general-purpose, modularized and recursive. (2) In [Simon & Rodner (2015) the constellation module is optimized in an EM-like algorithm, which is separate from the CNN optimization. Our constellation modules are seamlessly integrated into the current CNNs and jointly optimized with them. (3) Our ConstellationNet uses the dense cell features from the CNN feature maps, which considers all positions from the images potential parts and models their relation. However, (Simon al. 2015) extracts sparse part representations (i.c. it uses at most one part proposal per channel and selects even less parts later), which may not fully utilize the rich information from the CNN feature maps.
FEW-SHOT LEARNING
In a standard classification problem, we aim learn a model trained on the dataset Dhuse that can generalize its classification ability to unseen test set D belonging to same categories. In few-shot classification problem, we encourage Dhase and Dnovel be formed from different categories to emphasize model's 's generalization ability novel categories, where we denote training categories as Chase. test categories as Cravel, and Chase n Crovel = Ø to ensure the fairness.
In the training stage (a.k.a. meta-train stage), metric-based few-shot learning approaches (Snell et al. 2017 Vinyals et al. 2016 Oreshkin et al. 2018) usually learn a feature extractor d(x) on the dataset to obtain generic feature embedding by optimizing the loss L()):
((Φ) E2 ((x.a))) P.....(((( (((((xx) v)})
where (xx, y)} is sampled mini-batch of data points and É(-) is usually an episodic few-shot loss (Vinyals 2016) standard cross-entropy loss (Chen al. 2020).
In the inference stage (a.k.a. meta-tesí stage), a typical few-shot benchmark evaluates the model on K-way, N-shot classification tasks drawn from Dnovel where each task has support set and query set, i.c. T = upu Tquery). ). The support set Tuupp contains K classes and each class has N
images (e.g. K N E [1, 5)). Following (Sell et al. (2017). the prediction j' of query image x' E Trquery is given by the label of nearest prototype from 7 supe under cosine similarity d(-, :):
e' arg max di ((((x))x)))))))))))" Ct);
C₁
Σ
((x).
(2)
(x.))7TMH y=k
An extended description of the sho- learning framework can be found from Appendix A.. The generalization ability of the feature extractor ((x) is improved in terms of training scheme (e.g.
3

---

Published as a conference paper at ICLR 2021
episodic learning (Vinyals et al. 2016)), network design (e.g. task condition (Oreshkin al. 2018 or objective function (e.g. learnable distance (Sung et al. 2018)). In our method, we propose a novel network design by inserting constellation models into CNNs and strengthen the intermediate features.
CONSTELLATION MODEL
The concept of constellation has been introduced the few-shot learning scenario in early years (Fei-Fei et al. (006), in which the appearance and the shape are independently learned in a mixture model. In our work, we revisit the constellation model in an end-to-end learning framework: First, we define the cell feature the individual local feature at position in the feature map (see Figure D. We then employ cell feature clustering to model the underlying distribution of input cell features, implying part discovery procedure. We further obtain the distance map of the cell features from clustering and then perform cell relation modeling to build spatial relationships.
4.1 CELL FEATURE CLUSTERING
In convolutional neural networks (CNNs), the convolutional filters are learned to detect the dis- criminative patterns from low-level to high-level through back pack-propagation (Zeiler & Fergus 2014). In fact, the backward signal in the pack-propagation is not necessarily needed to obtain a pattern detector. With the feature map in the forward step of the CNN, are able to cluster the individual features at each location of the feature map (a.k.a. cell features) into multiple centers and employ the cluster centers as filters (Coates & Ng 2012 Krähenbühl et al., 2015). Assume we obtain convolutional feature map U with batch size B. spatial size H x W and channels C. We disensemble the feature map U Bxxxxxxxx into cell features set U {u,, U2, ..., un.) where n BHW and E RC is cell feature. Naively, we can conduct a k-means algorithm on input cell features 2 to solve the clustering objective:
min ΣΣ &
maillu; 12 s.t. mi E 10,
Em maR =1 I
(3)
where V [V1, .2.... }K} is set of cluster centers and my indicates if the input cell feature u, is assigned to cluster center Vk- The clustering- based filters can model the underlying cell feature distributions and capture the most frequent features, which can be explicitly interpreted as meaningful part patterns/part types. The hard assignment map m; = (ma ma2,- .... m&k) ) of input cell feature u, onto the cluster centers can be used as part-based representation, providing alternative information to the next layer in the CNN.
However, there are issues remaining unsolved in the naive design: Firstly, CNNs are typically optimized in stochastic gradient descent (SGD) manner. Thus, in each forward step, only mini- batch of images are proceeded provide cell features, which implies that the cluster centers cannot extract the global feature distribution across the whole dataset. Secondly, the hard assignment map has limited information due to its discrete representation. Therefore, inspired by [Sculley (2010), we design mini-batch soft k-means algorithm to cluster the cell features approximately:
Initialization. Randomly initialize global cluster centers V (V1, V2, .... vx} and a counter s ($1, 82.... sK) O.
Cluster Assignment. In forward step, given input cell features 2/ = {u, Ug un we compute the distance vector d, (d.1, ..dkk) between input cell feature u, and all cluster centers V. We then compute the soft assignment mak E R and generate the current mini-batch centers v'::
e Ada
Eg Emmau
de = uu 12
mik
Eee Adg
Σ, mik
()
where 3 > is an inverse temperature. Centroid Movement. We formulate a count update As = E, m, by summing all assignment maps m, (M,,, m.2....T.kk. The current mini-batch centers v2 are then updated to the global centers V2 with a momentum coefficient ry:
X
Vi (1 v) nVV.
(5)
Sk. Ask
Counter Update. Counter s is updated and distance vectors (d.) are reshaped and returned:
$ S+a 144A. As
4

---

Published as a conference paper at ICLR 2021
With gradually updating global cluster centers, the above algorithm is able address the issue of limited data in a mini-batch. In addition, we reshape the distance vectors {d,} } of all input cell features to distance map D R x2***x*x K Each distance vector d, can be seen codehook (dictionary) learning, which encodes a soft assignment of the visual word (i.e. cell feature) onto the codewords (i.e. cluster centers) and implies a part representation. The distance map D then can be viewed as a cell code map that represents spatial distribution of identified parts, which is passed following layers. Empirically, it is observed that when u; and Vę are La normalized, the training procedure is more stable and the Euclidean distance dije is equivalent to cosine similarity up to an affine transformation, Details of the cell feature clustering can be found in Appendix (A.9
learned cell code in
4.2 CELL RELATION AND SPATIAL CONFIGURATION MODELING
Before the deep learning era, traditional constellation models (Fei-Fei et al. 2006) decompose visual information into appearance and shape representation. The appearance of different parts in the image is treated independently while the shape of parts is assumed to have spatial connections. In our constellation model, we establish the spatial relationship among the individual part-based representations at different location from the distance map as well. Specifically, we apply the self-attention mechanism (Vaswani et al. (017) to build the spatial relationship and enhance the representation instead of using probabilistic graphical models in prior work (Fei-Fei et al. 2006).
In cell relation modeling, add positional encoding P E BxxxxXXC following Carion et al. (2020) for spatial locations to the distance map D and obtain the input feature map F1 for query and key layers. For value layer, we directly flatten the distance map D another input feature map F
Fi SpatialFlatten(D + PP Bxx*W*K, F SpatialFlatten(D)
BBxHWxK (7)
The input feature maps F,, Ff are transformed into query, key and value {F*, F*, F"} C BBXHW*K by three linear layers (W9, W W C R and further computes the output feature FA:
F F* F FF F,W F
(8)
F = Att(F", FR, F") softmax
F"
(9)
VK
The softmax of dot product between query and key matrix ( E RRBBHWWXE HHW calculates the similarity scores in the embedding space among features across the spatial dimension. This encodes the spatial relationships of input features and leads to an enhanced output feature representation FA. Besides, VK in the denominator is to stabilize the gradient. In practice, we adopt a multi-head attention to model the feature relation in the embedding subspaces:
F MIiaa MultiHeadAtt(F", Ft F*) FF,,..,F,)]W
F₁- -At(FF) Au,FF).E)F))
(10)
In head attention, the aforementioned similarity scores in the K' K dimensional embedding subspace are calculated using the query, key and value from j-th head, i.e. (F), F Fj} C R Bx HWW K" The output features F, of each head are computed following Eq. 9 All the output features (Fr I, .... F {r} are concatenated back into K dimension embedding and further processed with a linear layer W E R x to generate multi-head output features FMHA- Such multi-head attention settings could provide more diverse feature relation without introducing extra parameters.
4.3 ANTEGRATE CONSTELLATION MODEL WITH CNNS
Our constellation model has the capability to capture explicit structured features and encodes spatial relations among the cell features. The output features yield informative visual cues which are able to strengthen the convolutional features. Thus, as shown Figure T we place the constellation model after the convolution operation to extract its unique explicit features and concatenate them with the original convolutional feature map. A following 1 x convolutional layer is used on the concatenated features to restore the channels of convolutional feature map. In Table 3 we provide evidence that merging features from constellation model to the CNN backbone can significantly improve the representation ability. In contrast, increasing channels in CNNs alone to double the parameters (second row in Table can only improve the performance marginally. Optionally, we found it is useful to adopt auxiliary loss when training the constellation model in deeper networks (e.g. ResNet- 12). On top of each constellation model, we conduct standard classification to acquire additional regularization.
5

---

Published as a conference paper at ICLR 2021
4.4 WHY CLUSTERING AND
(CLUSTERING MAP POSITIONAL ENCODING)?
As described in Section 1 and 2 classical constellation models (Fergus al., 2003 Felzenszwalb & Huttenlocher 2005) extract parts with their spatial relationships; they are expressive but do not produce competitive results on modern image benchmarks. CNN models (Krizhevsky et 2012 He
et al. 2016) attain remarkable results on large-scale image benchmarks (Deng et al. 2009) but they are limited when training data is scarce. We take the inspiration from the traditional constellation models, but with realization that overcomes their previous modeling limitations.
The main contribution of our work a constellation module/block that performs cell-wise clustering. followed by self-attention on the clustering distance map + positional encoding. This separates our work from previous attempts, e.g. non-local block work [Wang et al. 2018) in which long-range non-linear averaging is performed on the convolution features (no clustering, nor positional encoding for the spatial configuration). The main properties of our constellation block include: (1) Cell based dense representation as opposed to the sparse part representation in (Weber et al. 2000) to make the cells recursively modeled in the self-attention unit a modularized and purepose way. (2) Clustering generate the cell code after clustering (codebook learning) that attains abstraction and is not dependent on the CNN feature dimensions. (3) Positional encoding (as in Carion et al.) (2220) for cells to encode the spatial locations. (4) Tokenized representation as expressive parts (code/clustering distance map + positional encoding) for the cells. (5) Self-attention to jointly model the cell code and positional encoding to capture the relationships between the parts together with their spatial configurations.
EXPERIMENT
5.1 DATASETS
We adopt three standard benchmark datasets that are widely used in few-shot learning, CIFAR-FS dataset (Bertinetto al. 2018), FC100 dataset (Oreshkin et al., 2008)) and mini-ImageNet dataset
(Vinyals et al.) 2016). Details about dataset settings in few-shot learning are in Appendix AA2 A.2
5.2 NETWORK WITH
We build ConstellationNet on two ProtoNet variants, namely Conv-4 and ResNet-12, which are commonly used in few-shot learning. Details of networks and the optimization are in Appendix.
We develop a new technique, Multi-Branch, to optimize standard classification loss and prototypical loss simultaneously. We find the two training schemes, standard classification scheme and prototypical scheme, can be a companion rather than a conflict. Details of these two schemes can be found from Appendix A.I Different from standard network backbone used in prior works, our embedding ((x) is separated into two branches after a shared stem (Y-shape). Details of our multi-branch design are elaborated in A.IO The detailed ablation study is described in Table
Feature Augmentation. During the meta-testing stage, we discover that concatenating features before average pooling to the final output can improve classification accuracy. The advantage of this technique is that no additional training and model parameters are introduced.
5.3 RESULTS ON STANDARD BENCHMARKS
Table 1 and summarize the results of the few-shot classification tasks on CIFAR-FS, FC100, and mini-ImageNNe. respectively. Our method shows a notable improvement over several strong baselines in various settings. Constellation significantly improves the performance on shallow networks (Conv-4). In Table our model outperforms SIB (Hu et al. 2020) 1-shot by 0.6% and 5-shot by 5.6%. In Table [ our model outperforms MetaOptNet (Lee al. 2019) by 5.95% in 1-shot and 6.24% in 5-shot. For deep networks with rich features, the constellation module still contributes to the performance, showing its complementary advantage to convolution. Our ResNet- 12 model beats (Lee et al. 2019) 1-shot result by 2.7% .7% on FC100, 3.4% on CIFAR-FS, and 1. 1.72% on mini-ImageNNer The consistent improvement over both shallow and deep networks across all three datasets shows the generality of our method. Our ConstellationNet is orthogonal to the margin loss based methods (Liu et al. 2020 Li et al. 2020), and we also do not use extra cross-modee information (Xing et al.. 2019: Li et al. 2020). On the contrary, our model enhances the embedding generalization ability by incorporating its own bart-based representation. Additionally, to verify the orthogonality of our method, we adapt the negative margin loss following Liu al. 20200 to Cony-4 models in

---

Published as a conference paper at ICLR 2021
Table I: Comparison to prior work on mini--mageNet. Average 5-way classification accuracies (%) on mini-ImageNet meta-test split are reported with 95% confidence intervals. Results of prior works are adopted from Lee et al. (2019) and original papers.
used extra cross cross-modal information.
mini-ImageNet 5-way
Model
Backhone
1-shot
5-shot
Meta- Meta-Leaming LSTM (Ravi & Larochelle 2016]
Conv-4
43.44 + 0.77
60,60 + 0.71
Matching Networks (Vinyals al, 2016
Conv-4
43,56 + 0.84
55.31 0.73
Prototypical Networks (Seell
Conv-4
49,42 +: 0.78
68.20 0.66
Transductive Prop Nets [L.u
2018
Conv-4
55.51 4: 0,86
69.86 1: 0.65
MetaOptNet [Lc et 2019
Conv-4
52.87 + 0.57
68.76 0.48
Negative Margin cl 2020
Conv-4
52.84 +: 0.76
70.41 0.66
ConstellationNet (ours)
Cony-4
58.82 +: 0.23
75.00 + 0.18
SNAIL. Mishra et al. 2018
ResNet- 12
55.71 +: 0.99
68.88 0.92
TADAM (Drsshknn
ResNet-12
$8.50 +:
76.70 0.30
TapNet Yoon el aF 2019
ResNet- 12
61.65 + 0.15
76.36 + 0.10
Variational FSL Iing
2019
ResNet-12
61.23 0.26
77.69 0.17
MetaOptNNe (Lee
2019
ResNet-12
62.64 0.61
78.63 0.46
CAN (Hou et a. 2019
ResNet-12
63.85 +: 0.48
79.44 0.34
SLA-AG al. 2020
ResNet-12
62.93 0.63
79.63 0.47
Meta- Baseline Thch
2020
ResNet-12
63.17 +: 0.23
79.26 0.17
AM3 Xing et al. 20191
ResNet-12
65.21 + 0.30
75.20 0.27
ProtoNats+ TRANI (Lá et
2020
ResNet-12
60.31 +: 0.48
77.94 0.57
AM3 + TRAML al 2020
ResNet-12
67.10 + 0.52
79.54 0.60
Negutive Margin
Lu 2020
ResNet-12
63.85 0.81
81.57 0.56
Constellation Net (ours)
ResNet-12
64.89 + 0.23
79.95 + 0.17
Table 2: Comparison to prior work on FC100 and CIFAR-FS. Average 5-y classification accuracies (%) on CIFAR-FS and FC100 meta-test split are reported with 95% confidence intervals. Results of prior works are adopted from Lee et al. (2019) and original papers.
CIFAR-FS 5-way
FC100 5-way
Model
Backhone
1-shot
5-shot
1-shot
5-shot
MAML (Finn et al. 2017
Conv-4
58.9 1 1.9
71.5 = 1.0
Prototypical Networks Shell et 2017
Conv-4
55.5
0.6
Relation Networks (Sung el al 2018
Cony-4
55.0 1.0
69.3 0.8
R2D2 Bertinetto et al 2018
Conv-4
65.3 0.2
79.4
2020
Conv-4
68.7
0.4
Constellation (aurs)
Cony-4
69.3 +: 0.3
82.7 + 0.2
Prototypical Networks (Snell
ResNet-12
72.2 0.7
83.5 : 0.5
37.5
52.5
TADAM (Oreshkin al. 201
ResNet-12
40.1 0.4
1.4
MetaOptNet (Lee et al. 2019
ResNet-12
72.6
84.3 + 0.5
40.5 0.6
MetaOpiNettSVMa [Lee
ResNet-12
0.7
84.2 0.5
41.1
0.6
1.6
ConstellationNet (ours)
ResNet-12
75.4 0.2
86.8 = 0.2
43.8 + 0.2
89.7 # 0.2
Appendix A.8
We observe ConstellationNet with negative margin brings 0.52% improvement to ConstellationNet, and obtains 6.93% gain compared with baseline on ImageNet.
MODEL ANALYSIS
6.1 ARCHITECTURE ALTERNATIVES
In Table first study the role of each module in ConstellationNet, where the number of parameters controlled approximately equivalent to the baseline's ' size. Our constellation model brings 6.41 % and 2.59% improvements over baseline on 1-shot Conv-4 and ResNet-12 results. Combined with our multi-branch training procedure, the model further improves additional 1.34% and 1.26% on 1-shot Conv-4 and ResNet-12 12, respectively. Finally, feature augmentation from penultimate layer final output embedding brings additional 0.45% and 0.27% improvements on two variants.
We also test the baseline model with extra channels in the Table 3 The new model only shows slight improvements over original baseline, and is outperformed by our ConstellationNet with large margin. We also obtain WRN-28-10 10 baseline results to validate our improvement. While making ResNet baselines deeper and wider, our ConstellationNet still outperforms this strong baseline. In ( we further study whether the performance gap between ConstellationNet and baseline can be reduced by simply altering the baseline's 's model complexity using e,g. more convolution channels. Although the trend of baseline accuracy increases when increasing the model parameter number gradually, the performance gap is still significant. This validates our concept that modeling hicrarchical part structures can greatly benefit features learned from convolution operation, and obtain more robust feature representation. In addition, applying self-attention the distance map (6-th
7

---

Published as a conference paper at ICLR 2021
Table 3: Effectiveness of modules. Average classification accuracies (%) on mini- ImageNee meta-test split. We compare our ConstellationNet with alternative architectures including the baseline and the modified haseline with extra channels based on Conv-4 and ResNet- 12. We also include a baseline with WideResNet-28- 10 Zagoruyko & Komodakis 2016) backbone for comparison.
Haseline
Fenture
Cal
Refation
Cell
Muhi
Peature
Extra
Canel
Res-e1112
Clustering
Modsligg
Brunch
Augincn
Choanels
Concolution
Con-4iBle-112
1-bbtt
S-tbok
I-shot
5-sbot
117K/4.0M
50 0:2.3
68:40 0.19
60.77 0:00
78.76 =: 0.17
3228/166
=0.18
61.45
0.16
184829174 7M
62.75
79.21 =00.7
192878 AM
53.46 0.23
72.52 0.18
61.54 0:034
76.51 2.1.5
200K-20-00 -M
57403
0:23
74:09:2000 0:11
0:.36-2:1:33
79 2:
300K.2
58.37
0:23
74.53 0.0.1
64.02
19.00 1.11
200K/9.4M
0.23
0.08
64.89
WRN
WalcRonNNet-8-10
0.23
row: 57.03% on Conv-4, I-shot) achieves better performance than directly applying it to the original cell features (i.e. convolutional feature map) (4-th row: 55.92% on Conv-4, 1-shot). We also tried to replace the cell feature clustering module with a Ixl convolution layer (output dimension is equal to the number of clusters) (5-th row: 55.4 55.46% on Conv-4, 1-shot). It is worse than our results (6-th row) as well. We observe that the 1x1 convolution layer less expressive than the cell feature clustering module, making it difficult to extract enough context information during cell relation modeling.
6.2 MODULES ANALYSIS
1965
1929 1255 12.8
5077 5200 55.4
(700
19.00
1250 Pol poer InBIAA Cansaiaaion
199c Nurooo
140 Martow Clutess
211 Hammitee
(2) DnsiONA CondeAttio
Modules analysis. (a, b, d) We study the effectiveness of changing the number of clusters, the number of heads in attention layer, and the layer indices with constellation based Conv-4, (e) We demonstrate the performance gain our ConstellationNet is unmatched by increasing model complexity our baselines. All experiments are done on mini ImageNet.
Figure 2:
In Figure 2 (a), we vary the number of clusters adapted in all layers to observe the performance change. We found that increasing the number of clusters improves the aceuracy in general, and set clusters to 64 is optimal in terms of both model size and classification performance. Figure 2 (b) shows the number of attention heads does not effect performance as much as the number of cluster, and 8-head attention obtains 1.80% performance gain on the 1-shot setting compared to I-head attention. In Figure (c. d), we also study the effectiveness of clustering algorithm applied to different layers. The results show both early features and high-level features benefit from introducing clusters algorithm into the original CNN architecture.
6.3 VISUAL.ZZATION
Figure demonstrates the visualization of cluster centers in each layer of Conv-4 model mini- ImageNet. In the upper part of the figure, each image shows patches corresponding to the nearest cell features cluster center (i.e. with lowest Euclidean distance). It is observed that clusters in early layers (e.g. layer 1,2) represent simple low-level patterns while the clusters in high layers (e.g. layer 3,4) indicate more complex structures and parts. In the lower part of the figure, we choose two cluster centers from layer 4 for further interpretation: The left one with green box could possibly represent legs since it consists of various types of legs from human, dog and other animals. The right one with the red box shows most nearest cell features to this cluster center are parts with bird's 's head or beetles, which share dotted structure (i.e. black dots on beetles / eyes on bird's head).
The left side of Figure 4 shows the visualization of cell features that are assigned to different clusters. For each image, we extract the assignment maps corresponding to three cluster centers generated in the last constellation module of Conv-4 and find multiple cell features with the highest assignments within each assignment map. The locations of cell features are projected back in the original image space, marked by three different colors of "." in the raw image to show three different feature clusters. For given class of images, the same cluster centers are selected for comparison across 6 samples. As shown in Figure 4 we observe part information of each class is explicitly discovered. For the bird
8

---

Published as a conference paper at ICLR 2021
Layer
Layer
Layer
Layer
logs
Bertles
Unicycle wheels legs)
Dog's legs
Bird's head
Other legs
Figure 3: Visualization of cluster centers. (Upper) We visualize four cluster centers in each layer by showing patches associated with cell features that have the nearest distance the clustering center. (Lower) Identifying parts from two cluster centers layer 4: Left one with green box represents various types legs. Right one with red box mostly shows beetles and bird's head, sharing a dotted structure.
cells assignment and attention maps. (Left) Each color represents a cluster, and each point, marked as "." represents a cell assigned to cluster center. We demonstrate samples for each class (bird, dog and tank). (Right) We visualize attention maps of one query feature (at the location of red point in left part) with all key features. The middle part shows the attention maps corresponding to heads in the multi-head attention. The right part shows overlapped map of all attention maps.
Figure 4: Visualization
see different parts in each image, including head (cyan "."), body (purple "-") and tail (yellow ":"). For the dog category, we see parts including heads (red "."), legs (green ".") and body (blue "."). For the tank category, we see parts like track (light blue ".") and turret (pink ".").
category,
The right side of Figure 4 visualizes the attention maps in the cell relation model. We use the last constellation module the ResNet- 12 model for visualization since it captures high-level features that better represent parts. We choose one query feature at the center of the object and show its attention map to all key features. The middle part of the figure shows the attention maps corresponding heads in the multi-heat attention. It is observed that some parts are identified such as head (second map in first row), legs (first two map second row), buttock (first map in first row) and body (second map in the second row). A merged attention map by overlaying all 8 attention maps is presented at figure. indicates that all the attention heads together can extract the features whole object, which would be useful for final classification.
right part
CONCLUSION
In this paper, we present ConstellationNet by introducing an explicit feature clustering procedure with relation learning via self-attention. We implement mini-batch soft k-means algorithm capture the cell feature distribution. With integrated implicit (standard CNN modules) and explicit (cell feature clustering cell relation modeling) representations, our proposed ConstellationNet achieves significant improvement over the competing methods on few-shot classification benchmarks.
ACKNOWLEDGMENTS
This work funded by NSF I1S-1618477 and NSF IIS 1717431. We thank Qualcomm Inc. for an award support. We thank Kwonjoon Lee, Tiange Luo and Hao Su for valuable feedhacks.

---

Published as a conference paper at ICLR 2021
REFERENCES
Luca Bertinetto, Joao F Henriques, Philip HS Torr, and Andrea Vedaldi. Meta-learning with differen- tiable closed-form solvers. arXiv preprint arXiv: 1805.888366 08136, 2018.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
Yinbo Chen, Xiaolong Wang, Zhuang Liu, Huijuan Xu, and Trevor Darrell. new meta-baseline for few-shot learning. arXiv preprint 20Xiv:003.003390 2020.
Adam Coates and Andrew Y Ng. Learning feature representations with k-means, In Neural networks: Tricks of the Irade, pp. 561-580. Springer, 2012.
Navneet Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In CVPR, 2005.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pp. 248-2552 leee, 2009.
Li Fei-Fei, Rob Fergus, and Pietro Perona.
One-shot learning of object categories. IEEE transactions on pattern analysis and machine intelligence, 28(4):594-611, 2006.
Pedro F Felzenszwalb and Daniel P Huttenlocher. Pictorial structures for object recognition. Interna- tional journal of computer vision, 61(1):55-79, 2005.
Pedro F Felzenszwalb, Ross B Girshick, David McAllester, and Deva Ramanan. Object detection with discriminatively trained part-based models. IEEE transactions on pattern analysis and machine intelligence, 32(9):1627-1645, 2009.
Robert Fergus, Pietro Perona, and Andrew Zisserman. Object class recognition by unsupervised scale-invariant learning. In CVPR, 2003.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. agdel-ag meta-learning for fast adaptation of deep networks. arXiv preprint arXiv: (003:/703.00330 0300003400 2017.
Weifeng Ge, Xiangru Lin, and Yizhou Yu. Weakly supervised complementary parts models for fine-grained image classification from the bottom up. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 3034-3043, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
Geoffrey E Hinton, Sara Sabour, and Nicholas Frosst. Matrix capsules with em routing. 2018.
Ruibing Hou, Hong Chang, MA Bingpeng, Shiguang Shan, and Xilin Chen. Cross attention network for sho- classification. Advances in Neural Information Processing Systems, pp. 4005 2019.
Shell Xu Hu, Pablo G Moreno, Yang Xiao, Xi Shen, Guillaume Obozinski, Neil D Lawrence, and Andreas Damianou. Empirical bayes transductive meta-learning with synthetic gradients. arXiv preprint arXiv:2004. 12666 2020.
Adam Kosiorek, Sara Sabour, Yee Whye Teh, and Geoffrey E Hinton. Stacked capsule autoencoders. In Advances Neural Information Processing Systems, pp. 15486-15496, 2019.
Philipp Krähenbühl, Carl Doersch, Jeff Donahue, and Trevor Darrell. Data-dependent initializations of convolutional neural networks. arXiv preprint arXiv:/511.0663 2015.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, 2012. 2.
10

---

Published as a conference paper at ICLR 2021
Svetlana Lazebnik, Cordelia Schmid, and Jean Ponce. Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In CVPR, 2006.
Hankook Lee, Sung Ju Hwang, and Jinwoo Shin. Self-supervised label augmentation via input transformations. 37 International Conference on Machine Learning, Vienna, Austria, PMLR 119, 2020, 2020.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 10657-10665, 2019.
Aoxue Li, Weiran Huang, Xu Lan, Jiashi Feng, Zhenguo Li, and Liwei Wang, Boosting few-shot learning with adaptive margin loss. Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 12576-12584, 2020.
Bin Liu, Yue Cao, Yutong Lin, Qi Li, Zheng Zhang, Mingsheng Long, and Han Hu. Negative margin matters: Understanding margin in few-shot classification. arXiv preprint arXiv:2003. 2020.
Yanbin Liu, Juho Lee, Minseop Park, Saehoon Kim, Eunho Yang, Sung Ju Hwang, and Yi Yang. Learning to propagate labels: Transductive propagation network for few-shot learning. arXiv preprint arviv: 1805. 10002, 2018.
David G Lowe. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 60(2):91-110, 2004.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. arXiv preprini arXiv: 2017.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. 2018.
Tsendsuren Munkhdalai, Xingdi Yuan, Soroush Mehri, and Adam Trischler. Rapid adaptation with conditionally shifted neurons. arXiv preprint arXiv: 1712.0996. 2017.
Boris Oreshkin, Alexandre Lacoste, and Pau Rodriguez. Tadam: Task dependent adaptive metric for improved few-shot learning. arXiv preprint frXiv:/805.10023. 2018.
Yuxin Peng, Xiangteng He, and Junjie Zhao. Object-part attention model for fine-grained image classification. IEEE Transactions on Image Processing, 27(3):1487-1555 2017.
Lei Qi, Xiaoqiang Lu, and Xuelong Li. Exploiting spatial relation for fine-grained image classification. Pattern Recognition, 91:47-55, 2019.
Sachin Ravi and Hugo Larochelle. Optimization model for few-shot learning. 2016.
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. In Advances in neural information processing systems, pp. 3856-3866, 2017.
Ruslan Salakhutdinov, Joshua Tenenbaum, and Antonio Torralba. Learning with hierarchical-deep models. IEEE transactions on pattern analysis and machine intelligence, 35(8): 1971, 2012.
David Sculley. Web-scale k- k-means clustering. In Proceedings of the 19th international conference on World wide web, pp. 1177-1178, 2010.
Marcel Simon and Erik Rodner. Neural activation constellations: Unsupervised part model discovery with convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 143-1151, 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, pp. 4077-4087, 2017.
11

---

Published as a conference paper at ICLR 2021
Erik B Sudderth, Antonio Torralba, William T Freeman, and Alan S Willsky, Learning hierarchical models of scenes, objects, and parts. In ICCV, volume 2, 2005.
Flood Sung, Yongxin Yang. Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1999-1008, 2018.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
Pavel Tokmakov, Yu-Xiong Wang, and Martial Hebert. Learning compositional representations for few-shot recognition. In Proceedings of the IEEE International Conference on Computer Vision, pp. 6372-6381, 2019.
Yao-Hung Hubert Tsai, Nitish Srivastava, Hanlin Goh, and Ruslan Salakhutdinov. Capsules with inverted dot-product attention routing. arXiv preprint arXiv:2002.0476. 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, pp. 3630-3638, 2016.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7794-7803, 2018.
Markus Weber, Max Welling, and Pietro Perona. Unsupervised learning of models for recognition. In ECCV, 2000.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In CVPR, 2017.
Chen Xing, Negar Rostamzadeh, Boris Oreshkin, and Pedro O O Pinheiro. Adaptive cross-modal few-shot learning. Advances in Neural Information Processing Systems, 32:4847-4857 2019.
Sung Whan Yoon, Jun Seo, and Jaekyun Moon. Tapnet: Neural network augmented with task aaaativeetive projection for few-shot learning. arXiv preprint 1iXi::1905006549 2019.
Alan Yuille, Peter W Hallinan, and David S Cohen. Feature extraction from faces using deformable templates. International journal of computer vision, 8(2):99-111, 1992.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv: 2016.
Matthew Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In ECCV, 2014.
Jian Zhang, Chenglong Zhao, Bingbing Ni, Minghao Xu, and Xiaokang Yang. Variational few-shot learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1685-1694, 2019.
Long-Chun Zhu and David Mumford. A stochastic grammar of images. Now Publishers Inc, 2007.
Yousong Zhu, Chaoyang Zhao, Jinqiao Wang, Xu Zhao, Yi Wu, and Hanqing Lu. Couplenet: Coupling global structure with local parts for object detection. In Proceedings of the IEEE international conference on computer vision, pp. 41 4126-4134, 2017.
12

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 FEW-SHOT LEARNING FRAMEWORK
In this section, we introduce background concepts of meta-learning and elaborate the few-shot learning framework used in our ConstellationNet.
Meta-Learning in Few-Shot Classification. Current few-shot learning is typically formulated as meta-learning task (Finn et al. 2017), in which dataset D is used to provide commonsense knowledge and dataset D for the few-shot classification. Dhuse has the classes Coase which are disjoint from the Caovel in D to ensure fairesss. There two stages, meta-training and meta-test, in the meta-learning framework: In meta-training stage, we attempt to train model to learn generic features from D meta-test stage, we adapt the model on the limited training split from D and evaluate the performance of the model on the test split.
ProtoNet-Basse Framework. In our ConstellationNet, we adopt ProtoNet (Snell et al. 2017) as the base few-shot learning framework. In ProtoNet, the dataset D is represented by series of K-way N-shot tasks {7} where each task consists of support set and a query set, i.e. T (Tsupp, 7. The support
7uupp contains K classes and each class has N examples from the training split of Duowel which are used to adapt the model in meta-test stage. The query set Tquay from the test split of Dnovel is then used to evaluate the model.
The ProtoNet attempts to learn a generic feature extractor @(x) on image x, and represent a class k by the prototype Cg. which is the average feature of examples from support set 7 with this class:
((x)
(11)
T y
During the meta-test stage, we use the prototypes to compute the probability pk of a query example x' E queery class ls and predict its label y/:
exp(d(x', cx))
P& p(y k|x',, TTuPp) Ex (o(d(x''c))))
y' arg max Pk-
(12)
where d(...) is a cosine similarity function (different from the Euclidean distance in Snell al. (2017).
During the meta-training stage, there are two different training schemes: The prototypical scheme from ProtoNet uses an episodic learning strategy that also formulates the dataset Dhase as a series of tasks (T). The negative log-likelihood loss (())) is optimized:
((prup T Exx' Tow log ply 1 T
(13)
(( (T (wm) l (T Truuery.
(14)
Another way is the standard classification scheme CCen et al 2020): It simply uses Dhnse as a standard classification dataset {(x, v)) consisting of Q classes in total. Thus, a cross-entropy loss L()) is optimized:
(())= = (xx,))) - log cxp(wy o(x))
(15)
Σ, exp(w, ((x))
where w,, is the linear weight for class q. In our ConstellationNet, scheme at default. For the experiment with multi-branch network, we use the prototypical scheme and standard classification scheme for separate branches.
the standard classification
A.2 DATASETS
The CIFAR-FS dataset (Bertinetto et al. 2018)) is a few-shot classification benchmark containing 100 classes from CIFAR-IOU 100 (Krizhevsky et al. 2009). The classes are randomly split into 64, 16 and 20 classes as meta-training, meta-validation and meta-testing set respectively. For each class, it
13

---

Published as a conference paper at ICLR 2021
contains 600 images of size 32 x 32. We adopt the split from Lee al. (2019). The FC100 dataset (Oreshkin et al., 2018) is another benchmark based on CIFAR- 100 where classes are grouped into 20 superclasses to void the overlap between the splits. The mini- mini-ImageNee dataset (Vinyals et al.) 2016) is a common benchmark for few-shot classification containing 100 classes from ILSVRC- 2012 Deng al., (009). The classes randomly split into 64, 16 and 20 classes meta-training, meta-validation and meta-testing set respectively. For each class, it contains 600 images of size 84 84. We follow the commonly-used split in Ravi & Larochelle (2016), Lee al.) (2019) and Chen et al. (2020). In all experiments. we conduct data augmentation for the meta training set of all datasets match Lee et al. (2019)'s 's implementation.
A.3 NETWORK BACKBONE
Conv-4. Following Lee al. (019)) we adopt the same network with 4 convolutional blocks. Each of the blocks has a x3 convolutional layer, batch normalization layer, a ReLU activation and 2x2 max-pooling layer sequentially. The numbers of filters are 64 for all 4 convolutional layers.
ResNet-12 12. Following Chen et al. (2020), we construct the residual block with 3 consecutive convolutional blocks followed by an addition average pooling layer where each convolutional block has a 3 convolutional layer, batch normalization layer, leaky ReLU activation, and poooloooing layers. The ResNet-12 12 network has residual blocks with each filter size set to 64, 128, 256, 512, respectively.
WRN-28-10. WideResNet expands the residual blocks by increasing the convolutional channels and layers Zagoruyko & Komodakis 2016). WRN-28-10 uses 28 convolutional layers with widening factor of 10.
A.4 CONSTELLATION MODULE CONFIGURATION
To achieve the best performance with constellation modules, we do not always fully enable them after all the convolutional layers. For Conv-4, we use constellation modules after all four convolutional layers, but the cell relation modeling module is disabled in first two constellation modules due to the high memory consumption. For ResNet-12, convolutional layer 1,7,8,9 and disable the relation modeling module in the first constellation module. We the deep supervision in 1e to stablize the training of constellation modules.
enable the constellation modules after the
A.5 SELF-AFTENTION SETTINGS
We follow the common practice in Vaswani et al. (2017) set the attention layer with residual connections, dropout and layer normalization. The sine positional encoding follows settings in Carion
al. 2020).
A.6 TRAINING DETAILS
Optimization Settings. We follow implementation Lee al. (2019), and use SGD optimizer with initial learning rate of 1, and set momentum to 0.9 and weight decay rate to 5 x 10 44 The learning rate reduces to 0.06, 0.012, and 0.0024 at epoch 20, 40 and 50. The inverse temperature 100.0 in the cluster assignment step, and À is set to 1.0 in the centroid movement step.
set to
A.7 ABLATION STUDY ON THE NUMBER OF CLUSTERS
Table 4 studies the number of clusters needed for random and similar classes. The result shows the optimal number of clusters are less affected by the number of clusters but more affected by the similarity between classes. Less number of clusters are needed for dataset with classes of high similarity, which aligns with our intuition, limited number of patterns exist in this dataset so that small number of clusters are enough to represent its bast-based information.
FC100 training dataset consists of 60 classes that are grouped evenly into 12 superclasses. In the random classes group, the training dataset includes 6 randomly selected super-classes (i.e., 30 classes) and models are trained with 8, 16, 32, 64 and 128 number of clusters. The highest accuracy occurs 16 clusters (1-shot: 39.12% in ResNet- 12). In the similar classes group, 30 classes randomly
14

---

Published as a conference paper at ICLR 2021
Table 4: Ablation study on the number of clusters for random and similar classes. We investigate how similaritics of images the training dataset affect optimal number of clusters. The first group experiments use training dataset with 30 similar classes while the second group use 30 random classes from FC dataset, all of which performed on ResNet- 12 with Constellation module.
Similar Classes
Random Classes
Clusters
5-shot
1-shot
5-shot
38.9 1 0.2
$2.8 1 0.2
40.9 1 0.2
54,5
0.2
39.1 + 0.2
51.8 +:
40.9 +
54,9
38.7 = 0.2
52.3 0.2
40.9 -
=
38.8 0.2
52.3
41.2
54.9
128
38.8 0.2
52.1 - 0.2
40.8 +
54.7 0.2
sampled from the original training dataset and we repeat the same experiments as above. The highest accuracy oceurs at 64 clusters (1-shot: 41. 41.22% ResNet- which is much more than the 16 clusters used for images from similar classes.
A.8 ADDITIONAL EXPERIMENTS WITH NEGATIVE MARGIN
Table 5: Additional experiments with the use of negative margin, Average classification accuracies (%) on mini-ImageNer meta-test split. We compare our ConstellationNer and baseline with and without the negative margin loss based Conv-4.
Cell Feature
Cell Relation Modeling
Negative
Conv-4
Baseline
Clustering
Margin
1-shot
5-shot
50.62 = 0.23
68.40 + 0.19
51.42 = 0.23
68.84 + 0.19
57.03 1 0.23
74.09 1 0.18
57.55 0.23
74.49 + 0.18
Table studies the use of negative margin loss (Liu et al. 2020) our Conv-4 models. In the negative margin loss, we use the inner-prote unner-product similarity, the temperature coefficient 1.0 and the negative margin
005,, which attains the best performance improvement Besides, we do not have the fine-tune step during meta-test. Our baseline with the negative margin loss obtains 0.80% improvement 1-shot and 0.44% improvement 5-shot compared with the baseline. Similarly, our ConstellationNet with the negative margin loss achieves 0.52% improvement on 1-shot and 0.40% improvement 5-shot. The consistent improvement of negative margin loss the baseline and our ConstellationNet indicates that our constellation module is orthogonal to the negative margin loss, and both modules can boost the performance few-shot classification.
models.
A.9 CLARIFICATION ON CLUSTERING PROCEDURE
cell feature clustering procedure in Sec. During the training stage, the global cluster centers = {v&} are updated by the computed clustering centers {v') in current mini-batch Each update cluster center Vı. is weighted by momentum coefficient 7 determined by the value of an associated counter S5. since we would like to avoid large adjustment from the current mini-bater in order to stabilize the global cluster centers. Besides, the mini-batches of examples are randomly drawn from the dataset following Sculley (2010), without specialized design to optimize clustering learning. During the evaluation stage, fix the global cluster centers V in the forward step of our model, avoiding the potential information leak or transduction from the test mini-batches,
In this section, add more clarification
A.10 MULTI-BRANCE DETAILS
Our embedding ((x) is separated into two branches after shared stem (Y-shape), which is defined as o(x) = {deh (x), (xoo((x)) and (x)(() = (f (x)em)))), (xm)n()) = n (fuem(xx) Two branches ())5((), (x), (x)oto(x) are trained by standard classification and prototypical schemes separately
15

---

Published as a conference paper at ICLR 2021
in multi-task learning fashion. During the testing time, (xx(x) and (x)))()) are concatenated together to compute distance between support prototypes and query images.
For our ConstellationNet, blocks (Conv-4) or the second residual blocks (ResNet-12). We keep the shared stem identical to the network backbone and reduce the channels of two separate branches to match the parameter size of the model without multi-branch.
split the network into two branches after the second convolutional
A.11 CONNECTION WITH CAPSULE NETWORKS
notable development to learning the explicit structured representation in an end-to-end framework is the capsule networks (CapsNets) (Sabour et al., (017)) The line of works on CapsNets (Sabour
ct al. 2017 Hinton et al., 2018; Kosiorek et al. 2019; Tsai et al. 2020) intends to parse a visual scene in interpretable and hierarchical way. Sabour al. (2017) represents parts and objects in vector-based capsules with a dynamic routing mechanism. Tsai et al. (2020) uses a stacked autoencoder architecture to model the hierarchical relation among parts, objects and scenes. Here our ConstellationNet maintains part modeling by enabling the joint learning of the convolution and constellation modules simultaneously attain implicit and explicit representations.
16


Reviews:
[{"review_id": "pXbHQbXFsK6", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThe paper proposes a constellation model that performs feature clustering and encoding dense part representations. The constellation module is placed after convolutional blocks. The module clusters cell features and calculates distance map between each cluster centroids and cell feature. The self-attention mechanism is applied on the distance map and concatenated to the original feature map to complement the feature representation. The resulting feature representation contains part representations. The few-shot experiments on the mini-Imagenet, CIFAR-FS, and FC100 datasets show the effectiveness of the proposed method.\n\n\nRating\n\nOverall, I lean toward accepting the paper. The paper suggests multiple technical components to improve few-shot performance. Among the contributions, a constellation module which is the main idea of the paper plays a major role in the performance gain. The performance is competitive and ablation studies show the importance of constellation module in the performance gain. There are a few questions while reading the paper but not critical.\n\nPros\n-      The proposed module is plugged into the backbone network. The authors verified the effectiveness of the approach on both 4-conv networks and resnet-12 networks.\n-      The paper showed several techniques to improve the performance including the constellation module, multi-branch structure, multi-head attention and feature augmentation.\n-      The proposed approach performs favourably against the competing methods. \n-      Ablation studies show that most of performance gain comes from the constellation module, which is the main idea of the paper while the other technical contributions give additional performance gain.\n\nCons (minor issues)\n-      The combination of multiple components requires more hyper-parameters to tune. (number of clusters, number of heads in attention layer, constellation module locations)\n-      The figure2.(c) shows the constellation module analysis graph. The graph shows more constellation modules results in better performance. However, it is not reported how the model performs when the constellation modules are located in higher layers only. (for example, the module located at only after layer 4.)\n\nQuestions\n-      Although the constellation module is using a clustering algorithm, there is no clustering loss term. I conjecture the network using the constellation module to converge much slower than without the module even though the module does not bring a significant amount of parameter increment. How does the module affect the training time?\n-      Euclidean distance is used for clustering and distance map calculation (equation 4). Is there any reason to choose Euclidean distance rather than the cosine similarity?\n-      In the attention calculation, a dot product is used on the distance map (equation 8). What is the intuition to apply self-attention on the distance map ‘D’ rather than the cell features ‘U’? Distances are the difference between each cluster centroids and cell features. Isn’t cell feature map is more suitable for attention mechanism?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "lnSEj_L7A76", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes ConstellationNet for few-shot learning, which is inspired by constellation models.  The constellation models firstly model local appearance of an image by visual codebooks, and then the local appearance and spatial configuration of keypoints are learned by generative models. The proposed ConstellationNet is a neural network that combines codebook learning and self-attention models, not generative models. The way of using spatial information is different from the constellation models. The experimental results show that the ConstellationNet outperforms state-of-the-art methods in the few-shot learning problem. Overall, the paper is well written, and the connection to the constellation model is interesting. \n\nPros\n+  The historical explanation of spatial parts-based models in computer vision researches is extensive and interesting. The writing of this paper is mostly clear. \n\n+ The mini-batch soft k-means is suitable to conduct codebook learning in the process of CNN training with SGD. \n\n+ The proposed ConstellationNet shows higher accuracies than state-of-the-art few-shot learning methods on standard mini-ImageNet/CIFAR-FS/FC100 datasets.  \n\n+ Ablation study shows the performance improvements are not because of increasing model parameters. The ConstellationNet adds additional layers to the backbone network, and increasing model size improves performance is obvious. To answer this concern, the authors compared with the case when the model size increased differently, and the constellation model outperformed that case. \n\n+ Visualization of codebooks shows similar parts are assigned to the same cluster. \n\nCons\n- My main concern is that if the clustering of cell features are really needed. It is known that the convolutional features correspond to (implicit) object parts. \nThough the similar points are activated in the same cluster in Fig.3, original convolutional channels would be consistently activated in the same semantic points. \nThe ablation study leaks the case when the codebook is not used. The recognition accuracies of ConstellationNet should be compared with the case when the codebook learning is not used. Also, 1x1 convolution can be used to aggregate different channel information and reduce the dimensions, instead of codebook learning.  This case also should be compaerd. \n \n- If the proposed method can be called a constellation model is somewhat questionable. The self-attention only learns feature channel weights for computing similarity of different cells. It does not explicitly learn the spatial configuration of local cells of an object category, unlike the constellation models. \n\n- Visualization of attention is lacking. What spatial relations among the cell features are learned should be shown. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Cn2xWFqe3hJ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The proposed contellation module is an improved version of non-local block [a], which contains a cell feature clustering module and a self-attention module for modeling pixel-wise (cell-wise) relationships. Inserting this block to the backbones could improve the performance for few-shot learning setting.\n\nConcerns:\n1. In my mind, the most important difference between the proposed constellation module and non-local block is the cell feature clustering module. (Multi-head design is widely-used in the extensions of transformer model) Hence, it is important to prove that the newly inserted cell feature clustering module is crucial in the constellation module at least in the few-shot learning setting. It would be bonus to do comparison in other important benchmarks. \n\n2. The corresponding intuition that why the cell feature clustering module works well is also needed. For example, why should we need a shared part patterns (cluster centers) for all different images. Is 128 shared centers enough for all data points? It would be great to visualize the 128 learnt part patterns, e.g. visualizing nearest cell features in the all images.\n\n3. For few-shot learning, there are several papers published recently with state-of-the-art results [b,c], which should be compared in the literature.\n\n[a] Non-local Neural Networks, CVPR 2018\n[b] Negative Margin Matters: Understanding Margin in Few-shot Classification, ECCV 2020\n[c] Boosting Few-Shot Learning With Adaptive Margin Loss, CVPR 2020", "extra_scores": {}, "extra_text": {}}, {"review_id": "9XV7_rpu0uv", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*** Update ***\n\nI thank the authors for their very detailed response. Most of my concerns have been addressed and I now recommend acceptance. \n\n***\n\n\nThe authors propose a method for few-shot learning that introduces a new module in the feature encoder of the prototypical network model. Their approach is inspired by constellation models, which represent objects as a set of parts and model their spatial relationships.   \nThe proposed model alternates between standard convolutional layers and a constellation inspired block. This block first clusters image features across a batch, computes a distance map between each feature and cluster center, then obtain a final feature vector via self attention on the distance map.   \n\nSTRENGTHS\n\nExploiting object parts and their relationships is a promising direction for few-shot learning, where one may aim to share knowledge about objects structures and common parts. The proposed approach achieves good overall performance, and a large performance gain in small capacity networks when introducing the constellation module.   \nThe experimental analysis of the model and parameters is quite thorough, in particular the experiment evaluating the influence of the extended model capacity.   \n\nWEAKNESSES\n\n** missing references **  \n-First, the literature review should be extended to comment on recent approaches relying on part discovery/recognition. All provided references are 10+  years old, and recent works on compositional and part based representations/models for classification should be discussed.   \nFew examples could be:   \nCoupleNet, Zhu et al, ICCV 2017  \nObject-Part Attention Model for Fine-GrainedImage Classification, IEEE TIP, 2018  \nWeakly Supervised Complementary Parts Models for Fine-Grained Image Classification from the Bottom Up, CVPR 2019  \nExploiting spatial relation for fine-grained image classification, Pattern Recognition, 2019  \nLearning Compositional Representations for Few-Shot Recognition, ICCV 2019  \nWhile these works are not necessarily closely related to the proposed work and do not reevaluate novelty, they are contemporary part based works that explore part based modelling. \n\nMore importantly, authors should mention and discuss    \n‘Neural activation models’ Simon et al 2015, ICCV  \nWhich aims to integrate a constellation model within a neural network model and is therefore strongly related work. \n\n** clarity/justifications **  \n-While the approach appears to yield good performance increase, it is difficult to comprehend intuitively why that is the case. Justification for certain model decisions is not clearly provided, and certain claims are not supported by evidence. For example, the claim that the clustering procedure explicitly identifies object parts is not obvious not clearly shown in experiments. \nIndeed, clusters and their centers are assumed to represent object parts. However, experiments demonstrate that larger numbers of clusters yield stronger performance, suggesting that a coarse superpixel type clustering might be more promising than hoping to identify object parts across images. Similarly, the results in Fig 3 suggest that the discovered clusters are more appearance oriented than identifying parts (cf brown vs white on dog instead of recognising parts e.g. ears). \n\nThe clustering process itself lacks clarity. Are cluster centers fixed after training?  Is the optimal number of cluster affected by the number of classes/similarities between classes? Are all cluster centers receiving assignments for each batches? Are batches constructed so as to optimise cluster learning? Can authors expand on the role of count parameter s? Are different clusters relevant to e.g. dog classes different from clusters for bird classes?\nAs intuitively these clusters represent object parts, providing more attention to these, both in terms of explanation and experiments would be preferable. \n\n-The motivation behind the use of a distance map is unclear. Could authors elaborate on why performing self attention on the distance map provides relevant information vs e.g. self attention on the clustered feature maps?\n\n-The paper would benefit from a clearer depiction of the constellation model and how the proposed approach relates to them, intuitively. With the current writing, the concept of constellation models is quickly brushed over, and the motivation behind the use of a distance map + attention is not clearly stated but only proposed as an alternative to the probabilistic modelling used in old school constellation models. The paper would strongly benefit from providing justification on why this is a valid alternative, and what we hope to learn using this strategy; and more precisely, why this strategy can be viewed as a constellation type model. \n\n-Several aspects of the few-shot learning formulation should be more clearly explained. While the paper is perfectly understandable for a reader accustomed to FSL methods and settings, the lack of explanations regarding episode training, meta-training/testing, and the protonet model itself would make it very difficult to follow for a non expert. \n\nRECOMMENDATION \n\nThe proposed work aims to introduce part representation in few-shot models, which is an appealing strategy. Adapting popular traditional models to contemporary settings is a promising idea, and the proposed method reaches SOTA performance on standard benchmark.  \n\nThe paper in its current state needs a little more attention, and I will be happy to increase my rating if my main concerns are addressed  \n1-\tRelating the model and its components more closely to constellation models, and justification as to why the proposed strategy is a better implementation of constellations in deep learning framework than Simon et al. 2015  \n2-\tProviding clarifications regarding design decisions, experimental setting, and more intuition. In particular regarding the distance map and cluster centers. If possible, according more attention to interpretability/observed behaviour of cluster centers in the experimental section.  \n\nADDITIONAL COMMENTS\n-\tExperiments are missing important details. For example, it is not specified for experiments in Figure 2 and 3 which dataset and parameter configurations are used. In particular for figure 3, is the number of clusters set to 64? Are all cluster centers relevant to a given class? \n-\tExamples from the same class provided in this figure look very similar in appearance. What happens when examples of the same class look different? Are same cluster patterns observed?\n-\tThe multi-branch training strategy is not new and was suggested in TADAM, Oreshkin et al., NeurIPS, 2018.\n-\tRegarding ablation experiments, it would be interesting to see the influence of having a single module on the last layer (where levels of abstraction would be higher) vs modules at every layer. \n-\tclaims regarding ‘explicit modelling of parts’ should be revised. There is no explicit part discovery (nor a guarantee that object parts are indeed discovered), nor a clear, explicit modelling of their interactions. Maybe a more accurate characterisation would be that the approach integrate spatial information between image regions of similar appearance/texture. Similarly, it is not obvious that CNNs extract object parts. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a meta-learning method that learns structured features based on constellation modules. Exploiting object parts and their relationships is a promising direction for few-shot learning as AnonReviewer3 described. The effectiveness of the proposed method is demonstrated with experiments using standard benchmark, and ablation study.  
Decision: Accept (Poster)