Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary

The paper proposes a constellation model that performs feature clustering and encoding dense part representations. The constellation module is placed after convolutional blocks. The module clusters cell features and calculates distance map between each cluster centroids and cell feature. The self-attention mechanism is applied on the distance map and concatenated to the original feature map to complement the feature representation. The resulting feature representation contains part representations. The few-shot experiments on the mini-Imagenet, CIFAR-FS, and FC100 datasets show the effectiveness of the proposed method.


Rating

Overall, I lean toward accepting the paper. The paper suggests multiple technical components to improve few-shot performance. Among the contributions, a constellation module which is the main idea of the paper plays a major role in the performance gain. The performance is competitive and ablation studies show the importance of constellation module in the performance gain. There are a few questions while reading the paper but not critical.

Pros
-      The proposed module is plugged into the backbone network. The authors verified the effectiveness of the approach on both 4-conv networks and resnet-12 networks.
-      The paper showed several techniques to improve the performance including the constellation module, multi-branch structure, multi-head attention and feature augmentation.
-      The proposed approach performs favourably against the competing methods. 
-      Ablation studies show that most of performance gain comes from the constellation module, which is the main idea of the paper while the other technical contributions give additional performance gain.

Cons (minor issues)
-      The combination of multiple components requires more hyper-parameters to tune. (number of clusters, number of heads in attention layer, constellation module locations)
-      The figure2.(c) shows the constellation module analysis graph. The graph shows more constellation modules results in better performance. However, it is not reported how the model performs when the constellation modules are located in higher layers only. (for example, the module located at only after layer 4.)

Questions
-      Although the constellation module is using a clustering algorithm, there is no clustering loss term. I conjecture the network using the constellation module to converge much slower than without the module even though the module does not bring a significant amount of parameter increment. How does the module affect the training time?
-      Euclidean distance is used for clustering and distance map calculation (equation 4). Is there any reason to choose Euclidean distance rather than the cosine similarity?
-      In the attention calculation, a dot product is used on the distance map (equation 8). What is the intuition to apply self-attention on the distance map ‘D’ rather than the cell features ‘U’? Distances are the difference between each cluster centroids and cell features. Isn’t cell feature map is more suitable for attention mechanism?

review_title: I lean toward acceptance. But I have few questions.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes ConstellationNet for few-shot learning, which is inspired by constellation models.  The constellation models firstly model local appearance of an image by visual codebooks, and then the local appearance and spatial configuration of keypoints are learned by generative models. The proposed ConstellationNet is a neural network that combines codebook learning and self-attention models, not generative models. The way of using spatial information is different from the constellation models. The experimental results show that the ConstellationNet outperforms state-of-the-art methods in the few-shot learning problem. Overall, the paper is well written, and the connection to the constellation model is interesting. 

Pros
+  The historical explanation of spatial parts-based models in computer vision researches is extensive and interesting. The writing of this paper is mostly clear. 

+ The mini-batch soft k-means is suitable to conduct codebook learning in the process of CNN training with SGD. 

+ The proposed ConstellationNet shows higher accuracies than state-of-the-art few-shot learning methods on standard mini-ImageNet/CIFAR-FS/FC100 datasets.  

+ Ablation study shows the performance improvements are not because of increasing model parameters. The ConstellationNet adds additional layers to the backbone network, and increasing model size improves performance is obvious. To answer this concern, the authors compared with the case when the model size increased differently, and the constellation model outperformed that case. 

+ Visualization of codebooks shows similar parts are assigned to the same cluster. 

Cons
- My main concern is that if the clustering of cell features are really needed. It is known that the convolutional features correspond to (implicit) object parts. 
Though the similar points are activated in the same cluster in Fig.3, original convolutional channels would be consistently activated in the same semantic points. 
The ablation study leaks the case when the codebook is not used. The recognition accuracies of ConstellationNet should be compared with the case when the codebook learning is not used. Also, 1x1 convolution can be used to aggregate different channel information and reduce the dimensions, instead of codebook learning.  This case also should be compaerd. 
 
- If the proposed method can be called a constellation model is somewhat questionable. The self-attention only learns feature channel weights for computing similarity of different cells. It does not explicitly learn the spatial configuration of local cells of an object category, unlike the constellation models. 

- Visualization of attention is lacking. What spatial relations among the cell features are learned should be shown. 

review_title: Connection to constellation models is interesting

final_score: rating: 6: Marginally above acceptance threshold
review_content: *** Update ***

I thank the authors for their very detailed response. Most of my concerns have been addressed and I now recommend acceptance. 

***


The authors propose a method for few-shot learning that introduces a new module in the feature encoder of the prototypical network model. Their approach is inspired by constellation models, which represent objects as a set of parts and model their spatial relationships.   
The proposed model alternates between standard convolutional layers and a constellation inspired block. This block first clusters image features across a batch, computes a distance map between each feature and cluster center, then obtain a final feature vector via self attention on the distance map.   

STRENGTHS

Exploiting object parts and their relationships is a promising direction for few-shot learning, where one may aim to share knowledge about objects structures and common parts. The proposed approach achieves good overall performance, and a large performance gain in small capacity networks when introducing the constellation module.   
The experimental analysis of the model and parameters is quite thorough, in particular the experiment evaluating the influence of the extended model capacity.   

WEAKNESSES

** missing references **  
-First, the literature review should be extended to comment on recent approaches relying on part discovery/recognition. All provided references are 10+  years old, and recent works on compositional and part based representations/models for classification should be discussed.   
Few examples could be:   
CoupleNet, Zhu et al, ICCV 2017  
Object-Part Attention Model for Fine-GrainedImage Classification, IEEE TIP, 2018  
Weakly Supervised Complementary Parts Models for Fine-Grained Image Classification from the Bottom Up, CVPR 2019  
Exploiting spatial relation for fine-grained image classification, Pattern Recognition, 2019  
Learning Compositional Representations for Few-Shot Recognition, ICCV 2019  
While these works are not necessarily closely related to the proposed work and do not reevaluate novelty, they are contemporary part based works that explore part based modelling. 

More importantly, authors should mention and discuss    
‘Neural activation models’ Simon et al 2015, ICCV  
Which aims to integrate a constellation model within a neural network model and is therefore strongly related work. 

** clarity/justifications **  
-While the approach appears to yield good performance increase, it is difficult to comprehend intuitively why that is the case. Justification for certain model decisions is not clearly provided, and certain claims are not supported by evidence. For example, the claim that the clustering procedure explicitly identifies object parts is not obvious not clearly shown in experiments. 
Indeed, clusters and their centers are assumed to represent object parts. However, experiments demonstrate that larger numbers of clusters yield stronger performance, suggesting that a coarse superpixel type clustering might be more promising than hoping to identify object parts across images. Similarly, the results in Fig 3 suggest that the discovered clusters are more appearance oriented than identifying parts (cf brown vs white on dog instead of recognising parts e.g. ears). 

The clustering process itself lacks clarity. Are cluster centers fixed after training?  Is the optimal number of cluster affected by the number of classes/similarities between classes? Are all cluster centers receiving assignments for each batches? Are batches constructed so as to optimise cluster learning? Can authors expand on the role of count parameter s? Are different clusters relevant to e.g. dog classes different from clusters for bird classes?
As intuitively these clusters represent object parts, providing more attention to these, both in terms of explanation and experiments would be preferable. 

-The motivation behind the use of a distance map is unclear. Could authors elaborate on why performing self attention on the distance map provides relevant information vs e.g. self attention on the clustered feature maps?

-The paper would benefit from a clearer depiction of the constellation model and how the proposed approach relates to them, intuitively. With the current writing, the concept of constellation models is quickly brushed over, and the motivation behind the use of a distance map + attention is not clearly stated but only proposed as an alternative to the probabilistic modelling used in old school constellation models. The paper would strongly benefit from providing justification on why this is a valid alternative, and what we hope to learn using this strategy; and more precisely, why this strategy can be viewed as a constellation type model. 

-Several aspects of the few-shot learning formulation should be more clearly explained. While the paper is perfectly understandable for a reader accustomed to FSL methods and settings, the lack of explanations regarding episode training, meta-training/testing, and the protonet model itself would make it very difficult to follow for a non expert. 

RECOMMENDATION 

The proposed work aims to introduce part representation in few-shot models, which is an appealing strategy. Adapting popular traditional models to contemporary settings is a promising idea, and the proposed method reaches SOTA performance on standard benchmark.  

The paper in its current state needs a little more attention, and I will be happy to increase my rating if my main concerns are addressed  
1-	Relating the model and its components more closely to constellation models, and justification as to why the proposed strategy is a better implementation of constellations in deep learning framework than Simon et al. 2015  
2-	Providing clarifications regarding design decisions, experimental setting, and more intuition. In particular regarding the distance map and cluster centers. If possible, according more attention to interpretability/observed behaviour of cluster centers in the experimental section.  

ADDITIONAL COMMENTS
-	Experiments are missing important details. For example, it is not specified for experiments in Figure 2 and 3 which dataset and parameter configurations are used. In particular for figure 3, is the number of clusters set to 64? Are all cluster centers relevant to a given class? 
-	Examples from the same class provided in this figure look very similar in appearance. What happens when examples of the same class look different? Are same cluster patterns observed?
-	The multi-branch training strategy is not new and was suggested in TADAM, Oreshkin et al., NeurIPS, 2018.
-	Regarding ablation experiments, it would be interesting to see the influence of having a single module on the last layer (where levels of abstraction would be higher) vs modules at every layer. 
-	claims regarding ‘explicit modelling of parts’ should be revised. There is no explicit part discovery (nor a guarantee that object parts are indeed discovered), nor a clear, explicit modelling of their interactions. Maybe a more accurate characterisation would be that the approach integrate spatial information between image regions of similar appearance/texture. Similarly, it is not obvious that CNNs extract object parts. 

review_title: Promising idea, paper can be strengthened with more clarifications and justifications

final_score: rating: 5: Marginally below acceptance threshold
review_content: The proposed contellation module is an improved version of non-local block [a], which contains a cell feature clustering module and a self-attention module for modeling pixel-wise (cell-wise) relationships. Inserting this block to the backbones could improve the performance for few-shot learning setting.

Concerns:
1. In my mind, the most important difference between the proposed constellation module and non-local block is the cell feature clustering module. (Multi-head design is widely-used in the extensions of transformer model) Hence, it is important to prove that the newly inserted cell feature clustering module is crucial in the constellation module at least in the few-shot learning setting. It would be bonus to do comparison in other important benchmarks. 

2. The corresponding intuition that why the cell feature clustering module works well is also needed. For example, why should we need a shared part patterns (cluster centers) for all different images. Is 128 shared centers enough for all data points? It would be great to visualize the 128 learnt part patterns, e.g. visualizing nearest cell features in the all images.

3. For few-shot learning, there are several papers published recently with state-of-the-art results [b,c], which should be compared in the literature.

[a] Non-local Neural Networks, CVPR 2018
[b] Negative Margin Matters: Understanding Margin in Few-shot Classification, ECCV 2020
[c] Boosting Few-Shot Learning With Adaptive Margin Loss, CVPR 2020
review_title: Review

Meta-review: This paper proposes a meta-learning method that learns structured features based on constellation modules. Exploiting object parts and their relationships is a promising direction for few-shot learning as AnonReviewer3 described. The effectiveness of the proposed method is demonstrated with experiments using standard benchmark, and ablation study.  
Decision: Accept (Poster)