Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a hierarchical memory to store features at different semantic levels for few-shot learning across domains. It introduces a hierarchical prototype model, where each level of the prototypes fetches the corresponding information from the hierarchical memory.  The authors follow the hyper network design to learn the weights when combining predictions from multiple levels.  The overall model is optimized using a variational inference framework. 

The proposed method is evaluated on 4 various datasets which have a domain gap between the training data.  The authors also show that this method is competitive on the commonly used few-shot image classification benchmarks. 


main_review: Pros
- The paper is well-written and easy to follow. The paper structure is very clear. 
- The idea of leveraging multi-level semantic information is interesting for few-shot learning across domains.  Although using multi-level features is not something new in the space in general, using it as a condition for the prototypes is novel for few-shot learning.  It seems to be a good extension to the previous memory networks. 

Concerns
- Would the proposed method with external memory significantly increase the model size?  The external memory might be something unfair to approaches without explicit external memory. 
- The baselines in the result tables (Table 4 and 5) may not be the current SODA methods. According to the public leader board https://paperswithcode.com/sota/few-shot-image-classification-on-mini-2. the `prototype completion for few-shot learning` (Zhang et al.) work achieves 69.68% +- 0.76% on the 5-way 1-shot experiment using ResNet-12, which is higher than the reported numbers in Table 5. 
- More ablation study in terms of model size (e.g., different backbones), training efficiency (e.g., how fast is the variational inference-based approach?). 



summary_of_the_review: The overall idea is interesting but there are concerns regarding the experiments as well as the introduction of external memory. 


=== Post-rebuttal Comments===
The authors addressed most of my concerns in the feedback.  I kept my score and learned towards acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a hierarchical version of the variational memory approach of [Zhen et al., 2020] for few-shot learning.  The core technical methodology follows that developed by Zhen et al., with the difference being in terms of the model: this work utilizes a deep model with per-layer memory, whereas Zhen et al. utilize a single memory for high-level concepts (i.e., deepest layer only).  Motivating the choice of per-layer memory is the desire to better handle few-shot learning tasks with domain shift, as the representations in earlier layers may be more relevant in such scenarios; the proposed framework learns a hypernetwork to predict attention weights over the per-layer prototypes.

main_review: From a technical standpoint, the work appears to be a relatively straightforward extension of [Zhen et al., 2020].  Hence, experimental validation of the impact of the proposed per-layer memory model is especially important.  The paper presents results on the same few-shot tasks as Zhen et al. (mini-Imagenet and tiered-Imagenet), as well as comparison to other meta-learning methods on few-shot cross-domain tasks.  Here, ablation experiments also show learned weighting of per-layer prototypes to be useful.

However, the experimental results leave open a critical question about the comparison of the proposed approach to the baseline variational semantic memory (VSM) of Zhen et al.  Specifically, the results quoted for VSM in Table 5 are worse than the results in [Zhen et al., 2020] for this same experiment.  In fact, the results reported in Table 6 of [Zhen et al., 2020] are better in 3 out of 4 settings than the results reported for the proposed system.  The discrepancy is:

Method mini-ImageNet (1-shot/5-shot) tiered-ImageNet (1-shot/5-shot)

VSM  65.72 82.73 72.01 86.77

VSM* 64.21 79.69 69.58 83.28

Ours 67.01 81.75 71.70 85.13

where VSM is as reported in [Zhen et al., 2020], VSM* is the "re-implementation" by this submission, and Ours is the submission's approach.  Since the proposed system is an extension of Zhen et al., it is quite detrimental to actually perform worse than the baseline.  These results are also on a central experimental setting (few-show learning with deep models).  It is not acceptable to present a "re-implementation" that flips the ranking of the methods; at minimum, some extensive explanation is required about differences between the original and "re-implementation" as well as why the original published results were not replicated.

summary_of_the_review: While this work appears to be a promising extension of the variational semantic memory of [Zhen et al., 2020] to models with multi-level prototypes, it omits proper comparison to [Zhen et al., 2020] on a central experiment, which, when included shows the proposed approach produces worse performance than the baseline.  The author response should address this discrepancy.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a novel model that focuses on improving cross domain few shot classification. To this end they introduce a hierarchical extension to the work proposed in (Zhen et al. 2020), wherein the latent variables at different levels capture distinct semantic information. The proposed framework enables generating class specific prototypes at different hierarchical levels, which are then used to make predictions at each level. These predictions are ensembled using domain specific weights obtained from the support set via a gradient based method. Through experiments on various cross-domain and in-domain tasks, the authors show considerable improvements over the baselines.
main_review: $\textbf{Strengths}$
1. The paper is well written and easy to follow. The authors do a good job of introducing their model elements, and contrasting them with previous works.
2. The improvements obtained over the baselines are impressive. Specifically, on the task of cross-domain few shot classification, the gap between the proposed method and the most competitive baseline is significant.
3. Additionally, the ablation experiments shown are extensive and do a good job of highlighting the importance of each component in the proposed framework. 

$\textbf{Weaknesses}$
1. Table 13 highlights that the proposed method is not that effective when using shallow feature extractors like Conv-4. This probably leads me to believe that ensembling is providing the major improvements and the proposed hierarchical formulation isn't actually that important. To this end, I have two questions:

a. What happens if you don't use an ensemble and rather just use the logits from the last level (keeping the rest of the architecture as is)? As the later latent variables depend on the earlier ones (Figure 1), if the hierarchical framework works well, you should still see improvements over the baselines.

b. What happens if you train $L$ instances of VSM (Zhen et al. 2020), where each instance is trained on the output of a residual block. That is, if you remove the connections between $\mathbf{z}$ and $\mathbf{m}$ in Figure 1. How crucial is it to have these dependencies between the latent variables.
It would be great if the authors could comment on this.

2. What is the memory overhead of using the proposed hierarchical method over the baseline (Zhen et al. 2020)? I can imagine having $L$ layers of memory to considerably increase the memory overhead. Additionally, due to the increase in the number of latent variables, is convergence slower as well?

3. The VSM numbers shown in Table 5 are lower than what is reported in (Zhen et al. 2020). The numbers in (Zhen et al. 2020) show that the proposed method is inferior when compared to VSM on within-domain few shot classification. I understand that the authors re-implemented their model, but is there an explanation as to why the reimplemented numbers are considerably lower?


summary_of_the_review: This work provides a logical extension to the existing work in (Zhen et al. 2020) by introducing a hierarchical variational memory framework. Through the experiment results it is evident that the proposed method provides considerable improvements over existing approaches. I have some concerns regarding the actual importance of dependencies within the latent variables. I'm still inclined to accept this paper, and would be willing to increase my rating if the authors address my concerns.
review_title: null

Meta-review: This paper presents a hierarchical memory for cross domain and few shot classification problems. The paper is well written, tackles an important topic, and the proposed approach which is an extension of VSM is interesting. Reviewer YEXZ has some concerns regarding comparison to a more proper baseline. I believe that the authors have adequately addressed this. Reviewer 2Ajk and g1Bf also have suggestions that the authors have incorporated in the revision. I recommend accepting this paper.
Decision: Accept (Poster)