Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors propose a method for cross domain few-shot classification that learns to generate domain specific data statistics from very few training examples for domain independent batch normalisation.  
They propose to train small auxiliary networks that generate data statistics for normalisation. Networks are trained within a meta-learning framework using a KL divergence loss, which enforces estimated statistics on small support/training examples to match statistics from query sets where more data is available. 


STRENGTHS

The paper is well written and motivated. The proposed approach is, for the most part, easy to follow and understand. The approach benefits from its simplicity and versatility (e.g. it is not tied to a specific FSL method) and promising performance is obtained.

The authors provide a very large set of experiments in multiple scenarios, and definitely demonstrates a strong effort in evaluating their approach. 


WEAKNESSES

Unfortunately, despite the fact that a significant amount of time was dedicated to evaluate the method, the experimental section needs substantial modifications. This is mainly due to the presentation of the experiments, as well as some key missing comparisons.   
Regarding presentation, too many implementation details and descriptions of the experiments are missing from the main paper (and in certain cases, missing altogether). Regarding datasets and implementation, it is ok to provide non essential details in the appendix (especially considering the large number of datasets considered). However, no information is provided at all in the main paper, which makes it very difficult to understand the setting of the experiments. For example, in Table 1 and Figure 2, it is not known on which datasets experiments are run, and it is never mentioned in the main text (for table 1) what FSL method is employed.   
In addition, Table 2 experiments comparing MetaNorm to different approaches sorely lacks description. The approach is compared to 9 different algorithms, none of which are given a description or reference to learn more about the method. It is therefore impossible, besides guessing, to know what the model is being compared to. This issue is also noticeable in Table 3-5, in particular with a baseline in Table 4 that is never described.   
Finally, experiments in Figure 2 would be a lot more interesting if compared to standard methods. It would be interesting to see how the proposed strategy allows to be more sample efficient and reach stronger performance than transductive batch norm in situations where sample size is the smallest. As this is one of the cited main limitations of TBN, this experiment is highly important2- With regards to related work, I would suggest to move the section after the introduction, where it provides much better context to facilitate method comprehension, in particular regarding the description of the TBN strategy. 

Authors should also comment on how their work, and in particular FSL domain generalisation setting, relates to cross domain few-shot learning works, 
e.g. Tseng et al ICLR 2020, Cross-Domain Few-Shot Classification via Learned Feature-Wise Transformation
It is currently presented as a completely new approach to FSL, and appears to ignore past cross domain FSL works. Please provide additional context regarding such works.


RECOMMENDATION

In summary, the authors propose an interesting, simple strategy for more robust BN that can be of interest to the community. I would strongly recommend that the authors make the following modifications to strengthen their paper:

1-	Reorganise and expand on the experimental evaluation to provide necessary details and make the main paper self-contained, even if it requires moving an experiment to the supplementary materials.  
2-	Move the related works section after the introduction, to provide additional context before delving into the method  
3-	Relate the proposed work to past work on cross-domain FSL and potentially tone down claims that few-shot cross domain learning is a completely new problem investigated here.  
4-	Please provide a comparison to standard TBN in Figure 2


Additional suggestions

5-	If possible, please provide an overview figure of the proposed method.  
6-	In result tables, please sort methods according to their overall performance, and correct bolding in table 2 (protonet 5-way-5 shot MetaNorm is not best performing method) and highlight setting where methods have very similar performance (as in Table 14) for clarity.   
7-	Provide a clear list of contributions at the end of the introduction section  
8-	While KL divergence and hypernetworks are well known terms, it would make the paper more accessible to add a sentence (and equation in the case of the KL divergence) or two describing the terms. In particular, hypernetworks are generally used to characterise weight generators for entire architectures and might lead to confusion.  
9-	It could be nice to provide more attention to how this work relates to conditional batch norm works, and whether they can be complementary.  

review_title: Interesting approach, some improvements to the paper (experiments) needed

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: The paper proposes an effective meta-learning normalization, named MetaNorm, to infer adaptive statistics for batch normalization by minimizing the KL divergence. The module is lightweight. The proposed method is evaluated on few-shot learning, domain generalization, and few-shot domain generalization. 

Justification of rating:  Overall, the proposed approach is logical and sound. The formulation and methodology seem to be correct. As I am not personally working on the specific topics, I might not be able to discover major issues in this work.

Strengths:
+ The proposed MetaNorm leverage meta-learning approach with KL divergence for learning to learn normalization statistics from data. 
+ The proposed approach is model agnostic and can be easily embedded into meta-learning approaches.
+ Evaluation demonstrates it is able to learn with few examples, as well as handle variations in domain for domain generalization problem. The experimental results show consistent improvement over compared baselines (Table 2-5)
+ The paper provide sufficient information for researcher to reproduce the results. Code will be released in the future.

Minor comments:
- The implementation of the hypernetworks is effectively MLP with one hidden layers of 128 units. Has the author conduct ablation on hypernetwork on various configuration?
review_title: This work address batch normalization by introduce a KL term for learning to learn statistic

final_score: rating: 7: Good paper, accept
review_content: This paper proposes to replace batch normalization statistics, which are typically computed as the batch moments during training or a fixed training average during testing, with the outputs of learned neural networks. These networks are trained to minimize the KL divergence between their output and the expected or desired batch statistics. In this way, the statistics computation is amortized and can hopefully generalize in the face of small batches and distribution shift.

Pros:

+ The paper is generally well written and easy to follow.
+ There are extensive experiments demonstrating the empirical effectiveness of the proposed approach.

Cons:

- Perhaps missing some additional details that would help the reader better understand why and how the method works.

Currently, I recommend acceptance because I believe the pros rather handily outweigh the cons. I provide more details below.


Quality
---

The paper is well written and the experiments are thorough and well executed. Related to the con listed above, a few additional experiments could be useful for gaining a deeper understanding of the method.

First, why do we expect an amortized procedure to produce reasonable normalization statistics when tested "out of distribution", e.g., on a new domain? It would be interesting to report, during testing, are the inferred statistics actually close to the ground truth statistics on the test domain? Or otherwise, are they interpretable in some other fashion? Any other insight that the authors could supply regarding this general question would also be appreciated.

Second, a similar question can be asked about the fully "in distribution" setting, i.e., where batch normalization was invented. If one were to apply the proposed method in a standard supervised learning setting (e.g., minimize KL between inferred statistics on each training point and the ground truth statistics computed on the training batch), would the inferred test time statistics come close to either the average statistics computed through training or perhaps even the ground truth test statistics? Is it any worse than standard batch norm? Could it perhaps even be better?

Clarity
---

Most of my concerns about clarity, which are small, relate to the experiments suggested above, i.e., better exposition of how and why the method works. Providing intuition throughout the paper regarding this point would strengthen the paper.

The few shot domain generalization setting is not too difficult to understand if the reader is familiar with both the few shot learning and domain generalization settings individually -- it is probably difficult to parse otherwise, but that is perhaps hard to rectify given the space constraints.

There also seem to be a few instances of "meta" vs "non meta" terms being misused, e.g., "training" vs "meta-training" when describing domain generalization.

Originality
---

As far as I am aware, this work presents a novel method, though I am not an expert regarding the relevant prior work. The citations to batch normalization as used in domain adaptation seem appropriate. A few other papers in a similar vein perhaps should also be cited:

https://arxiv.org/abs/1603.04779
https://arxiv.org/abs/2002.04019
https://arxiv.org/abs/2006.10963
https://arxiv.org/abs/2006.16971

So although this general line of work seems to be attracting a decent amount of attention, this work is still novel in that it amortizes the inference process and presents extensive empirical results.

Significance
---

This work seems significant to researchers interested in meta-learning, domain generalization, and problems involving distribution shift in general. The proposed method is also relatively simple, allowing for easier adoption and further testing.
review_title: Official Blind Review #1

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper describes a new method for normalizing few-shot learning episodes. The authors point out that the statistics of an episode are unreliable when the size of the episode is small or when the data distribution changes from episode to episode. To remedy this, the authors propose a method called ‘MetaNorm’ which uses a meta-learning approach to infer the means and variances to be used in the batch normalization layers that are employed in the feature extractor component. In particular, they meta-learn the parameters for a set of hypernetworks in an amortized fashion that learn to generate the means and variances of the batch normalization layers conditioned on the contents of the episode. The paper focuses entirely on the few-shot image classification scenario where MetaNorm is evaluated in various settings including standard few-shot classification and domain generalization (including a novel few-shot domain generalization setting).

**Pros:**
Fundamentally, the concept behind MetaNorm is innovative and promising. The idea of using meta-learned hypernetworks to generate the means and variances of the normalization layers is good. The paper also makes a good case for the effectiveness of a KL term that is added to the classifer loss function that helps the hypernetworks learn a good set of parameters. The range of experiments is sufficient.

**Cons:**
The paper is poorly executed and missing too much information to ascertain that the method outperforms competitive approaches.

Specific concerns:  
(1) There is no mention of the MetaNorm training efficiency (which is generally understood to be the primary goal of batch normalization, see [1,2]). How does training efficiency of MetaNorm compare to other methods? And if training efficiency is not a goal, MetaNorm should be compared to methods whose goal is to adapt a classification system to a variety of data distributions (e.g. [3,4,5,6, etc.]).

(2) Reproducibility: The paper is currently missing details that are required to reproduce the results and ensure fair comparison with other methods including:
- Which existing codebases were used (if any)?
- What hyper-parameters and training settings are used? (e.g. learning rates, number of training iterations, the use of early stopping, the number of test episodes, weight decay, etc.)
- In Section 4, “Impact of Target Set Size” indicates that the number of samples in the query set is a key parameter for MetaNorm and that 125 for the few-shot scenario and 128 for the domain generalization scenario are the best values to use. Are these the values used in the experiments? If not, what values are used?

(3) There are many omissions in the paper:
- The variable ‘m’ as used in equations 3, 4, 8, 9, 10 is not defined.
- The KL expressions in equations 4, 8, and 10 should be written as a proper minimization expression where the parameters that are minimized are explicitly stated.
- The total loss function for MetaNorm is never stated. One could assume that it is a sum between the regular loss function for the particular few shot learning problem and the KL term, but it should be explicitly stated. Also, is there a weight on the KL term in the loss function? If so, how is the optimal weight determined?
- Various methods in the tables are not defined or referenced including RN, TaskNorm-I, TaskNorm-L, ‘class’, and ‘example’.
- How are the confidence intervals on the results in the table calculated? Are they 95% confidence intervals, or something else?
- In the tables, how do you decide which entry to ‘bold’ given the confidence intervals? In table 2, TBN (and sometimes TaskNorm) are often within error bars of MetaNorm but are not bolded. Same for tables 11 and 13, 14.
- It would be good to have more information on the hypernetworks that generate the normalization parameters. In Section 2, it says: “They are three layer networks with one hidden layer of 128 units and the input size depends on the feature dimensions of the convolutional layers.” How big is the other hidden layer and output size of the network? What sort of non-linearities are used, etc.?
- In section 2 it says: “Note that the inference functions $f_\mu^l()$ and $f_\sigma^l()$ are shared by different channels in the same layer and we will learn L pairs of those functions if we have L convolutional layers in the meta-learning model”. What is the justification for this parameterization (say versus having unique functions per channel)? Was an ablation study that compares various configurations done?

(4) There are factual errors in the paper:
- In Section 1 and Section 3, the authors state that Prototypical networks uses transductive batch normalization. This is not the case (refer to https://github.com/jakesnell/prototypical-networks for the version of Prototypical networks authored by Jake Snell). Please correct.
- In section 1 it states that TaskNorm “remains under-performing compared to transductive batch normalization”. While the results listed in Table 2 support this statement, the results in Table 3 refute this statement. Please clarify.

(5) There are several missing attributions and some exact text taken from other papers that is not quoted or cited: 
- In Table 2, the competitive results for MAML and ProtoNets seem to be extracted from [2], yet these are not attributed. Did you reproduce them? Were the MetaNorm numbers generated with a similar code base and hyper-parameters?
- In Tables 11, 13, and 14, the competitive results also seem to be extracted from [2] and were not attributed. Were they reproduced, or just stated? Were the MetaNorm numbers generated with a similar code base and hyper-parameters?
- The following sentences seem to be directly copied from [2]. If so, there should be quotation marks and attribution:
“Batch normalization relies on the implicit assumption that the dataset comprises i.i.d. samples from some underlying distribution.”
“The challenge constructs few-shot learning tasks by drawing from the following distribution. First, one of the datasets is sampled uniformly; second, the “way” and “shot” are sampled randomly according to a fixed procedure; third, the classes and context / target instances are sampled.”
“In the meta-test phase, the identity of the original dataset is not revealed and the tasks must be treated independently (i.e. no information can be transferred between them). The meta-training set comprises a disjoint and dissimilar set of classes from those used for meta-test.”

(6) In Figure 2, the title says “Impact of target set size.”. However, for the left and middle plot, the horizontal axis is labeled as |S|, which is the support set size. Please resolve the ambiguity.
Minor Comments:
- In section 2, MetaNorm for Few-Shot Classification: The sentence “The $p(m|Q)$ can be estimated by directly calculating statistics using the query set, which however performs inferior to inference by optimization.” is grammatically incorrect.
- In section 2, MetaNorm for Few-Shot Classification: should “multiple layer perception networks” be “multi-layer perceptron networks”? Similarly, in section 2 it says: “…which are realized as multi-layer perceptions and we call hypernetworks.” ‘perceptions’ should be ‘perceptrons’.
- Sometimes the nomenclature support set / query set is used, and other places it refers to the same as context set / target set. Please use consistent nomenclature.
- In section 2, it says “…which are realized as multi-layer perceptions and we call hypernetworks”. And later in section 2, it says: “They are parameterized by feed-forward multiple layer perception networks, which we call hypernetworks”.  Note that hypernetworks are an established concept, you should cite [7].
- Section 1 should clarify in the batch normalization exposition that the methods described in the paper apply only to normalizing 2D convolutional layers (i.e. does not apply to fully connected layers as they do not have feature maps).

**References:**  
[1] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 448–456, 2015.  
[2] John Bronskill, Jonathan Gordon, James Requeima, Sebastian Nowozin, and Richard Turner. Tasknorm: Rethinking batch normalization for meta-learning. In International Conference on Machine Learning. 2020.  
[3] Rebuffi, Sylvestre-Alvise, Hakan Bilen, and Andrea Vedaldi. "Learning multiple visual domains with residual adapters." Advances in Neural Information Processing Systems. 2017.  
[4] Rebuffi, Sylvestre-Alvise, Hakan Bilen, and Andrea Vedaldi. "Efficient parametrization of multi-domain deep neural networks." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.  
[5] Requeima, James, et al. "Fast and flexible multi-task classification using conditional neural adaptive processes." Advances in Neural Information Processing Systems. 2019.  
[6] Tseng, Hung-Yu, et al. "Cross-domain few-shot classification via learned feature-wise transformation." arXiv preprint arXiv:2001.08735 (2020).  
[7] Ha, David, Andrew Dai, and Quoc V. Le. "Hypernetworks." arXiv preprint arXiv:1609.09106 (2016).

review_title: Review of "MetaNorm: Learning to Normalize Few-Shot Batches Across Domains"

Meta-review: This paper proposes an lightweight method for cross-domain few-shot learning, using a meta-learning approach to predict batch normalization statistics.
After the extensive paper revisions and discussion, the reviewers all agreed that this paper is above the bar for acceptance, assuming that the authors will include results for both the standard and expanded target set size in the final version of the paper. The authors are strongly encouraged to include these results in the camera-ready version of the paper.
Decision: Accept (Poster)