MetaNorm: Learning to Normalize Few-Shot Batches Across Domains
Authors: Yingjun Du
Xiantong Zhen
Ling Shao
Cees G. M. Snoek
Venue: ICLR 2021 Poster (2021)
Keywords: Meta-learning
batch normalization
few-shot domain generalization

Abstract: Batch normalization plays a crucial role when training deep neural networks. However, batch statistics become unstable with small batch sizes and are unreliable in the presence of distribution shifts. We propose MetaNorm, a simple yet effective meta-learning normalization. It tackles the aforementioned issues in a unified way by leveraging the meta-learning setting and learns to infer adaptive statistics for batch normalization. MetaNorm is generic, flexible and model-agnostic, making it a simple plug-and-play module that is seamlessly embedded into existing meta-learning approaches. It can be efficiently implemented by lightweight hypernetworks with low computational cost. We verify its effectiveness by extensive evaluation on representative tasks suffering from the small batch and domain shift problems: few-shot learning and domain generalization. We further introduce an even more challenging setting: few-shot domain generalization. Results demonstrate that MetaNorm consistently achieves better, or at least competitive, accuracy compared to existing batch normalization methods.  

Published as a conference paper at ICLR 2021
METANORM: LEARNING TO NORMALIZE FEW-SHOT BATCHES ACROSS DOMAINS
Yingjun Du', Xiantong Zhen Ling Shao?, Cees G. M. Snoek
AIM Lab, University of Amsterdam IIccppioon Institute of Artificial Intelligence
ABSTRACT
Batch normalization plays a crucial role when training deep neural networks. How- ever, batch statistics become unstable with small batch sizes and are unreliable in the presence of distribution shifts. We propose MetaNorm, a simple yet effective meta-learning normalization. It tackles the aforementioned issues unified way by leveraging the meta-learning setting and learns to infer adaptive statistics for batch normalization. MetaNorm is generic, flexible and model-agnostic, mak- ing it simple woug-add-pla. module that is seamlessly embedded into existing meta-learning approaches. It can be efficiently implemented by lightweight hyper- networks with low computational cost. We verify its effectiveness by extensive evaluation on representative tasks suffering from the small batch and domain shift problems: few-shot learning and domain generalization. We further introduce even more challenging setting: few-shot domain generalization. Results demon- strate that MetaNorm consistently achieves better, at least competitive, accuracy compared to existing batch normalization methods.
INTRODUCTION
Batch normalization (loffe & Szegedy, 2015) is crucial for training neural networks, and with its variants, c.g., layer normalization (Ba al., 2016), group normalization (Wu & He, 2018) and instance normalization (Ulyanov et al., 2016), has thus become an essential part of the deep learning toolkit (Bjorck et al., 2018; Luo et al., 2018a; Yang et al., 2019; Jia et al., 2019; Luo et al., 2018b; Summers & Dinneen, 2020). Batch normalization helps stabilize the distribution of internal activations when a model is being trained. Given a mini-batch B, the normalization is conducted along each individual feature channel for 2D convolutional neural networks. During training, the batch normalization moments are calculated as follows:
M
1 M
Oi
04
pB)2
where a, indicates the i-th element of the M activations in the batch, M [B] H x W, in which H and W are the height and width of the feature map in each channel. We can now apply the normalization statistics to each activation:
de (N((). = 2ā, + 8, where, a,
(2)
Vo +
where y and B are parameters learned during training, is small scalar to prevent division by 0, and operations between vectors are element-wise, At test time, the standard practice normalize activations using the moving average over mini-batch means μg and variance og. Batch normalization based on an implicit assumption that the samples in the dataset are independent and identically distributed. However, this assumption does not hold in challenging settings like few-shot learning and domain generalization. In this paper, we strive for batch normalization when batches are small size and suffer from distributions shifts between source and target domains.
Batch normalization for few-shot learning and domain generalization problems have so far been considered separately, predominantly in a meta-lea setting. For few-shot meta-learning (Finn

---

Published as a conference paper at ICLR 2021
et al., 2017; Gordon et al., 2019). most existing methods rely critically on transductive batch nor- malization, except those based on prototypes (Snell et al., 2017; Allen et al., 2019; Zhen et al., 2020a). However, the nature of transductive learning restricts its application due to the require- ment to sample from the test set. To address this issue, Bronskill et al. (2020) proposes TaskNorm, which leverages other statistics from both layer and instance normalization. As a non-transductive normalization approach, it achieves impressive performance and outperforms conventional batch normalization (loffe& & Szegedy, 2015). However, its performance is not always performing better than transductive batch normalization. Meanwhile, domain generalization (Muandet et al., 2013; Balaji et al., 2018; et al., 7a;b) suffers from distribution shifts from training to test, which makes it problematic to directly apply statistics calculated from a seen domain to test data from unseen domains (Wang et al., 2019; Seo et al., 2019). Recent works deal with this problem by learning a domain specific normalization (Chang et al., 2019; Seo et al., 2019) or transferable normalization in place of existing normalization techniques (Wang et al., 2019). We address the batch normalization challenges for few-shot classification and domain generalization in a unified way by learning new batch normalization under the meta- meta-leerning setting.
We propose MetaNorm, simple but effective meta-learning normalization. We leverage the meta- learning setting and learn to infer normalization statistics from data, instead of applying direct calculations or blending various normalization statistics. MetaNorm is general batch normalization approach, which is model-agnostic and serves as a plug-and-play module that can be seamlessly embedded into existing meta- meta-learning approaches. We demonstrate its effectiveness for few-shot classification and domain generalization, where it learns task-specific statistics from limited data samples in the support set for each few-shot task; and it can also learn generate domain-specilfe statistics from the seen source domains for unseen target domains. We verify the effectiveness of MetaNorm by extensive evaluation on few-shot classification and domain generalization tasks. For few-shot classification, we experiment with representative gradient, metric and model-based meta-learning approaches on fourteen benchmark datasets. For domain generalization, we evaluate the model three widely-used benchmarks for cross-domain visual object classification. Last but not least, we introduce the challenging new task of few-shot domain generalization, which combines the challenges of both few-shot learning and domain generalization. The experimental results demonstrate the benefit of MetaNorm compared to existing batch normalizations.
2 RELATED WORKS
Transductive Batch Normalization For conventional batch normalization under supervised set- tings, i.i.d. assumptions about the data distribution imply that estimating moments from the training set will provide appropriate normalization statistics for test data. However, in the meta- meta-learning scenario data points are only assumed to be i.i.d. within a specific task. Therefore, it is critical to select the moments when batch normalization is applied to support and query set data points during meta training and meta testing. Hence, in the recent meta-learning literature the running moments no longer used for normalization at meta-test test time, but instead replaced with support/query set statistics. These statistics are used for normalization, both at meta- meta-train and meta-test time. This approach is referred to as transductive batch normalization (TBN) (Bronskill et al., 2020). Competit- ive meta-learning methods (e.g., Gordon et al., 2019; Finn et al., 2017; Zhen et al., 2020b) rely on TBN to achieve state-of-the-att performance. However, there are two critical problems with TBN. First, TBN is sensitive to the distribution over the query set used during meta-training, and as such is less generally applicable than -rnndranndeee learning. Second, TBN uses extra information for multiple test samples, compared to non- non-transductive batch normalization at prediction time, which could be problematic as we are not guaranteed to have set of test samples available during training in practical applications. In contrast, MetaNorm is non-transductive normalization. It generates statistics from the support set only, without relying on query samples, making it more practical.
Meta Batch Normalization To address the problem of transductive batch normalization and improve conventional batch normalization, meta-batch normalization (MetaBN) was introduced (Triantafillou et al., 2020; Bronskill et al., 2020). In MetaBN, the support set alone is used to compute the normalization statistics for both the support and query sets at both meta- meta-training training and meta-test time. MetaBN is non- non-transductive since the normalization of test input does not depend inputs in the query set. However, Bronskill et al. (2020) observe that MetaBN performs less well for small-sized support sets. This leads to high variance in moment estimates, which is similar to the
other test
2

---

Published as a conference paper at ICLR 2021
diffieulty of using batch normalization with small-batch training (Wu & He, 2018). To address this issue, Bronskill et al. (2020) proposed TaskNorm, which learns to combine statistics from both layer normalization and instance normalization, with lending parameter to be learned at meta-train time. As non-transductive normalization, TaskNorm achieves impressive performance, outperforming conventional batch normalization. However, it not always perform better than transductive batch normalization. TaskNorm indicates non-transductive batch normalization estimates proper normalization statistics by involving learning in the normalization process. We also propose to learn batch normalization within the meta-learning framework, but instead of employing a learnable combination of existing normalization statisties, we directly learn to infer statistics from data. At meta-train time, the model learns to acquire the ability to generate statistics only from the support set and at meta-test time we directly apply the model to infer statistics for new tasks.
Batch Normalization for Domain Adaptation and Domain Generalization Domain adaption suffers from distribution shift between source and target domains, which makes it sub-optimal to directly apply batch normalization (Bilen & Vedaldi, 2017). Li et al. (2016) proposed adaptive batch normalization to increase the generalization ability of a deep neural network. By modulating the statistical information of all batch normalization layers in the neural network. it achieves deep adaptation effects for domain- aaaativve tasks. Nado et al. (2020) noted the possibility of accessing small unlabeled batches of the shifted data just before prediction time. To improve model accuracy and calibration under covariate shift, they proposed prediction-time batch normalization. Since the activation statisties obtained during training do not reflect statistics of the test distribution, when lesting in an cut-of-distribution environment, Schneider et al. (2020) proposed estimating the batch statistics on the corrupted images. Kaku et al. (2020) demonstrated that standard non-adaptive feature normalization fails to correctly normalize the features of convolutional neural networks on held-out data where extraneous variables take values not seen during training. Learning domain- -specfffc batch normalization has been explored (Chang et al., 2019; Wang et al., 2019). Wang et al. (2019) introduced transferable normalization, TransNorm, which normalizes the feature representations from source and target domain separately using conmain-ppccffc statistics. Along similar vein, Chang et al. (2019) proposed a domain-specific batch normalization layer, which consists of two branches, each in charge of single domain exclusively. The hope is that, through the normalization, the feature representation will become domain invariant. Nevertheless, these normalization methods specifically designed for domain adaptation tasks, where data from target domains are available, though often unlabelled. This makes them inapplicable to domain generalization tasks where data from target domains are inaccessible at training time. Seo al. (2019) proposed learning to optimize domain specific normalization for domain generalization tasks. Under the meta- meta-learning settings, a mixture of different normalization techniques is optimized for each domain, where the mixture weights are learned specifically for different domains. Instead of combining different normalization statistics, MetaNorm learns from data to generate adaptive statistics specific to each domain. Moreover, we introduce even more challenging setting, i.c., few-shot domain generalization, which combines the challenges of few-shot classification and domain generalization.
Conditional Batch Normalization de Vries et al. (2017) proposed conditional batch normalization to modulate visual processing by predicting the scalars y and ,3 of the batch normalization conditioned on the language from an early processing stage. Conditional batch normalization has also been applied to align different data distributions for domain adaptation (Li et al., 2016). Oreshkin et al. (2018) applies conditional batch normalization to metric-based models for the few-shot classification task. of the
Tseng et al. (2020) proposed a carning-to-lean method to optimize the feature-wise transformation layers by conditional batch normalization for cross-domain classification. Unlike conditional batch normalization, we use extra data (the query set) to generate normalization statistics under the meta-learning setting, rather than the scalars.
3 METHODOLOGY
We view finding appropriate statistics for batch normalization as density estimation problem. We need to infer the distribution parameters, such as, μ and a when Gaussian distribution is presumed, as in existing batch normalization approaches. The motivation behind MetaNorm is to leverage the meta-learning setting and learn from data to generate adaptive normalization statisties. MetaNorm is generic and model-as aggossic,, addressing batch normalization in a unified way for different settings by minimizing the KI. divergence, which is a common metric to measure the difference between two
3

---

Published as a conference paper at ICLR 2021
probability distributions:
DKL (mm[((m)))ppmm))]
(3)
where m is a random variable that represents the distribution of activations, po(m) and (m)
are defined as Gaussian distributions with different implementations depending on the task of interest, e.g., few-shot classification or domain generalization, We leverage the amortized inference technique (Kingma & Welling, 2013) and implement this by inference networks. To be more specífic, for each individual channel in each convolutional layer, we infer the moments and a by (.) and Sá (.), respectively, which are realized as multi-layer perceptrons and we call hypernetworks (Ha et al., 2016). Hypernetworks use one network to generate the weights for another network. Our hypernetworks generate the statistics from data by using amortization techniques.
We simply incorporate the DKL, term into the optimization of the existing model with the cross-entropy loss LCE, resulting in a general loss function as follows:
L =
Lce ADRL (go(m))
(4)
where X > O is regularization hyper- parameter.
MetaNorm for Few-Shot Classification In the few-shot classification scenario, we define the C-way KCshot problem using the episodic formulation from (Vinyals et al., 2016). Each task T, is classification problem sampled from a task distribution p(T). The tasks are divided into a training meta-set T", validation meta-set T and test meta-ser T each with disjoint set of target classes (i.e., a class seen during testing is not seen during training). The validation meta-set is used for model selection, and the testing meta-set is used only for final evaluation. Each task instance T; ~ p (T) is composed of a support set S and query set Q, and only contains N classes randomly selected from the appropriate meta-set.
We aim to infer statistics from the support set that better match the query set. Therefore, we adopt a straightforward criterion for the inference:
DKL
where define (Sm (s))) and p(ml@)=N(xx=-0)) which are the distributions inferred from the support and query sets in a few-shot learning task. By minimizing the KL term in conjunction with the prime objective of a meta-learning algorithm, we are able to find the appropriate statistics from limited data samples for batch normalization. The KL term adheres to a closed form, which makes easy to implement and computationally efficient. The p(m Q) can be estimated by directly calculating statistics using the query set, which however performs inferior to inference by optimization. We note the inference from the query set only happens during meta-training time and we use the learned inference network to generate normalization statistics at meta-test time for test task using its support set.
To infer u.s. we deploy an inference function (+(() that takes activations of sample as input, and the outputs from all samples are then averaged as the final ps:
1 [S]
(a.),
As
S
where a, E RRwh the flattened vector of the activation map of the i-th sample in the support set, w is the width of activations, and h is the height of the activation map. To infer ds, we use the obtained us and deploy a separate inference function f-(:::
|
1
Σ (aa-
0s
It is worth mentioning that we actually use each sample infer the statistics and take the average of all inferred statistics as the final normalization statistics. This enables us to fully exploit the samples to generate more accurate statistics.
Note that the inference functions f.(() and f-()) are shared by different channels in the same layer and we will learn L pairs of those functions if we have I. convolutional layers in the meta-lea
4

---

Published as a conference paper at ICLR 2021
model. They are parameterized by ferd-forward multiple layer perception networks, which call hypernetworks. Using these hypemetworks, we generate support moments (jig, as)) and query moments (uQ. from the support and query sets, which are used for calculating the KL term in Eq. (5) for optimization during meta-training time. At meta-training time, we apply the statistics inferred from the support set for normalization of both support and query samples:
d'=77-00
d'=7 a Us
1 +C
+
(8)
where 7 and are jointly learned with parameters of the hypernetoorks meta-training time and directly applied at meta-test time, as in conventional batch normalization. At meta-test time, given a test task, use hypernetworks that take the support set as input to generate normalization statistics directly used for the query set.
MetaNorm for Domain Generalization
the domain generalization scenario, we adopt the meta- learning setting from (Li et al., 2018a; Balaji et al., 2018; Du et al., 2020), and divide a dataset into the source domains used for training and the target domains held out for testing. At meta-training time, data in the source domains is episodically divided into sets of meta-source D and meta-target D' domains.
In similar vein to few-shot classification, we would like to learn to acquire the ability to generate somain-specific statistics from a single example, which can then be applied to unseen domains. We assume generate reasonable normalization statistics by using only one sample from the new domain, because, intuitively, a single sample already carries sufficient domain information. We use a single example and all the examples in the same domain to infer the domain-specift statistics and minimize the KL term:
DEL go (m) aa:) (me
where we define g(mla.)=N(P,, oa), and (m)(D"aa)=-(( a similar way as Eq. (6) and Eq. (7), and a; is an example from its own domain D In both the meta-source and meta-target domains, each example is normalized using the statistics generated by itself, like in Eq. (8), in which we make y and /3 shared across all domains. The minimization of the KL term in Eq. (9) is to encourage the model to generate Nomain--spccife statisties for normalization from only single example. This enables us to generate domain-specific statistics on target domains that are never seen at meta-trainin meta-training time.
which are implemented in
In practice, we take the sum of all samples in all source domains as follows:
DD' ΣΣ
DKL [go(m a
aa))]
(10)
where Dj denotes the j-th of J meta-source domains. The inference networks are first at meta- training time learned and then directly used as examples from the target domain at meta-test time. Note that on the meta-targe domain we do not apply the KL term; instead, we simply rely on each example to generate its statistics for normalization.
MetaNorm for Few-Shot Domain Generalization We introduce an even more challenging setting, i.e., few-shot domain generalization, that combines the challenges of both few-shot classification and domain generalization. Specifically, we aim learn model from a of classification tasks, each of which has only a few samples in a support set for training and test the model on tasks in query set, which are in a different domain from the support set. Like few-shot classification, the label space is not shared between training and testing. Cross-domain few-shot learning has been explored recently by Tseng et al. (2020) and Guo et al. (2020). However, the setting of our few-shot domain generalization is different and considered to be more challenging, the support and query set are from different domains in the meta-test stage and the target domain is also unseen throughout the training stage. An example for the few-shot domain generalization setting is provided in Figure 1.
We divide a dataset into the source domains used for training and the target domains T held out for testing. During training time, data in the source domains S is episodically divided into sets of meta-train D* and meta-test D' domains. We sample C-way k-shot data as the support set from each meta-source domain Dd where & is the number of labelled examples for each of the classes. We
5

---

Published as a conference paper at ICLR 2021
3
?
D
D
Quackdrow
Clipurt
Infograph
Train
D
D
Sketch
Infograph
Quickdraw
Real
Intograph
Clipart
Quickhruw
Painting
Sketch
Figure 1: Illustration of the novel few-shot domain generalization scenario using the 5-way, 1-shot setting. The training in the upper box contains the meta-source domains DD and the meta-target domain D', which are from different domains. Each training task contains meta-source domains with five different classes and one example of each meta-source domain, and more than four examples for evaluation in the meta-target domain. The test set is defined in the same way but with all source domains S covering classes not present in any of the datasets in the training set, and more than four examples are used for evaluation in the target domain T.
sample C classes from the meta-test D' domain as the query set. At test time, we sample C-way k-shot data as the support set from each of the source domains The model learned at meta-training time is then fine-tuned on few-shot tasks samples from the source domains and tested on the target domain T. To learn the normalization statistics, we minimize the following KL term:
D
Σ
(11)
where a, is the activation associated with each sample from the meta-source domain D Likewise, q(ml aa)) and p(m| (D*) are also defined as factorized Gaussian distributions. We also adopt y and B which are shared across tasks and jointly learned. MetaNorm learns to acquire the ability to generate proper statistics for itself, and applies to the samples the meta-target domain.
EXPERIMENTAL RESULTS
We conduct an extensive set of experiments total of 17 datasets containing more than 15 million images. We use three representative approaches to meeaallerningg as our base models, i.e., MAML (Finn et al., 2017), ProtoNets (Snell et al., 2017), and VERSA (Gordon et al., 2019), which can verify our MetaNorm is generic, flexible and model-agnostic, making it simple plug-and-play module that is seamlessly embedded into existing meta-learning approaches. We further compare different normalization methods: transductive batch normalization (TBN), "example" that denotes testing with one example at time by using TBN, "class" that denotes testing with one class time by using TBN, w/o BN which is not using batch normalization, CBN which is using conventional batch normalization, RN (Nichol et al., 2018), MetaBN (Bronskill et al., 2020), TaskNorm-L (Bronskill et al., 2020), and TaskNorm-1 (Bronskill et al., 2020). All details about datasets and implementation settings are provided in the appendix. More experimental results, including convergence analysis, are also provided in the appendix. Our code will be publicly released.
Effect of KL Term We first conduct ablation studies that measure the effectiveness of MetaNorm. The key of MetaNorm is the introduced KL term for learning to learn statistics. We test the per- formance of MetaNorm without the KL term by directly using the statistics generated from data.
httpp:://i thub. com YYUJ-AI /VetaNorm.

---

Published as a conference paper at ICLR 2021
Table 1: Effect of KL Term in MetaNorm for few-shot classification with MAML (Finn & Levine, 2018) on minilmageNet and domain generalization on PACS with ResNet-18. More few-shot classification results with ProtoNets (Snell et al., 2017) and VERSA (Gordon et al., 2019), as well as domain generalization results Office-Home are provided in the appendix. Best performing methods and any other runs within the 95% confidence margin in bold. The KL term crucial.
Fhw- classification
Domain generalization
MetaNorm
5-way, 1-shot
5-way, 5-shot
Photo
Art
Cartoon
Sketch
Mean
w/o KL
34.3 1.5
50.7 0.8
88.96
71.25
65.37
69.23
73.72 85.70
w/ KL.
46.8 + 1.6
60.1 0.8
95.99
85.01
78.63
83.17
Fenatet Chocl (freay, tehe)
Howarerr Cliss Busioo threey S-tta
120
Donen Opearokotion
Accuecy (9
Acaurcy (%)
70-
TP4
100 50
100
612
[Q]
Figure 2: Impact of Target Set Size. The performance increases for larger target sets and plateaus at around 125 for few-shot classification on minilmageNet and around 256 for domain generalization on PACS. TBN here is based on VERSA. MetaNorm generates proper normalization statistics with a reasonable batch size.
In this case, also use the hypernetworks to generate the moments, μ and by simply removing the KL term in the objective function. In Table we present results for few-shot classification on minilmageNet (Vinyals et al., 2016) and for domain generalization on PACS (Li et al., 2017a). The performance of MetaNorm without KL degrades significantly. This expected, as without the KL term the generation process of normalization statistics lacks direct supervision from the target distribution, resulting in improper statistics.
Impact of Target Set Size The other key parameter in MetaNorm the size of the target set; that is, the number [Q] of samples in the query set (in few-shot classification) and the number 'D' of samples in each domain (in domain generalization). This parameter important when learning normalization statistics because we use the statistics generated by the target set as the 'ground truth'. We evaluate its impact on the performance of MetaNorm in Figure 2. The experimental results show that TBN is not affected by the target size, both in the 5-way, 1-shot and 5 way, 5-shot tasks. MetaNorm performance rises as the size of the target set increases and plateaus at a reasonable size. In the few-shot setting, the performance reaches its peak at size of about 125, which is slightly larger than the standard size of 75, while in the domain generalization setting, the performance plateaus at a size of about 128. This demonstrates that we able generate proper statisties with the mini-bater gradient descent optimization. In scenarios demanding a very small target set size, we could leverage image synthesis techniques generate more samples for the targets sets.
Sensitivity to Algorithm We evaluate MetaNorm using the MAML (Finn et al., 2017). ProtoNets (Snell et al., 2017) and VERSA (Gordon et al., 2019) algorithms, which are representative gradient, metric and model based meta-learning approaches for few-shot classification. These experiments are conducted on the Omniglot and minilmageNet datasets under different settings. The comparison results on minilmageNNt are summarized in Table 2 and the results on Omniglot are provided in the appendix. For all three meta- learning approaches under all settings, MetaNorm consistently achieves comparable performance both to the -rnnuuanndee and transductive normalization methods. Being non -on-transductive, TaskNorm can achieve impressive performance on all the tasks, but its perform- ance is not always better than transductive batch normalization. MetaNorm achieves comparable
7

---

Published as a conference paper at ICLR 2021
Table 2: Sensitivity to Algorithm. Few-shot results on minilmageNet using different algorithms.
Results on Omniglot are provided in the appendix. Best performing methods and any other runs within the 95% confidence margin in bold. Transductive results indicated above dashed line. MetaNorm is consistent top-performer, regardless of the meta-learning algorithm.
ProtoNets'
MAML
VERSA
way,
I-shot
way. shot
5 way,
I shot
5 way. S-shol
5 way, I I-shot
5 way, 5 shol
TBN
45,9 0:06
65,5 #0
45.5 #1
59.7 2.09
53.4 #1.5
67.3 =0.
example
43,9
60.1
26.9
+07
class
43.1 18
59.8 0:
26.9 1:
27.2 #.66
43.8 1.
$9.7 06
44.1
60.1
CBN (loffe& & Szegedy, 2015)
47.8 1:06
66.7 0.5
20.1 1:00
20.2 1.02
45.7 1.44
60.7 1:03
(Nichol al. 2018)
39.7
63.1
MetaBN (Bronskill et a 2020)
42.6 1:06
64.6 a:05
41.6
58.6 1:09
50.1 1:1
65.8 9:09
'issNNrm-L (Bronskill al. 2020)
47,5
65.3
"TasKNorm- I (Bronsklll et al., 2020)
43.2 1:06
63.9 1:05
42.4 1:12
58.7 1009
52.9 1.
66.5 :08
MetaNorm g 75)
47.3 1.06
65.4 1.05
1.1
1
MetaNorm ( @ 125)
48.1 au
65.9 0.
46.8
1.6
60.1 1.0
53.7 1.44
68.1 0.
Results for MAML and ProtoNets (except w/o BN) provided by (Bronskill al, 2020), and VHRSA with TBN provided by (Gordon et il., 2019), All other results based on our ro implementation..
performance transductive batch normalization, especially under the 5-way-I-shot setting, which is challenging since only a few examples are available to generate statistics. Notice that, MetaNorm performs well with the standard query set size [Q] of 75 (15 per category). It is slightly better than non- -rnntransductive TaskNorm and comparable with TBN. MetaNorm achieves its best performance with query size [2] of 125 (25 per category), only slightly larger than the standard size of 75. This demonstrates the benefit of leveraging meta-learning by MetaNorm for batch normalization. We conclude that MetaNorm is general and serves plug-and-play module for existing meta-learning models to improve their performance.
Sensitivity Dataset We evaluate MetaNorm on demanding few-shot classification challenge called Meta-Dataset (Triantafillou et al., 2020), which com- posed of thirteen image classification datasets (eight for training, five testing). To compare with previous work, perform experiments with ProtoNets and re- port the results in Table 3. All thirteen per-dataset results can be found in the appendix. MetaNorm achieves high performance in terms of average rank, with highest accuracy on eight of the thirteen datasets. MetaNorm outperforms transductive batch normaliza- tion on eleven datasets. It achieves comparable perform- ance with transductive batch normalization on Omni- glot and MNIST, which are relatively less challenging. Moreover, MetaNorm performs better than TarkNorm on seven of the thirteen datasets. We conclude that MetaNorm is effective, outperforming alternative nor- malizations for most datasets.
Table 3: Sensitivity to Dataset. Few-shot classification on Meta-Datase using Pro- toNets. MetaNorm performs best overall.
Wins
Rank
IN
10.61
CBN
9.11
LN
3
8.19
TaskNorm-r
7.88
BRN
1
6.23
TBN
2
4.81
MetaBN
4
4.78
RN
3
4.73
TaskNorm-L
4
4.19
TaskNorm-I
6
3.07
MetaNorm
10
2.35
Sensitivity to Domains For this experiment adopt two widely-used benchmarks for domain generalization of visual object recognition, i.e., PACS (Li et al., 2017a) and Office-Home (Ven- kateswara al., 2017). Detailed descriptions on the experimental settings and implementations provided in the appendix. For fair comparison with prior methods (Balaji et al., 2018; Li et al., 2018b; Sco al., 2019), we employ ResNet-18 as the backbone network in all experiments. As shown in Table 4. MetaNorm achieves the best performance on PACS and Office-Home in terms of average accuracy. On PACS, MetaNorm consistently outperforms other normalization approaches including domain-specific normalization (Seo et al., 2019), on all four domains. It is worth mentioning that the baseline normalization uses the statisties from the source domains for the batch normalization of the target domain. As expected, the baseline method produces relatively poor performance on most domains, since the source domains cannot provide proper statistics for target domains due to the distribution shift. We have also done experiment using standard batch normalization. In the training stage, we compute the ground truth statistics using all the test data on the meta-target domain D' instead of using inferred statistics p(m] \ MetaNorm is still better on most domains and on
8

---

Published as a conference paper at ICLR 2021
Table 4: Sensitivity Domains. Performance comparison on domain generalization. MetaNorm
consistently achieves the best performance among all normalization methods.
PACS
Office-Home
Photo
Art
Cartoon
Sketch
Mean
Art
Clipart
Product
Real World
Mean
Baseline
95.87
78.47
70.41
70.68
78.86
58.71
44.20
71.75
73.19
61.96
IBN (Pan et al., 2018)
92.04
75.29
72.95
77.42
79.43
55.41
44.82
68.28
71.95
60.09
DSBN (Chang et al., 2019)
95.51
78.61
66.17
70.15
77.61
59.04
45.02
72.67
71.98
62.18
SN (Luo et al., 2018a)
93.47
82.50
76.80
80.77
83,38
54.10
44.97
64.54
71.40
58.75
DSON (Seo al., 2019)
95.87
84.67
77.65
82.23
85.11
59.37
45.70
71.84
74.68
62.90
Ground truth statistics
95.78
85.17
78.15
82.91
85.50
59.35
46.12
72.77
75.08
63.33
MetaNorm
95.99
85.01
78.63
83.17
85.70
59.77
45.98
73.13
75.29
63.55
average. This is reasonable because ground truth statistics from the test data do not necessarily reflect the true data distribution. The experimental results demonstrate MetaNorm can generate reasonable normalization statistics from only one sample in its domain. We conclude that MetaNorm is effective for domain generalization.
Few-Shot Domain Generalization In our final experiment, adopt the DomainNet dataset (Peng et al., 2019) and introduce a new, more challenging setting to evaluate the performance for few-shot domain generalization. Detailed descriptions on the dataset and experimental settings are provided in the appendix. We conduct the experiments with the MAMI. and ProtoNets algorithms under both 5-way 1-shot and 5-way 5-shot settings, and the results are reported in Table 5. We implement transductive batch normalization, MetaBN and the variants of TaskNorm for direct comparison. Under both settings, our MetaNorm produces the best performance and surpasses the transductive batch normalization by large margins of up to 4.0% on the challenging 5-way 1-shot setting with MAML. MetaNorm also achieves better results than the non-transductive TaskNorm approaches. At the same time, with ProtoNet our MetaNorm again consistently delivers the best performance and surpasses both transductive and non-transductiie normalizations, The performance on the challenging few-shot domain generalization scenario with different meta-lcarning algorithms again demonstrates the effectiveness of MetaNorm handling the challenges of batch normalization for small batches and across domains.
Table 5: Few-Shot Domaín Generalization. Comparison with different normalizations using MAML and ProtoNets on the Few-shot DomainNet dataset. Best performing methods and any other runs within the 95% confidence margin denoted in bold. Reported results use "Painting" the target domain, all based on our implementations. MetaNorm consistently achieves top performance.
MAML
ProtoNets
5-way, 1-shot
5-way, 5-shot
5-way,
I-shot
5-way, 5-shot
TBN
28.7 1.8
49.3 1
27.9
47.1 +:08
w/o BN
23.5
+
23.8 18
45.9 0.
CBN
20.0 = 0.0
20.1 02
28.4
47.9
MetaBN
24.7 + 1.6
46.1 +
25.1 = 1.8
46.1 0.08
TaskNorm-I.
2..9
47.4
29.5 1.6
48.3 +.08
TaskNorm-1
27.5
26.8 1.8
46.9 0.
MetaNorm
32.7 1.7
51.9 1.09
30.7 1.
0.9
5
CONCLUSION
In this paper we present MetaNorm, a meta-learning based batch normalization. MetaNorm tackles the challenging scenarios where the batch size is too small to produce sufficient statistics or when training statistics are not directly applicable to test data due to domain shift. MetaNorm learns to learn adaptive statistics that are specific to tasks domains. It is generic and model-agnostic, which enables it to be used with various meta-learning algorithms for different applications. We evaluate MetaNorm on two well-known existing tasks, i.c., few-shot classification and domain generalization, and we also introduce the challenging evaluation scenario of few-shot domain generalization that addresses the small batch and distribution shift problems simultaneously. An extensive evaluation on 17 datasets reveals that MetaNorm consistently achieves results that are better, at least competitive, compared to other normalization approaches, verifying its effectiveness as new meta-learning based batch normalization approach.

---

Published as a conference paper at ICLR 2021
REFERENCES
Kelsey Allen, Evan Shelhamer, Hanul Shin, and Joshua Tenenbaum. Infinite mixture prototypes for few-shot learning. International Conference on Machine Learning, pp. 231-241. PMLR, 2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint
arXiv:1607.064400 2016. 1, 22
Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. MetaReg: Towards domain gener- alization using meta-regularization. In Advances in Neural Information Processing Systems, pp. 998-1008, 2018. 2, 5,
Hakan Bilen and Andrea Vedaldi. Universal representations: The missing link between faces, text, planktons, and cat breeds. arXiv preprint arXiv: 2017.
Nils Bjorck, Carla Gomes, Bart Selman, and Kilian Q Weinberger. Understanding batch normaliza- tion. In Advances in Neural Information Processing Systems, pp. 7694-7705, 2018.
John Bronskill, Jonathan Gordon, James Requeima, Sebastian Nowozin, and Richard Turner. TaskNorm: Rethinking batch normalization for meta-learning In International Conference on Machine Learning, pp. 4683-4694. 2020. 2, 3, 6, 8, 18, 20,
Fabio Maria Carlucci, Antonio D'Innocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Domain generalization by solving jigsaw puzzles. In IEEE Conference on Computer Vision and
Pattern Recognition, 2019. 15
Woong-Gi Gi Chang, Tackgeun You, Seonguk Seo, Suha Kwak, and Bohyung Han. Domain-specific batch normalization for unsupervised domain adaptation. In IEEE Conference on Computer Vision and Puttern Recognition, pp. 7354-7366 2019. 3,
Mircea Cimpoi. Subhransu Maji. lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In IEEE Conference on Computer Vision and Pattern Recognition, 2014. 16
Harm de Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron C Courville. Modulating early visual processing by language. In Advances in Neural Information Processing Systems, pp. 6594-6604, 2017.
Yingjun Du, Jun Xu, Huan Xiong, Qiang Qiu, Xiantong Zhen, Cees G.M. Snoek, and Ling Shao. Learning to learn with variational information bottleneck for domain generalization. In European Conference on Computer Vision, pp. 200-216. Springer, 2020. 5
Abdul Fatir. A re-implementationn of "prototypical networks for few-shot learning". https:
// //github.
2018. 18
Chelsea Finn. Code for "Model-agnostic meta-learning for fast adaptation of deep networks".
https:/ ggtptbb..bb/.g./ggg/ www.ua...oo///// woeffinnmmaatn 2017. 17
Chelsea Finn and Sergey Levine. Meta-learning and universality: Deep representations and gradi- ent descent can approximate any learning algorithm. In International Conference on Learning Representations, 2018. 7, 23
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pp. 1126-1135, 2017. 1, 2, 6,
Jonathan Gordon. Code for "Meta-leaming probabilistic inference for prediction". https: :
github. www.c/o//oo.o.. Gorm//ooddojjooo /eeees 2019. 18
Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard Tumnr. Meta- learning probabilistic inference for prediction. In International Conference on Learning Represent- ations, 2019. 2, 6, 7, 8, 19, 20
10

---

Published as a conference paper at ICLR 2021
Yunhui Guo, Noel C Codella, Leonid Karlinsky, James V Codella, John R Smith, Kate Saenko, Tajana Rosing, and Rogerio Feris. broader study of cross-domain few-shot learning. In European Conference Computer Vision, 2020.
David Ha, Andrew Dai, and Quoe V Le. Hypernetworks. arXiv preprini arXiv: 1609009160 2016.
Sebastian Houben, Johannes Stallkamp, Jan Salmen, Mare Schlipsing, and Christian Igel. Detection of traffic signs in weal-wordd images: The German traffic sign detection benchmark, In International joint conference on neural networks, pp. 1-8, 2013. 16
Sergey loffe. Batch renormalization: Towards reducing minibatch dependence in batch coacc-hoommalzed models. In Advances in Neural Information Processing Systems. pp. 1945-1953. 2017. 22
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. International Conference on Machine Learning, pp. 448-456 2015. 1, 2, 8, 20,22 22
Songhao Jia, Ding-Jic Chen, and Ewann-Tzong Chen. Instance-level meta normalization. In IEEE Conference Computer Vision and Pattern Recognition, pp. 4865-4873, 2019.
Aakash Kaku, Sreyas Mohan, Avinash Parnandi, Heidi Schambra, and Carlos Semaandz-GGrande Be like water: Robustness to extraneous variables via adaptive feature normalization. arXiv preprint arXiv:2002.04013 2020. 3
Diederik P Kingma and Max Welling. Auto-encoding variational bayes.
arXiv preprint
arXiv:1312.6114 2013. 4
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 1332-1338, 2015. 15, 16
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In IEEE International Conference on Computer Vision, pp. 5542-5550, 2017a. 2, 7,8,1
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Learning to generalize: Meta- learning for domain generalization. arXiv preprint 1rXiv:1710.03463, 2017b.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Learning to generalize: Meta- learning for domain generalization. In AAAI Conference on Artificial Intelligence, 2018a.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. IEEE Conference on Computer Vision and Pattern Recognition, pp. 5400-5409, 2018b. 8, 15
Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou. Revisiting batch normalization for practical domain adaptation. arXiv preprint arXiv:1603.044 04779, 2016.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft COCO: Common objects in context. In European Conference on Computer Vision, pp. 740-755, 2014. 16
Ping Luo, Jiamin Ren, Zhanglin Peng, Ruimao Zhang, and Jingyu Li. Differentiable learning-to- normalize via switchable normalization. arXiv preprint arXiv: 1806. 10779, 2018a. 1,9
Ping Luo, Xinjiang Wang, Wenqi Shao, and Zhanglin Peng. Towards understanding regularization in batch normalization. arXiv preprint
2018b.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained
visual classification of aircraft. Technical report, 2013. 16
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant
feature representation. In International Conference on Machine Learning, pp. 10-18, 2013.
11

---

Published as a conference paper at ICLR 2021
Zachary Nado, Shreyas Padhy, D Sculley, Alexander D' Amour, Balaji Lakshminaryyanan, and Jasper Snoek. Evaluating prediction time batch normalization for robustness under covariate shift. arXiv preprint 10XYv::00661096633 2020. 3
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta- meta-learnnng algorithms. arXiv preprint arXiv:
2018. 6, 8, 20, 22
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification large number of classes. In Indian Conference on Computer Vision, Graphics & Image Processing, pp. 722-729,
2008. 16
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. Advances in Neural Information Processing Systems, 31: 721-731, 2018.
Xingang Pan, Ping Luo, Jianping Shi, and Xiaoou Tang. Two at once: Enhancing learning and generalization capacities via ibn-net. European Conference on Computer Vision, pp. 464 479, 2018. 9
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In JEEE International Conference on Computer Vision, pp- 1406-1415, 2019. 9, 15
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In International Conference Learning Representations, 2017. 15
James Requeima, Jonathan Gordon, John Bronskill, Scbastian Nowozin, and Richard E Turner. Code for "Fast and flexible multi-task classification using conditional neural adaptive processes", 2019. 18
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-252. 2015. 16
Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. Advances in Neural Information Processing Systems, 33, 2020.
Seonguk Seo, Yumin Suh, Dongwan Kim, Jongwoo Han, and Bohyung Han. Learning to optimize domain specific normalization for domain generalization. arXiv preprint arXiv: 042755 2019.
2, 3, 8, 9
Jake Snell, Kevin Swersky, and Richard Zeme. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, pp. 4077-4087. 2017. 2, 6, 7, 19
Cecilia Summers and Michael Dinneen. Four things everyone should know to improve batch normalization. In International Conference on Learning Representations, 2020.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta- dataset: A dataset of datasets for learning to learn from few examples. In International Conference on Learning Representations, 2020. 2, 8, 16, 18
Hung- Yu Tseng, Hsin- King Lee, Jia-Bin Huang, and classification via learned fcature-wise transformation. arXiv preprint arXhv:2001.08735. 202. 3,
Yang. Cross-domain few-shot
Dmitry Ulyanov, Andrea Vedaldi. and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022, 2016. 1,22
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. IEEE Conference on Computer Vision and Pattern Recognition, 2017. 8, 15
12

---

Published as a conference paper at ICLR 2021
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. In Advances in Neural Information Processing Systems, pp. 3630-3638. 2016. 4, 14, 15
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech- UCSD Birds-200-2011 Dataset. Technical report, 2011. 16
Ximei Wang. Ying Jin, Mingsheng Long, Jianmin Wang, and Michael I Jordan. Transferable normalization: Towards improving transferability of deep neural networks. In Advances Neural Information Processing Systems, pp. 1953-1963, 2019. 2,
Yuxin Wu and Kaiming He. Group normalization. In European Conference Computer Vision, pp-
3-19, 2018. 1,3
Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, and Samuel S Schoenholz. A mean field theory of batch normalization. arXiv preprint arXiv:1902.08129. 2019.
Xiantong Zhen, Yingjun Du, Huan Xiong, Qiang Qiu, Cees G.M. Snoek, and Ling Shao. Learning to learn variational semantic memory. Advances in Neural Information Processing Systems, 2020a.
Xiantong Zhen, Haoliang Sun, Yingjun Du, Jun Xu, Yilong Yin, Ling Shao, and Cees Snoek. Learning to learn kernels with variational random features. In International Conference on Machine Learning, pp. 11409-1 1419. PMLR, 2020b. 2
13

---

Published as a conference paper at ICLR 2021
A ALGORITHMS DESCRIPTIONS
In this Appendix we provide the detailed MetaNorm algorithm descriptions to conduct batch normal- ization for few-shot classification (Algorithm 1), domain generalization (Algorithm and few-shot domain generalization (Algorithm 3). The dataflow of the implementation is shown in Figure 3.
Algorithm MetaNorm for Few-Shot Classification
Meta-train:
Input values over support set as,; and query set aqui 3 Initialize parameters.
[8]
12
jss [ST Σ (as.)): pa = 15
Σ
(aq.)):
[s]
|0|
08 [sT Σ
fo ((as,) ps)) -
faqq - =uq)²);
1=1
ds,; = 7
+ 3: do. =
- Vot+c
+ 3:
Votte
ΣKL. DRL (s) ae)]
return dg., MetaNorm (as,;); de. (letaNorm(aq.)) LXe
Meta-test: Input values of over support set as,; and query ag,
[8
Σ H (as.)):
Hs
M
0s = [s]
fo ((as.) - )ss)));
1=1
do.: =7
"Q.1-u 3: Vag++
return 0Q.1 MetaNorm(aq.a)
Algorithm 2 MetaNorm for Domain Generalization
Train: Input values of a over meta-source domain as,; and meta-target domain ar.;; 8 + Initialize parameters.
Ms.. (ass)): ps
Σ
f (as.);;
[8]
Ss = L ((as.: () = া
Σ
1e ((as, - 1s))))
pr, (ar,)): oT, =/6 [o((a,, -urt,,:-r,,;)))) )rr;))):
dr. = 7 ATa-KTT + B3
Vot.
LKL = DKL [M(ss,, (l:) os)]
return a/y,a MetaNorm(ar..) Lxi.
Test: Input values of a over test domain a; (a.); 0: (a - )))
d, = Va? + 9: return d MetaNorm(a.)
B DATASETS
We conduct an extensive set of experiments on a total 17 datasets containing more than 15 million images. All dataset details and settings provided in this Appendix.
minilmageNett. The minilmageNet is originally proposed (Vinyals et al., 2016) and has been widely used for evaluating few-shot learning algorithms. It consists of 60,000 color images from 100
14

---

Published as a conference paper at ICLR 2021
Algorithm 3 MetaNorm for Few-Shot Domain Generalization
Input values of over meta-source domain set ag,i and meta-target domain set aQ.: 7: Initialize parameters.
Meta-train:
[8]
(as,)): pa = -ē Σ
[a
Ms 13T
0s 181 2
((ss. -ss)): = |0
[( ((a)))
dg
B: d0.. =
"9.1
+ 8::
Vog+. +
Voz
LK.. = DRt. os)s ra)]
as
return ds MetaNorm (as,)): de.. MetaNorm (aq.s): EKL
Meta-test: Input values of over support set as, and query set aq.6
[8] (ass
1s
Σ
is]
08 [S]
Σ
f/ ((as.) )=))²):
do.i 77
+ 3:
Vo
return dq.+
MetaNorm(aq..)
NN..HWW -PSC
Support:
XXCHIW
fz
&xc
ussc
fe
XH...W y(
)
1330 +
(N of (u)
XXCCH.W -832
Query:
XMCH-W
f/2
AMC
Ho
f
Ki...WW 7( The +
+
Figure 3: The dataflow of the implementation for few-shot learning. "N" indicates support size, "M" indicates query size, "C" indicates the channel of activations, "W" indicates the width of activations, "H" indicates the height of activations.
classes with 600 examples per class. The images have dimensions of 84x pixels. We follow the train/val/ test split introduced in (Ravi & Larochelle, 2017), which uses 64 classes for meta-training, 16 classes for meta-validation, and the remaining 20 classes for meta-testing.
Omniglot. Omniglot (Lake al., 2015) is few-shot learning dataset consisting of 1,623 handwritten characters (each with 20 instances) derived from 50 alphabets. We follow the pre-processing and training procedure defined in (Vinyals et al., 2016). We resize images 28x28. The training, validation and test sets consist of random split of 1,100, 100, and 423 characters.
PACS (Li et al., 2017a) contains total of 9,991 images of the size 224x224 from 4 domains, iee, photo, art-painting, cartoon and sketch, which demonstrate huge domain gaps. Images are from object classes, i.e., dog, elephant, giraffe, guitar, horse, house, and person. We follow the "leave-one-out" protocol in (Li et al., 2017a; 2018b; Carlucci et al., 2019), where the model is trained on any three of the four domains, which we call source domains, and tested on the last (target) domain. The val-tstl splits are the same as in (Li al., 2017a).
Office- Home (Venkateswara et al., 2017) also has domains: art, product, clipart and real-worid. For each domain, the dataset contains images of 65 object categories found typically in office and home settings. We use the same experimental protocol as for PACS.
NomainNet (Peng al., 2019) contains 6 distinct domains, i.c., clipart, infograph, painting, quick- draw, real, and sketch for 345 categories. The categories are from 24 divisions, which are: Furniture, Mammal, Tool, Cloth, Electricity, Building, Office, Human Baby, Road Transportation, Food, Nature,
15

---

Published as a conference paper at ICLR 2021
Cold Blooded, Music, Fruit, Sport, Tree, Bird, Vegetable, Shape, Kitchen, Water Transportation, Sky Transportation, Insect, Others.
Meta-Dataset (Triantafillou et al., 2020) is composed of ten (eight train, two test) existing image clas- sification datasets. These are: ILSVRC- 2012 (ImageNet, (Russakovsky et al., 2015)), Omniglot (Lake et al., 2015), Aircraft (Maji al., 2013), CUB-200-2011 (Birds, (Wah et al., 201 2011)), Describable Textures (Cimpoi et al. 2014), Quick Draw, Fungi, VGG Flowr (Nilsback & Zisserman, 2008), Traffic Signs (Houben et al., 2013) and MSCOCO (Lin et al., 2014). Each episode generated in Meta-Dataset uses classes from single dataset. Two of these datasets, Truffic Signs and MSCOCO, are fully reserved for evaluation, means no classes from these sets participate in the training set. Except for Traffic Signs and MSCOCO, the remaining datasets contribute some classes to each of training, validation and test splits of classes. There are about 14 million images in total in Meta-Dataset.
C FEW-SHOT DOMAINNET
To construct Few-shot DomainNet, we chose 200 random classes from DomainNet and used 1 140 for training. 20 for validation and the last 40 for testing. Note that the last 40 object classes were never seen during training. The dataset consists of 200,000 colour images of size 84 84x84 with each of the 200 classes having 1,000 examples. Please see Table 6, Table and Table 8 for training, validation, and test classes.
Table 6: Training classes of Few-shot DomainNet
Furniture: bathtub, ceiling fan, couch, fence, hot tub, mailbox
Mammal: tiger, rhinoceros. bat, cat, lion, panda
Tool: anvil, basket, broom, sword, pliers
Cloth: belt, camouflage, eyeglasses, crown, bowtic
Electricity: calculator, computer, camera, cooler, dishwasher
Building: bridge, jail, pool, tent, castle
Office: alarm clock, binoculars, backpack, book, bandage
Human Body: arm, ear, face, beard, elbow, finger, brain, eye, foot, knee
Road Transportation: ambulance, hus motorbike, hicycle, train
Food: birthday cake, cookie, hot dog, peanut, sandwich, bread, donut, pizza, steak, lollipop
Nature: beach, lightning, ocean, river, sun, cloud, moon, rain, tornado
Cold Blooded: crab, frog, crocodile, lobster, fish, octopus, shark
Music: cello, guitar, saxophone, violin, clarinet, harp, trombone
Fruit: apple, banana, blackberry, blueberry, grapes, pear
Sport: baseball, baseball bat, basketball, snorkel, yoga, tennis racquet
Tree: bush, grass, cactus, tree, flower
Bird: bird, owl
Vegetable: asparagus, braccoli, carrot, mushroom, onion
Shape: circle, hexagon
Kitchen: fork, frying pan. hourglass, knife, lighter
Water Transportation: aircraft carrier, canoe, cruise ship, submarinc
Sky Transportation: airplanc, helicopter
Insect: ant, bee
Others: angel, cannon, dragon, mermaid, stop sign, snowman, feather
16

---

Published as a conference paper at ICLR 2021
Table 7: Validation classes of Few-shot DomainNet
Furniture: stairs, ladder
Mammal: monkey
Tool: paint can
Cloth: purse, t-shirt
Electricity: radio
Building: pond
Office: nail
Human Body: skull, tooth
Road Transportation: firetruck
Food:
Nature: star, hurricane
Cold Blooded: sea turtie
Music:
Fruit: strawberry
Sport: hockey stick
Tree:
Bird: penguin
Vegetable:
Shape:
Kitchen: wine bottle
Water Transportation:
Sky Transportation:
Insect:
Others: teddy-bear
D IMPLEMENTATION DETAILS.
In the few-shot learning task, MAML and ProtoNets use a simple CNN containing 4 convolutional layers, each of which is 3x3 convolution with 32 filters, followed by MetaNorm, ReLU non- linearity, and finally 2 max max-pooling. VERSA uses CNN containing convolutional layers, each of which is 3x3 convolution with 64 filters, followed by MetaNorm, ReLU non-linearity, and finally a 2x2 max- max-pooling.
the domain generalization task, we rely on ResNet- 18 as backbone for fair comparison with previous work. Each convolutional layer is followed by MetaNorm. The hypernetwork is a 3-layer MLP with 128 units per layer and rectifier nonlinearitics. We implemented NVIDIA Tesla V100. All code will be www.cooo///////////// YDU-AI AABO0AAA/MMtttt Norrm.
all models in the Tensorflow framework and tested available at:
com/
D. MAML EXPERIMENTS
For MAML experiments, we used the codebase by Finn (Finn, 2017). We use the Adam optimizer with default parameters, and a meta batch size of tasks. The number of test episodes set as 600. The number of training iterations is 60,000. We set 0=0.001. The other byper-parameters the default MAML parameters. No early stopping was used. We used the first-order approximation of MAML for the experiments.
17

---

Published as a conference paper at ICLR 2021
Table 8: Test classes of Few-shot DomainNet
Furniture: teapot, toothpaste, stove, umbrella
Mammul: mouse,
Tool: bucket, paint can
Cloth: sweater, shoc, flip flops
Electricity; television, stereo, toaster, flashlight
Building: waterslide, garden
Office: map, clock, calendar, scissors
Human Body: finger, nose, toe
Road Tronsportation: bulldozer
Food: peanut
Nature: mountain, sun
Cold Blooded: lobster, scorpion
Music: harp
Fruit: pineapple
Sport: soccer ball, hockey stick
Tree: house plant, leaf
Bird: swan
Vegetable: string hean
Shape: squiggle
Kitchen:
Water Transportation:
Sky Transportation:
Insect:
Others: feather, snowman
D.2 PROTONETS EXPERIMENTS
For ProtoNets, we used the codebase by Fatir (Fatir, 2018). For minilmageNett, we used the following ProtoNets options: learning rate of 0.001, 60,000 training iterations, 200 validation episodes, 600 test episodes and >=0.0001. We choose the units of hidden layers and À by cross-validation. For Meta-Dataset, we reproduce the code provided by CNAPS (Requeima et al., 2019) with TensorFlow. We simply replace its normalization method with our MetaNorm method and add the KL term to the final loss. We are consistent with the dataset configuration and follow the training process as specified in (Triantafillou et al., 2020). The number of training iterations is 80,000. We use constant learning rate of 0.0001. We set >=0.001. We follow TaskNorm's (Bronskill et al., 2020) options: they do not use feature adaptation, and allow updates pre-trained feature extractor weights during meta- traiitrriiinng stage.
D.3 VERSA EXPERIMENTS
For VERSA, used the codebase by Gordon (Gordon, 2019), For the 5-way 5-shot model, we train using the setting of 8 tasks per batch for 100,000 iterations and use constant learning rate of 0.0001, A= 0.001. For the 5-way 1-shot model, train with the setting of 8 tasks per batch for 150,000 iterations and use a constant learning rate of 0.00025, >=0.01. We set validation episodes as 200, and test episodes 600. The units of hidden layers and À were chosen by validaain.....
18

---

Published as a conference paper at ICLR 2021
Table 9: Inference function (a)()
Output size
Layers
h
Input flattened vector of the activation map fully connected, ELU
128
128
fully connected, ELU
x h
fully connected to μr
Table 10: Inference function f' (-)
Output size
Layers
h
Input flattened vector of the activation map and fully connected, ELU fully connected, ELU fully connected to o
128
128
W
E EXTRA RESULTS FOR EFFECT OF KL TERM
In this Appendix we consider extra results for the ablation on measuring the effect of the KL term. We report results for few-shot classification minilmageNet with ProtoNets (Snell et al., 2017) and VERSA (Gordon et al., 2019) in Table 11. We also report domain generalization results on Office-Homee in Table 12. In all cases the KL term is crucial.
Table 11: Effect of KL Term in MetaNorm for few-shot classification on minilmageNee with ProtoNets and VERSA. Best performing methods and any other within 95% confidence margin denoted in bold.
ProtoNets
VERSA
MetaNorm
5-way, 1-shot
5-way, 5-shot
5-way, 1-shot
5-way, 5-shot
w/o KL. w/ KL
40.1
58.7 1: 0.88
48.7 4: 1.6
64.3 0.8
48.1
65.9 1 0.9
53.7 1: 1.6
68.1 0.8
Table 12: Effect of KL Term in MetaNorm for domain generalization on Office-Home
Office-Home
MetaNorm
Art
Clipart
Product
Real-World
Mean
w/o KL
51.25
39.27
69.75
68.19 75.29
57.12 63.55
w/ KL
59.77
45.98
73.13
F SENSITIVITY TO ALGORITHM ON OMNIGLOT
The experiments Omniglot for few-shot classification under the meta-learning settings of MAML, VERSA and ProtoNets are reported in Tables 13, 14 and 15. MetaNorm consistently outperforms both transductive and non-transductive normalization approaches.
G SENSITIVITY TO DATASET
The complete of results for each of the thirteen datasets in Meta-Dataset are provided in Table 16.
19

---

Published as a conference paper at ICLR 2021
Table 13: Sensitivity to Algorithm. Few-shot results on Omniglot using MAML. Best performing methods and any other runs within the 95% confidence margin in bold. Transductive results indicated above dashed line.
Omniglot
5-way, 1-shot
5-way, 5-shot
20-way, 1-shot
20-way, 5-shot
TBN
98.4
99.2 = 0.2
90.9 + 0.5
96.6 + 0.2
example
21.6
22.0 + 0.5
3.7 +
5.5 +
class
21.6 =
23.2 +
+ 0.2
14.5 4:
w/o BN
92.6 0.9
90.
84.3 + 0.4
91.77 -
CBN (loffe & Szegedy.2011
20.1 00
20.0 0.0
+ 5.0
5,0 4:
RN (Nichol et al., 2018)
92.6
98.2 0.2
89.0 0.6
96.8
MetaBN (Bronskill al., 2020)
91.8 09
98.1 0.3
89.6 0.5
96.4 1
TaskNorm- (Bronskill al., 2020)
94.0 0.8
98.0 + 0.3
89.6
96.4 0.2
TaskNorm- (Bronskill et al., 2020)
94.4 0.8
98.6 0.2
90.0
96.3 0.2
MetaNorm
98.8
99.3 0.2
91.3 0.5
97.1 + 0.2
Results (except w/o BN and our MetaNorm) provided by (Bronskill et al., 2020).
Table 14: Sensitivity to Algorithm. Few-shot results on Omniglot using VERSA. Best performing methods and any other runs within the 95% confidence margin in bold. Transductive results indicated above dashed line.
Omniglot
5-way, I-shot
5-way, 5-shot
20-way, 1-shot
20-way, 5-shot
TBN
99.7
99.8 + 0.2
97.7 + 0.2
98.8 + 0.1
example
94.9
95.1 0.3
92.9 + 0.2
95.9 + 0.2
class
94.3 = 0.3
94.8 +
91.8 +
95.1 + 0.4
w/o BN
95.6 07
96.5 i.i
93.1
96.3 i 0.2
CBN (loffe & Szegedy, 2015)
94.3 0.3
95.7 0.0
92.7 0.2
95.2 1 0.3
MetaBN (Bronskill al., 2020)
96.7
1..3
95.8 1.2
97.1 1 0.2
TaskNorm- L (Bronskill al., 2020)
97.9
99.2
98.0 + 0.2
TaskNorm- (Bronskill ct al., 2020)
98.3 0.2
+9.5
96.7 -
98. + 0.1
MetaNorm
99.8 0.1
99.9 0.1
97.9 0.2
98.8 + 0.2
Results of TBN provided by (Gordon al., 2019). All other results based on our re-implementations.
Table 15; Sensitivity to Algorithm. Few-shot results Omniglot using ProtoNets. Best performing methods and any other runs within 95% confidence margin denoted in bold. Transductive results indicated above dashed line.
Omniglot
5-way, 1-shot
5-way, 5-shot
20-way, 1-shot
20-way, 5-shot
TBN
98.4 02
99.6 - 0.2
94.5 02
98.6 =
examplc
98. 4 + 02
99.5 0.2
94.3 0
98.5 0.1
class
98.4 02
99.3 022
94.2 =2
98.4
w/o BN
94.6 -
91.7
94.3
CBN (loffe& & Szegedy, 2015)
98.5 02
99.6
94.5 =
98.6 0.1
RN (Bronskill ct al., 2020)
98.0 02
99.6
94.1
98.6 0.1
MetaBN (Bronskill et al., 2020)
98.4 02
99.6
94.5
98.6
TaskNorm-L (Bronskill et al., 2020)
98.6 02
99.6
95.0
98.7 :0
TaskNNrm-1 (Bronskill et al., 2020)
98.4 + 0.2
99.6 -
93.4 02
98.6 :01
MetaNorm
98.9 : 0.2
99.7 0.2
95.8 0.2
98.9 102
Results (except w/o BN and our MetaNorm) provided by (Bronskill et al., 2020).
20

---

Published as a conference paper at ICLR 2021
MetaNorm
TaskNorm-
TaskNorm-L
MetaBN
TBN
Training Loss
10000
70000
80000
Iteration
Figure 4: Training loss. Results of using the ProtoNets algorithm minilmageNet with respect training loss versus iterations. Our MetaNorm achieves fastest training convergence.
H TRAINING SPEED
We plot the training loss versus training iterations by using the ProtoNets algorithm in Figure 4. MetaNorm achieves fastest training convergence. From Table and Figure 4, MetaNorm achieves best classification accuracy and training efficiency, which demonstrates the benefit of leveraging meta-lcarning by MetaNorm for batch normalization.
21

---

VDG Published as a conference paper at ICLR 2021
Table 16: Sensitivity to Dataset. Few-shot fassification results on Meta- Meta-Datase Datasel using ProtoNets. The + sign indicates the 95% confidence interval over tasks. Best performing methods and any other rums within the 95% confidence margin in bold. Results of other methods provided by Bronskill et al. (2020).
481
6.33
8.9
10.00
4.7
7.58
4.09
300
235
1.1
1.0
0.9
1.0
10
1.0
52.7.00
0.9
45.9 4:
34.0
47
470
2-
45
500
52.
1 0.8
03
- 0.8
0.8
08
0.8
0.7
414000
-
57.
601
$1.5
62.
630
614
674
67.
0.4
0.4
1 0.4
04
1 0.4
0.
1:
3
91.9
94
939
92.5
91.9
92
1.0
1.0
0.
1000000
0.
1.1
1.0
1.0
0.9
30.
38
36.
386
35.4
30.7
64.0 0.8
-2 0.8
- 0.7
26 0.8
- 08
- 0.8
0.7
0.6
--
65.7
69.1
06.0
632
6-
65.8
0.6
0.6
1 0.8
0.8
4- 066
4 0.7
0.7
0.7
04.5
76.1
06.
81
W6.
46.4 1.0
1.0
1.1
1.0
10
11
1.0
0.6
51
212
7
1-
51.1
48
31
46
43
46
72.5 1 08
07
08
07
06
74 4:
754
742
77
75.0
70.8
24
713
77.
0.7
0.7
0.7
0.7
02
07
08
0.7
06
2
61.2
57.7
580
5588
605
608
659
667
00.0 0.9
41 0.9
4 1.0
4 0.8
4 0.9
0.9
0.8
4:
3
090
541
68.6
68
64
71.
796
0.6
3- 0.5
0.7
0.6
- 06
0.6
0.6
X3.3
77.0
844
73.9
75.0
809
82.3
79.6
14. =
833
0.6
0.7
1: 0.6
0.3
4 06
0.7
0.6
05
4.64000
90.4
66
90.
90.6
903
80
1.0
430 3: 1.0
339 1000000 09
442 1 10
43. 1 1.0
1.0
125
45. 3:3:1:0
Teok Nom-No (Bronshi al 200)
Nonn-1, (Boonkil 2020)
Nome- (Bronkil 2000)
McbBNN Mc (Benokil al. 2020)
CBN (lotte Seggsly, 2015)
2005)
RN (Nichol et al, 2015)
LN (Bact al, 2016)
(Rum (umt, 20177
IN (Ulyanow
Meno Normn
BRN
22

---

Published as a conference paper at ICLR 2021
Table 17: Effect of number of units of hidden layers in MetaNorm for few-shot classification with MAML (Finn & Levine, 2018) on manelm The + sign indicates the 95% confidence interval tasks. We achieve best results with 128 units of hidden layers.
MAML
5-way, 1-shot
5-way, 5-shot
64
44.3 1.5
58.1 + 0.8
128
46.8 1.6
60.1 + 0.8
n 256
46.2 4
59.8 + 0.8
512
45.9 1.5
59.5 1 0.9
n 1024
44.9 155
58.7 0.8
23


Reviews:
[{"review_id": "oacCnNvvbE9", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The paper proposes an effective meta-learning normalization, named MetaNorm, to infer adaptive statistics for batch normalization by minimizing the KL divergence. The module is lightweight. The proposed method is evaluated on few-shot learning, domain generalization, and few-shot domain generalization. \n\nJustification of rating:  Overall, the proposed approach is logical and sound. The formulation and methodology seem to be correct. As I am not personally working on the specific topics, I might not be able to discover major issues in this work.\n\nStrengths:\n+ The proposed MetaNorm leverage meta-learning approach with KL divergence for learning to learn normalization statistics from data. \n+ The proposed approach is model agnostic and can be easily embedded into meta-learning approaches.\n+ Evaluation demonstrates it is able to learn with few examples, as well as handle variations in domain for domain generalization problem. The experimental results show consistent improvement over compared baselines (Table 2-5)\n+ The paper provide sufficient information for researcher to reproduce the results. Code will be released in the future.\n\nMinor comments:\n- The implementation of the hypernetworks is effectively MLP with one hidden layers of 128 units. Has the author conduct ablation on hypernetwork on various configuration?", "extra_scores": {}, "extra_text": {}}, {"review_id": "KqMgD1x2Y9", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to replace batch normalization statistics, which are typically computed as the batch moments during training or a fixed training average during testing, with the outputs of learned neural networks. These networks are trained to minimize the KL divergence between their output and the expected or desired batch statistics. In this way, the statistics computation is amortized and can hopefully generalize in the face of small batches and distribution shift.\n\nPros:\n\n+ The paper is generally well written and easy to follow.\n+ There are extensive experiments demonstrating the empirical effectiveness of the proposed approach.\n\nCons:\n\n- Perhaps missing some additional details that would help the reader better understand why and how the method works.\n\nCurrently, I recommend acceptance because I believe the pros rather handily outweigh the cons. I provide more details below.\n\n\nQuality\n---\n\nThe paper is well written and the experiments are thorough and well executed. Related to the con listed above, a few additional experiments could be useful for gaining a deeper understanding of the method.\n\nFirst, why do we expect an amortized procedure to produce reasonable normalization statistics when tested \"out of distribution\", e.g., on a new domain? It would be interesting to report, during testing, are the inferred statistics actually close to the ground truth statistics on the test domain? Or otherwise, are they interpretable in some other fashion? Any other insight that the authors could supply regarding this general question would also be appreciated.\n\nSecond, a similar question can be asked about the fully \"in distribution\" setting, i.e., where batch normalization was invented. If one were to apply the proposed method in a standard supervised learning setting (e.g., minimize KL between inferred statistics on each training point and the ground truth statistics computed on the training batch), would the inferred test time statistics come close to either the average statistics computed through training or perhaps even the ground truth test statistics? Is it any worse than standard batch norm? Could it perhaps even be better?\n\nClarity\n---\n\nMost of my concerns about clarity, which are small, relate to the experiments suggested above, i.e., better exposition of how and why the method works. Providing intuition throughout the paper regarding this point would strengthen the paper.\n\nThe few shot domain generalization setting is not too difficult to understand if the reader is familiar with both the few shot learning and domain generalization settings individually -- it is probably difficult to parse otherwise, but that is perhaps hard to rectify given the space constraints.\n\nThere also seem to be a few instances of \"meta\" vs \"non meta\" terms being misused, e.g., \"training\" vs \"meta-training\" when describing domain generalization.\n\nOriginality\n---\n\nAs far as I am aware, this work presents a novel method, though I am not an expert regarding the relevant prior work. The citations to batch normalization as used in domain adaptation seem appropriate. A few other papers in a similar vein perhaps should also be cited:\n\nhttps://arxiv.org/abs/1603.04779\nhttps://arxiv.org/abs/2002.04019\nhttps://arxiv.org/abs/2006.10963\nhttps://arxiv.org/abs/2006.16971\n\nSo although this general line of work seems to be attracting a decent amount of attention, this work is still novel in that it amortizes the inference process and presents extensive empirical results.\n\nSignificance\n---\n\nThis work seems significant to researchers interested in meta-learning, domain generalization, and problems involving distribution shift in general. The proposed method is also relatively simple, allowing for easier adoption and further testing.", "extra_scores": {}, "extra_text": {}}, {"review_id": "IVYcb3BAOWR", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a method for cross domain few-shot classification that learns to generate domain specific data statistics from very few training examples for domain independent batch normalisation.  \nThey propose to train small auxiliary networks that generate data statistics for normalisation. Networks are trained within a meta-learning framework using a KL divergence loss, which enforces estimated statistics on small support/training examples to match statistics from query sets where more data is available. \n\n\nSTRENGTHS\n\nThe paper is well written and motivated. The proposed approach is, for the most part, easy to follow and understand. The approach benefits from its simplicity and versatility (e.g. it is not tied to a specific FSL method) and promising performance is obtained.\n\nThe authors provide a very large set of experiments in multiple scenarios, and definitely demonstrates a strong effort in evaluating their approach. \n\n\nWEAKNESSES\n\nUnfortunately, despite the fact that a significant amount of time was dedicated to evaluate the method, the experimental section needs substantial modifications. This is mainly due to the presentation of the experiments, as well as some key missing comparisons.   \nRegarding presentation, too many implementation details and descriptions of the experiments are missing from the main paper (and in certain cases, missing altogether). Regarding datasets and implementation, it is ok to provide non essential details in the appendix (especially considering the large number of datasets considered). However, no information is provided at all in the main paper, which makes it very difficult to understand the setting of the experiments. For example, in Table 1 and Figure 2, it is not known on which datasets experiments are run, and it is never mentioned in the main text (for table 1) what FSL method is employed.   \nIn addition, Table 2 experiments comparing MetaNorm to different approaches sorely lacks description. The approach is compared to 9 different algorithms, none of which are given a description or reference to learn more about the method. It is therefore impossible, besides guessing, to know what the model is being compared to. This issue is also noticeable in Table 3-5, in particular with a baseline in Table 4 that is never described.   \nFinally, experiments in Figure 2 would be a lot more interesting if compared to standard methods. It would be interesting to see how the proposed strategy allows to be more sample efficient and reach stronger performance than transductive batch norm in situations where sample size is the smallest. As this is one of the cited main limitations of TBN, this experiment is highly important2- With regards to related work, I would suggest to move the section after the introduction, where it provides much better context to facilitate method comprehension, in particular regarding the description of the TBN strategy. \n\nAuthors should also comment on how their work, and in particular FSL domain generalisation setting, relates to cross domain few-shot learning works, \ne.g. Tseng et al ICLR 2020, Cross-Domain Few-Shot Classification via Learned Feature-Wise Transformation\nIt is currently presented as a completely new approach to FSL, and appears to ignore past cross domain FSL works. Please provide additional context regarding such works.\n\n\nRECOMMENDATION\n\nIn summary, the authors propose an interesting, simple strategy for more robust BN that can be of interest to the community. I would strongly recommend that the authors make the following modifications to strengthen their paper:\n\n1-\tReorganise and expand on the experimental evaluation to provide necessary details and make the main paper self-contained, even if it requires moving an experiment to the supplementary materials.  \n2-\tMove the related works section after the introduction, to provide additional context before delving into the method  \n3-\tRelate the proposed work to past work on cross-domain FSL and potentially tone down claims that few-shot cross domain learning is a completely new problem investigated here.  \n4-\tPlease provide a comparison to standard TBN in Figure 2\n\n\nAdditional suggestions\n\n5-\tIf possible, please provide an overview figure of the proposed method.  \n6-\tIn result tables, please sort methods according to their overall performance, and correct bolding in table 2 (protonet 5-way-5 shot MetaNorm is not best performing method) and highlight setting where methods have very similar performance (as in Table 14) for clarity.   \n7-\tProvide a clear list of contributions at the end of the introduction section  \n8-\tWhile KL divergence and hypernetworks are well known terms, it would make the paper more accessible to add a sentence (and equation in the case of the KL divergence) or two describing the terms. In particular, hypernetworks are generally used to characterise weight generators for entire architectures and might lead to confusion.  \n9-\tIt could be nice to provide more attention to how this work relates to conditional batch norm works, and whether they can be complementary.  \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "0HKa-K3gSo9", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes a new method for normalizing few-shot learning episodes. The authors point out that the statistics of an episode are unreliable when the size of the episode is small or when the data distribution changes from episode to episode. To remedy this, the authors propose a method called ‘MetaNorm’ which uses a meta-learning approach to infer the means and variances to be used in the batch normalization layers that are employed in the feature extractor component. In particular, they meta-learn the parameters for a set of hypernetworks in an amortized fashion that learn to generate the means and variances of the batch normalization layers conditioned on the contents of the episode. The paper focuses entirely on the few-shot image classification scenario where MetaNorm is evaluated in various settings including standard few-shot classification and domain generalization (including a novel few-shot domain generalization setting).\n\n**Pros:**\nFundamentally, the concept behind MetaNorm is innovative and promising. The idea of using meta-learned hypernetworks to generate the means and variances of the normalization layers is good. The paper also makes a good case for the effectiveness of a KL term that is added to the classifer loss function that helps the hypernetworks learn a good set of parameters. The range of experiments is sufficient.\n\n**Cons:**\nThe paper is poorly executed and missing too much information to ascertain that the method outperforms competitive approaches.\n\nSpecific concerns:  \n(1) There is no mention of the MetaNorm training efficiency (which is generally understood to be the primary goal of batch normalization, see [1,2]). How does training efficiency of MetaNorm compare to other methods? And if training efficiency is not a goal, MetaNorm should be compared to methods whose goal is to adapt a classification system to a variety of data distributions (e.g. [3,4,5,6, etc.]).\n\n(2) Reproducibility: The paper is currently missing details that are required to reproduce the results and ensure fair comparison with other methods including:\n- Which existing codebases were used (if any)?\n- What hyper-parameters and training settings are used? (e.g. learning rates, number of training iterations, the use of early stopping, the number of test episodes, weight decay, etc.)\n- In Section 4, “Impact of Target Set Size” indicates that the number of samples in the query set is a key parameter for MetaNorm and that 125 for the few-shot scenario and 128 for the domain generalization scenario are the best values to use. Are these the values used in the experiments? If not, what values are used?\n\n(3) There are many omissions in the paper:\n- The variable ‘m’ as used in equations 3, 4, 8, 9, 10 is not defined.\n- The KL expressions in equations 4, 8, and 10 should be written as a proper minimization expression where the parameters that are minimized are explicitly stated.\n- The total loss function for MetaNorm is never stated. One could assume that it is a sum between the regular loss function for the particular few shot learning problem and the KL term, but it should be explicitly stated. Also, is there a weight on the KL term in the loss function? If so, how is the optimal weight determined?\n- Various methods in the tables are not defined or referenced including RN, TaskNorm-I, TaskNorm-L, ‘class’, and ‘example’.\n- How are the confidence intervals on the results in the table calculated? Are they 95% confidence intervals, or something else?\n- In the tables, how do you decide which entry to ‘bold’ given the confidence intervals? In table 2, TBN (and sometimes TaskNorm) are often within error bars of MetaNorm but are not bolded. Same for tables 11 and 13, 14.\n- It would be good to have more information on the hypernetworks that generate the normalization parameters. In Section 2, it says: “They are three layer networks with one hidden layer of 128 units and the input size depends on the feature dimensions of the convolutional layers.” How big is the other hidden layer and output size of the network? What sort of non-linearities are used, etc.?\n- In section 2 it says: “Note that the inference functions $f_\\mu^l()$ and $f_\\sigma^l()$ are shared by different channels in the same layer and we will learn L pairs of those functions if we have L convolutional layers in the meta-learning model”. What is the justification for this parameterization (say versus having unique functions per channel)? Was an ablation study that compares various configurations done?\n\n(4) There are factual errors in the paper:\n- In Section 1 and Section 3, the authors state that Prototypical networks uses transductive batch normalization. This is not the case (refer to https://github.com/jakesnell/prototypical-networks for the version of Prototypical networks authored by Jake Snell). Please correct.\n- In section 1 it states that TaskNorm “remains under-performing compared to transductive batch normalization”. While the results listed in Table 2 support this statement, the results in Table 3 refute this statement. Please clarify.\n\n(5) There are several missing attributions and some exact text taken from other papers that is not quoted or cited: \n- In Table 2, the competitive results for MAML and ProtoNets seem to be extracted from [2], yet these are not attributed. Did you reproduce them? Were the MetaNorm numbers generated with a similar code base and hyper-parameters?\n- In Tables 11, 13, and 14, the competitive results also seem to be extracted from [2] and were not attributed. Were they reproduced, or just stated? Were the MetaNorm numbers generated with a similar code base and hyper-parameters?\n- The following sentences seem to be directly copied from [2]. If so, there should be quotation marks and attribution:\n“Batch normalization relies on the implicit assumption that the dataset comprises i.i.d. samples from some underlying distribution.”\n“The challenge constructs few-shot learning tasks by drawing from the following distribution. First, one of the datasets is sampled uniformly; second, the “way” and “shot” are sampled randomly according to a fixed procedure; third, the classes and context / target instances are sampled.”\n“In the meta-test phase, the identity of the original dataset is not revealed and the tasks must be treated independently (i.e. no information can be transferred between them). The meta-training set comprises a disjoint and dissimilar set of classes from those used for meta-test.”\n\n(6) In Figure 2, the title says “Impact of target set size.”. However, for the left and middle plot, the horizontal axis is labeled as |S|, which is the support set size. Please resolve the ambiguity.\nMinor Comments:\n- In section 2, MetaNorm for Few-Shot Classification: The sentence “The $p(m|Q)$ can be estimated by directly calculating statistics using the query set, which however performs inferior to inference by optimization.” is grammatically incorrect.\n- In section 2, MetaNorm for Few-Shot Classification: should “multiple layer perception networks” be “multi-layer perceptron networks”? Similarly, in section 2 it says: “…which are realized as multi-layer perceptions and we call hypernetworks.” ‘perceptions’ should be ‘perceptrons’.\n- Sometimes the nomenclature support set / query set is used, and other places it refers to the same as context set / target set. Please use consistent nomenclature.\n- In section 2, it says “…which are realized as multi-layer perceptions and we call hypernetworks”. And later in section 2, it says: “They are parameterized by feed-forward multiple layer perception networks, which we call hypernetworks”.  Note that hypernetworks are an established concept, you should cite [7].\n- Section 1 should clarify in the batch normalization exposition that the methods described in the paper apply only to normalizing 2D convolutional layers (i.e. does not apply to fully connected layers as they do not have feature maps).\n\n**References:**  \n[1] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 448–456, 2015.  \n[2] John Bronskill, Jonathan Gordon, James Requeima, Sebastian Nowozin, and Richard Turner. Tasknorm: Rethinking batch normalization for meta-learning. In International Conference on Machine Learning. 2020.  \n[3] Rebuffi, Sylvestre-Alvise, Hakan Bilen, and Andrea Vedaldi. \"Learning multiple visual domains with residual adapters.\" Advances in Neural Information Processing Systems. 2017.  \n[4] Rebuffi, Sylvestre-Alvise, Hakan Bilen, and Andrea Vedaldi. \"Efficient parametrization of multi-domain deep neural networks.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.  \n[5] Requeima, James, et al. \"Fast and flexible multi-task classification using conditional neural adaptive processes.\" Advances in Neural Information Processing Systems. 2019.  \n[6] Tseng, Hung-Yu, et al. \"Cross-domain few-shot classification via learned feature-wise transformation.\" arXiv preprint arXiv:2001.08735 (2020).  \n[7] Ha, David, Andrew Dai, and Quoc V. Le. \"Hypernetworks.\" arXiv preprint arXiv:1609.09106 (2016).\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an lightweight method for cross-domain few-shot learning, using a meta-learning approach to predict batch normalization statistics.
After the extensive paper revisions and discussion, the reviewers all agreed that this paper is above the bar for acceptance, assuming that the authors will include results for both the standard and expanded target set size in the final version of the paper. The authors are strongly encouraged to include these results in the camera-ready version of the paper.
Decision: Accept (Poster)