METANORM: LEARNING TO NORMALIZE FEW-SHOT BATCHES ACROSS DOMAINS
Year: 2021

Abstract:
Batch normalization plays a crucial role when training deep neural networks. However, batch statistics become unstable with small batch sizes and are unreliable in the presence of distribution shifts. We propose MetaNorm, a simple yet effective meta-learning normalization. It tackles the aforementioned issues in a unified way by leveraging the meta-learning setting and learns to infer adaptive statistics for batch normalization. MetaNorm is generic, flexible and model-agnostic, making it a simple plug-and-play module that is seamlessly embedded into existing meta-learning approaches. It can be efficiently implemented by lightweight hypernetworks with low computational cost. We verify its effectiveness by extensive evaluation on representative tasks suffering from the small batch and domain shift problems: few-shot learning and domain generalization. We further introduce an even more challenging setting: few-shot domain generalization. Results demonstrate that MetaNorm consistently achieves better, or at least competitive, accuracy compared to existing batch normalization methods.

heading: 1 INTRODUCTION
text: Batch normalization (Ioffe & Szegedy, 2015) is crucial for training neural networks, and with its variants, e.g., layer normalization (Ba et al., 2016), group normalization (Wu & He, 2018) and instance normalization (Ulyanov et al., 2016), has thus become an essential part of the deep learning toolkit (Bjorck et al., 2018; Luo et al., 2018a; Yang et al., 2019; Jia et al., 2019; Luo et al., 2018b; Summers & Dinneen, 2020). Batch normalization helps stabilize the distribution of internal activations when a model is being trained. Given a mini-batch B, the normalization is conducted along each individual feature channel for 2D convolutional neural networks. During training, the batch normalization moments are calculated as follows:
µB = 1
M M∑ i=1 ai, σ 2 B = 1 M M∑ i=1 (ai − µB)2, (1)
where ai indicates the i-th element of the M activations in the batch, M = |B| ×H ×W , in which H and W are the height and width of the feature map in each channel. We can now apply the normalization statistics to each activation:
a′i ← BN(ai) ≡ γâi + β, where, âi = ai − µB√ σ2B + , (2)
where γ and β are parameters learned during training, is a small scalar to prevent division by 0, and operations between vectors are element-wise. At test time, the standard practice is to normalize activations using the moving average over mini-batch means µB and variance σ2B. Batch normalization is based on an implicit assumption that the samples in the dataset are independent and identically distributed. However, this assumption does not hold in challenging settings like few-shot learning and domain generalization. In this paper, we strive for batch normalization when batches are of small size and suffer from distributions shifts between source and target domains.
Batch normalization for few-shot learning and domain generalization problems have so far been considered separately, predominantly in a meta-learning setting. For few-shot meta-learning (Finn
et al., 2017; Gordon et al., 2019), most existing methods rely critically on transductive batch normalization, except those based on prototypes (Snell et al., 2017; Allen et al., 2019; Zhen et al., 2020a). However, the nature of transductive learning restricts its application due to the requirement to sample from the test set. To address this issue, Bronskill et al. (2020) proposes TaskNorm, which leverages other statistics from both layer and instance normalization. As a non-transductive normalization approach, it achieves impressive performance and outperforms conventional batch normalization (Ioffe & Szegedy, 2015). However, its performance is not always performing better than transductive batch normalization. Meanwhile, domain generalization (Muandet et al., 2013; Balaji et al., 2018; Li et al., 2017a;b) suffers from distribution shifts from training to test, which makes it problematic to directly apply statistics calculated from a seen domain to test data from unseen domains (Wang et al., 2019; Seo et al., 2019). Recent works deal with this problem by learning a domain specific normalization (Chang et al., 2019; Seo et al., 2019) or a transferable normalization in place of existing normalization techniques (Wang et al., 2019). We address the batch normalization challenges for few-shot classification and domain generalization in a unified way by learning a new batch normalization under the meta-learning setting.
We propose MetaNorm, a simple but effective meta-learning normalization. We leverage the metalearning setting and learn to infer normalization statistics from data, instead of applying direct calculations or blending various normalization statistics. MetaNorm is a general batch normalization approach, which is model-agnostic and serves as a plug-and-play module that can be seamlessly embedded into existing meta-learning approaches. We demonstrate its effectiveness for few-shot classification and domain generalization, where it learns task-specific statistics from limited data samples in the support set for each few-shot task; and it can also learn to generate domain-specific statistics from the seen source domains for unseen target domains. We verify the effectiveness of MetaNorm by extensive evaluation on few-shot classification and domain generalization tasks. For few-shot classification, we experiment with representative gradient, metric and model-based meta-learning approaches on fourteen benchmark datasets. For domain generalization, we evaluate the model on three widely-used benchmarks for cross-domain visual object classification. Last but not least, we introduce the challenging new task of few-shot domain generalization, which combines the challenges of both few-shot learning and domain generalization. The experimental results demonstrate the benefit of MetaNorm compared to existing batch normalizations.

heading: 2 RELATED WORKS
text: Transductive Batch Normalization For conventional batch normalization under supervised settings, i.i.d. assumptions about the data distribution imply that estimating moments from the training set will provide appropriate normalization statistics for test data. However, in the meta-learning scenario data points are only assumed to be i.i.d. within a specific task. Therefore, it is critical to select the moments when batch normalization is applied to support and query set data points during meta training and meta testing. Hence, in the recent meta-learning literature the running moments are no longer used for normalization at meta-test time, but instead replaced with support/query set statistics. These statistics are used for normalization, both at meta-train and meta-test time. This approach is referred to as transductive batch normalization (TBN) (Bronskill et al., 2020). Competitive meta-learning methods (e.g., Gordon et al., 2019; Finn et al., 2017; Zhen et al., 2020b) rely on TBN to achieve state-of-the-art performance. However, there are two critical problems with TBN. First, TBN is sensitive to the distribution over the query set used during meta-training, and as such is less generally applicable than non-transductive learning. Second, TBN uses extra information for multiple test samples, compared to non-transductive batch normalization at prediction time, which could be problematic as we are not guaranteed to have a set of test samples available during training in practical applications. In contrast, MetaNorm is a non-transductive normalization. It generates statistics from the support set only, without relying on query samples, making it more practical.
Meta Batch Normalization To address the problem of transductive batch normalization and improve conventional batch normalization, meta-batch normalization (MetaBN) was introduced (Triantafillou et al., 2020; Bronskill et al., 2020). In MetaBN, the support set alone is used to compute the normalization statistics for both the support and query sets at both meta-training and meta-test time. MetaBN is non-transductive since the normalization of a test input does not depend on other test inputs in the query set. However, Bronskill et al. (2020) observe that MetaBN performs less well for small-sized support sets. This leads to high variance in moment estimates, which is similar to the
difficulty of using batch normalization with small-batch training (Wu & He, 2018). To address this issue, Bronskill et al. (2020) proposed TaskNorm, which learns to combine statistics from both layer normalization and instance normalization, with a lending parameter to be learned at meta-train time. As a non-transductive normalization, TaskNorm achieves impressive performance, outperforming conventional batch normalization. However, it can not always perform better than transductive batch normalization. TaskNorm indicates non-transductive batch normalization estimates proper normalization statistics by involving learning in the normalization process. We also propose to learn batch normalization within the meta-learning framework, but instead of employing a learnable combination of existing normalization statistics, we directly learn to infer statistics from data. At meta-train time, the model learns to acquire the ability to generate statistics only from the support set and at meta-test time we directly apply the model to infer statistics for new tasks.
Batch Normalization for Domain Adaptation and Domain Generalization Domain adaption suffers from a distribution shift between source and target domains, which makes it sub-optimal to directly apply batch normalization (Bilen & Vedaldi, 2017). Li et al. (2016) proposed adaptive batch normalization to increase the generalization ability of a deep neural network. By modulating the statistical information of all batch normalization layers in the neural network, it achieves deep adaptation effects for domain-adaptive tasks. Nado et al. (2020) noted the possibility of accessing small unlabeled batches of the shifted data just before prediction time. To improve model accuracy and calibration under covariate shift, they proposed prediction-time batch normalization. Since the activation statistics obtained during training do not reflect statistics of the test distribution, when testing in an out-of-distribution environment, Schneider et al. (2020) proposed estimating the batch statistics on the corrupted images. Kaku et al. (2020) demonstrated that standard non-adaptive feature normalization fails to correctly normalize the features of convolutional neural networks on held-out data where extraneous variables take values not seen during training. Learning domain-specific batch normalization has been explored (Chang et al., 2019; Wang et al., 2019). Wang et al. (2019) introduced transferable normalization, TransNorm, which normalizes the feature representations from source and target domain separately using domain-specific statistics. Along a similar vein, Chang et al. (2019) proposed a domain-specific batch normalization layer, which consists of two branches, each in charge of a single domain exclusively. The hope is that, through the normalization, the feature representation will become domain invariant. Nevertheless, these normalization methods are specifically designed for domain adaptation tasks, where data from target domains are available, though often unlabelled. This makes them inapplicable to domain generalization tasks where data from target domains are inaccessible at training time. Seo et al. (2019) proposed learning to optimize domain specific normalization for domain generalization tasks. Under the meta-learning settings, a mixture of different normalization techniques is optimized for each domain, where the mixture weights are learned specifically for different domains. Instead of combining different normalization statistics, MetaNorm learns from data to generate adaptive statistics specific to each domain. Moreover, we introduce an even more challenging setting, i.e., few-shot domain generalization, which combines the challenges of few-shot classification and domain generalization.
Conditional Batch Normalization de Vries et al. (2017) proposed conditional batch normalization to modulate visual processing by predicting the scalars γ and β of the batch normalization conditioned on the language from an early processing stage. Conditional batch normalization has also been applied to align different data distributions for domain adaptation (Li et al., 2016). Oreshkin et al. (2018) applies conditional batch normalization to metric-based models for the few-shot classification task. Tseng et al. (2020) proposed a learning-to-learn method to optimize the hyper-parameters of the feature-wise transformation layers by conditional batch normalization for cross-domain classification. Unlike conditional batch normalization, we use extra data (the query set) to generate normalization statistics under the meta-learning setting, rather than the scalars.

heading: 3 METHODOLOGY
text: We view finding appropriate statistics for batch normalization as a density estimation problem. We need to infer the distribution parameters, such as, µ and σ when a Gaussian distribution is presumed, as in existing batch normalization approaches. The motivation behind MetaNorm is to leverage the meta-learning setting and learn from data to generate adaptive normalization statistics. MetaNorm is generic and model-agnostic, addressing batch normalization in a unified way for different settings by minimizing the KL divergence, which is a common metric to measure the difference between two
probability distributions: DKL [ qφ(m)|pθ(m) ] , (3)
where m is a random variable that represents the distribution of activations, pθ(m) and qφ(m) are defined as Gaussian distributions with different implementations depending on the task of interest, e.g., few-shot classification or domain generalization. We leverage the amortized inference technique (Kingma & Welling, 2013) and implement this by inference networks. To be more specific, for each individual channel in each ` convolutional layer, we infer the moments µ and σ by f `µ(·) and f `σ(·), respectively, which are realized as multi-layer perceptrons and we call hypernetworks (Ha et al., 2016). Hypernetworks use one network to generate the weights for another network. Our hypernetworks generate the statistics from data by using amortization techniques.
We simply incorporate theDKL term into the optimization of the existing model with the cross-entropy loss LCE, resulting in a general loss function as follows:
L = LCE − λDKL [ qφ(m)|pθ(m) ] (4)
where λ > 0 is a regularization hyper-parameter.
MetaNorm for Few-Shot Classification In the few-shot classification scenario, we define the C-way K-shot problem using the episodic formulation from (Vinyals et al., 2016). Each task Ti is a classification problem sampled from a task distribution p(T ). The tasks are divided into a training meta-set T tr, validation meta-set T val, and test meta-set T test, each with a disjoint set of target classes (i.e., a class seen during testing is not seen during training). The validation meta-set is used for model selection, and the testing meta-set is used only for final evaluation. Each task instance Ti ∼ p (T ) is composed of a support set S and a query set Q, and only contains N classes randomly selected from the appropriate meta-set.
We aim to infer statistics from the support set that better match the query set. Therefore, we adopt a straightforward criterion for the inference:
DKL [ qφ(m|S)||pθ(m|Q) ] , (5)
where we define q(m|S)=N (µS , σS) and p(m|Q)=N (µQ, σQ), which are the distributions inferred from the support and query sets in a few-shot learning task. By minimizing the KL term in conjunction with the prime objective of a meta-learning algorithm, we are able to find the appropriate statistics from limited data samples for batch normalization. The KL term adheres to a closed form, which makes it easy to implement and computationally efficient. The p(m|Q) can be estimated by directly calculating statistics using the query set, which however performs inferior to inference by optimization. We note the inference from the query set only happens during meta-training time and we use the learned inference network to generate normalization statistics at meta-test time for a test task using its support set.
To infer µS , we deploy an inference function f `µ(·) that takes activations of a sample as input, and the outputs from all samples are then averaged as the final µS :
µS = 1
|S| |S|∑ i=1 f `µ(ai), (6)
where ai ∈ Rw×h is the flattened vector of the activation map of the i-th sample in the support set, w is the width of activations, and h is the height of the activation map. To infer σS , we use the obtained µS and deploy a separate inference function f `σ(·):
σS = 1
|S| |S|∑ i=1 f `σ ( (ai − µS)2 ) . (7)
It is worth mentioning that we actually use each sample to infer the statistics and take the average of all inferred statistics as the final normalization statistics. This enables us to fully exploit the samples to generate more accurate statistics.
Note that the inference functions f `µ(·) and f `σ(·) are shared by different channels in the same layer and we will learn L pairs of those functions if we have L convolutional layers in the meta-learning
model. They are parameterized by feed-forward multiple layer perception networks, which we call hypernetworks. Using these hypernetworks, we generate support moments (µS , σS) and query moments (µQ, σQ) from the support and query sets, which are used for calculating the KL term in Eq. (5) for optimization during meta-training time. At meta-training time, we apply the statistics inferred from the support set for normalization of both support and query samples:
a′ = γ ( a− µS√ σ2S + ) + β, (8)
where γ and β are jointly learned with parameters of the hypernetworks at meta-training time and directly applied at meta-test time, as in conventional batch normalization. At meta-test time, given a test task, we use hypernetworks that take the support set as input to generate normalization statistics directly used for the query set.
MetaNorm for Domain Generalization In the domain generalization scenario, we adopt the metalearning setting from (Li et al., 2018a; Balaji et al., 2018; Du et al., 2020), and divide a dataset into the source domains used for training and the target domains held out for testing. At meta-training time, data in the source domains is episodically divided into sets of meta-source Ds and meta-target Dt domains. In a similar vein to few-shot classification, we would like to learn to acquire the ability to generate domain-specific statistics from a single example, which can then be applied to unseen domains. We assume we can generate reasonable normalization statistics by using only one sample from the new domain, because, intuitively, a single sample already carries sufficient domain information. We use a single example and all the examples in the same domain to infer the domain-specific statistics and minimize the KL term:
DKL [ qφ(m|ai)||pθ(m|Ds\ai) ] , (9)
where we define q(m|ai)=N (µa, σa), and p(m|Ds\ai)=N (µD, σD), which are implemented in a similar way as Eq. (6) and Eq. (7), and ai is an example from its own domain Ds. In both the meta-source and meta-target domains, each example is normalized using the statistics generated by itself, like in Eq. (8), in which we make γ and β shared across all domains. The minimization of the KL term in Eq. (9) is to encourage the model to generate domain-specific statistics for normalization from only a single example. This enables us to generate domain-specific statistics on target domains that are never seen at meta-training time.
In practice, we take the sum of all samples in all source domains as follows:
|Ds|∑ i J∑ j DKL [ qφ(m|ai)||pθ(m|Dsj\ai) ] , (10)
where Dsj denotes the j-th of J meta-source domains. The inference networks are first at metatraining time learned and then directly used as examples from the target domain at meta-test time. Note that on the meta-target domain we do not apply the KL term; instead, we simply rely on each example to generate its statistics for normalization.
MetaNorm for Few-Shot Domain Generalization We introduce an even more challenging setting, i.e., few-shot domain generalization, that combines the challenges of both few-shot classification and domain generalization. Specifically, we aim to learn a model from a set of classification tasks, each of which has only a few samples in a support set for training and test the model on tasks in a query set, which are in a different domain from the support set. Like few-shot classification, the label space is not shared between training and testing. Cross-domain few-shot learning has been explored recently by Tseng et al. (2020) and Guo et al. (2020). However, the setting of our few-shot domain generalization is different and considered to be more challenging, as the support and query set are from different domains in the meta-test stage and the target domain is also unseen throughout the training stage. An example for the few-shot domain generalization setting is provided in Figure 1.
We divide a dataset into the source domains S used for training and the target domains T held out for testing. During training time, data in the source domains S is episodically divided into sets of meta-train Ds and meta-test Dt domains. We sample C-way k-shot data as the support set from each meta-source domain Ds, where k is the number of labelled examples for each of the C classes. We
sample C classes from the meta-test Dt domain as the query set. At test time, we sample C-way k-shot data as the support set from each of the source domains S . The model learned at meta-training time is then fine-tuned on few-shot tasks samples from the source domains and tested on the target domain T . To learn the normalization statistics, we minimize the following KL term:
|Ds|∑ i DKL[qφ(m|ai)||pθ(m|Ds)], (11)
where ai is the activation associated with each sample from the meta-source domain Ds. Likewise, q(m|ai) and p(m|Ds) are also defined as factorized Gaussian distributions. We also adopt γ and β, which are shared across tasks and jointly learned. MetaNorm learns to acquire the ability to generate proper statistics for itself, and applies it to the samples in the meta-target domain.

heading: 4 EXPERIMENTAL RESULTS
text: We conduct an extensive set of experiments on a total of 17 datasets containing more than 15 million images. We use three representative approaches to meta-learning as our base models, i.e., MAML (Finn et al., 2017), ProtoNets (Snell et al., 2017), and VERSA (Gordon et al., 2019), which can verify our MetaNorm is generic, flexible and model-agnostic, making it a simple plug-and-play module that is seamlessly embedded into existing meta-learning approaches. We further compare different normalization methods: transductive batch normalization (TBN), “example” that denotes testing with one example at a time by using TBN, “class” that denotes testing with one class at a time by using TBN, w/o BN which is not using batch normalization, CBN which is using conventional batch normalization, RN (Nichol et al., 2018), MetaBN (Bronskill et al., 2020), TaskNorm-L (Bronskill et al., 2020), and TaskNorm-I (Bronskill et al., 2020). All details about datasets and implementation settings are provided in the appendix. More experimental results, including convergence analysis, are also provided in the appendix. Our code will be publicly released. 1
Effect of KL Term We first conduct ablation studies that measure the effectiveness of MetaNorm. The key of MetaNorm is the introduced KL term for learning to learn statistics. We test the performance of MetaNorm without the KL term by directly using the statistics generated from data.
1https://github.com/YDU-AI/MetaNorm.
In this case, we also use the hypernetworks to generate the moments, µ and σ by simply removing the KL term in the objective function. In Table 1 we present results for few-shot classification on miniImageNet (Vinyals et al., 2016) and for domain generalization on PACS (Li et al., 2017a). The performance of MetaNorm without KL degrades significantly. This is expected, as without the KL term the generation process of normalization statistics lacks direct supervision from the target distribution, resulting in improper statistics.
Impact of Target Set Size The other key parameter in MetaNorm is the size of the target set; that is, the number |Q| of samples in the query set (in few-shot classification) and the number |Ds| of samples in each domain (in domain generalization). This parameter is important when learning normalization statistics because we use the statistics generated by the target set as the ‘ground truth’. We evaluate its impact on the performance of MetaNorm in Figure 2. The experimental results show that TBN is not affected by the target size, both in the 5-way, 1-shot and 5 way, 5-shot tasks. MetaNorm performance rises as the size of the target set increases and plateaus at a reasonable size. In the few-shot setting, the performance reaches its peak at a size of about 125, which is slightly larger than the standard size of 75, while in the domain generalization setting, the performance plateaus at a size of about 128. This demonstrates that we are able to generate proper statistics with the mini-batch gradient descent optimization. In scenarios demanding a very small target set size, we could leverage image synthesis techniques to generate more samples for the targets sets.
Sensitivity to Algorithm We evaluate MetaNorm using the MAML (Finn et al., 2017), ProtoNets (Snell et al., 2017) and VERSA (Gordon et al., 2019) algorithms, which are representative gradient, metric and model based meta-learning approaches for few-shot classification. These experiments are conducted on the Omniglot and miniImageNet datasets under different settings. The comparison results on miniImageNet are summarized in Table 2 and the results on Omniglot are provided in the appendix. For all three meta-learning approaches under all settings, MetaNorm consistently achieves comparable performance both to the non-transductive and transductive normalization methods. Being non-transductive, TaskNorm can achieve impressive performance on all the tasks, but its performance is not always better than transductive batch normalization. MetaNorm achieves comparable
TBN 45.9 ± 0.6 65.5 ± 0.9 45.5 ±1.8 59.7 ± 0.9 53.4 ±1.8 67.3 ±0.9 example 43.9 ± 1.9 60.1 ± 0.8 26.9 ± 1.5 30.3 ± 0.7 44.1 ± 1.7 60.3 ± 0.7 class 43.1 ± 1.8 59.8 ± 0.8 26.9 ± 1.5 27.2 ± 0.6 43.8 ± 1.8 59.7 ± 0.6 w/o BN 44.1 ± 0.5 60.1 ± 0.6 34.7 ± 1.5 51.3 ± 0.8 48.1 ± 1.5 63.8 ± 0.6 CBN (Ioffe & Szegedy, 2015) 47.8 ± 0.6 66.7 ± 0.5 20.1 ± 0.0 20.2 ± 0.2 45.7 ± 1.4 60.7 ± 0.8 RN (Nichol et al., 2018) 39.7 ± 0.5 63.1 ± 0.5 40.7 ± 1.7 57.6 ± 0.9 - - MetaBN (Bronskill et al., 2020) 42.6 ± 0.6 64.6 ± 0.5 41.6 ± 1.6 58.6 ± 0.9 50.1 ± 1.7 65.8 ± 0.9 TaskNorm-L (Bronskill et al., 2020) 47.5 ± 0.6 65.3 ± 0.5 42.0 ± 1.7 58.1 ± 0.9 52.1 ± 1.6 66.1 ± 0.7 TaskNorm-I (Bronskill et al., 2020) 43.2 ± 0.6 63.9 ± 0.5 42.4 ± 1.7 58.7 ± 0.9 52.9 ± 1.7 66.5 ±0.8 MetaNorm (|Q| = 75) 47.3 ± 0.6 65.4 ± 0.5 44.7 ± 1.5 59.6 ± 0.8 52.7 ± 1.6 67.5 ± 0.8 MetaNorm (|Q| = 125) 48.1 ± 0.6 65.9 ± 0.9 46.8 ± 1.6 60.1 ± 0.8 53.7 ± 1.6 68.1 ± 0.8 † Results for MAML and ProtoNets (except w/o BN) provided by (Bronskill et al., 2020), and VERSA with TBN provided by (Gordon et al., 2019). All other results based on our re-implementations.
performance to transductive batch normalization, especially under the 5-way-1-shot setting, which is challenging since only a few examples are available to generate statistics. Notice that, MetaNorm performs well with the standard query set size |Q| of 75 (15 per category). It is slightly better than non-transductive TaskNorm and comparable with TBN. MetaNorm achieves its best performance with a query size |Q| of 125 (25 per category), only slightly larger than the standard size of 75. This demonstrates the benefit of leveraging meta-learning by MetaNorm for batch normalization. We conclude that MetaNorm is general and serves as a plug-and-play module for existing meta-learning models to improve their performance.
Sensitivity to Dataset We evaluate MetaNorm on a demanding few-shot classification challenge called Meta-Dataset (Triantafillou et al., 2020), which is composed of thirteen image classification datasets (eight for training, five testing). To compare with previous work, we perform experiments with ProtoNets and report the results in Table 3. All thirteen per-dataset results can be found in the appendix. MetaNorm achieves high performance in terms of average rank, with highest accuracy on eight of the thirteen datasets. MetaNorm outperforms transductive batch normalization on eleven datasets. It achieves comparable performance with transductive batch normalization on Omniglot and MNIST, which are relatively less challenging. Moreover, MetaNorm performs better than TaskNorm on seven of the thirteen datasets. We conclude that MetaNorm is effective, outperforming alternative normalizations for most datasets.
Sensitivity to Domains For this experiment we adopt two widely-used benchmarks for domain generalization of visual object recognition, i.e., PACS (Li et al., 2017a) and Office-Home (Venkateswara et al., 2017). Detailed descriptions on the experimental settings and implementations are provided in the appendix. For fair comparison with prior methods (Balaji et al., 2018; Li et al., 2018b; Seo et al., 2019), we employ ResNet-18 as the backbone network in all experiments. As shown in Table 4, MetaNorm achieves the best performance on PACS and Office-Home in terms of average accuracy. On PACS, MetaNorm consistently outperforms other normalization approaches including domain-specific normalization (Seo et al., 2019), on all four domains. It is worth mentioning that the baseline normalization uses the statistics from the source domains for the batch normalization of the target domain. As expected, the baseline method produces relatively poor performance on most domains, since the source domains cannot provide proper statistics for target domains due to the distribution shift. We have also done an experiment using standard batch normalization. In the training stage, we compute the ground truth statistics using all the test data on the meta-target domain Dt instead of using inferred statistics p(m|Ds\ai). MetaNorm is still better on most domains and on
average. This is reasonable because ground truth statistics from the test data do not necessarily reflect the true data distribution. The experimental results demonstrate MetaNorm can generate reasonable normalization statistics from only one sample in its domain. We conclude that MetaNorm is effective for domain generalization.
Few-Shot Domain Generalization In our final experiment, we adopt the DomainNet dataset (Peng et al., 2019) and introduce a new, more challenging setting to evaluate the performance for few-shot domain generalization. Detailed descriptions on the dataset and experimental settings are provided in the appendix. We conduct the experiments with the MAML and ProtoNets algorithms under both 5-way 1-shot and 5-way 5-shot settings, and the results are reported in Table 5. We implement transductive batch normalization, MetaBN and the variants of TaskNorm for direct comparison. Under both settings, our MetaNorm produces the best performance and surpasses the transductive batch normalization by large margins of up to 4.0% on the challenging 5-way 1-shot setting with MAML. MetaNorm also achieves better results than the non-transductive TaskNorm approaches. At the same time, with ProtoNet our MetaNorm again consistently delivers the best performance and surpasses both transductive and non-transductive normalizations, The performance on the challenging few-shot domain generalization scenario with different meta-learning algorithms again demonstrates the effectiveness of MetaNorm in handling the challenges of batch normalization for small batches and across domains.

heading: 5 CONCLUSION
text: In this paper we present MetaNorm, a meta-learning based batch normalization. MetaNorm tackles the challenging scenarios where the batch size is too small to produce sufficient statistics or when training statistics are not directly applicable to test data due to a domain shift. MetaNorm learns to learn adaptive statistics that are specific to tasks or domains. It is generic and model-agnostic, which enables it to be used with various meta-learning algorithms for different applications. We evaluate MetaNorm on two well-known existing tasks, i.e., few-shot classification and domain generalization, and we also introduce the challenging evaluation scenario of few-shot domain generalization that addresses the small batch and distribution shift problems simultaneously. An extensive evaluation on 17 datasets reveals that MetaNorm consistently achieves results that are better, or at least competitive, compared to other normalization approaches, verifying its effectiveness as a new meta-learning based batch normalization approach.

heading: A ALGORITHMS DESCRIPTIONS
text: In this Appendix we provide the detailed MetaNorm algorithm descriptions to conduct batch normalization for few-shot classification (Algorithm 1), domain generalization (Algorithm 2) and few-shot domain generalization (Algorithm 3). The dataflow of the implementation is shown in Figure 3.
Algorithm 1 MetaNorm for Few-Shot Classification
Meta-train: Input values of a over support set aS,i and query set aQ,i; γ, β← Initialize parameters.
µS = 1 |S| |S|∑ i=1 f `µ(aS,i); µQ = 1|Q| |Q|∑ i=1 f `µ(aQ,i);
σS = 1 |S| |S|∑ i=1 f `σ ( (aS,i − µS)2 ) ; σQ = 1|Q| |Q|∑ i=1 f `σ ( (aQ,i − µQ)2 ) ;
a′S,i = γ ( aS,i−µS√ σ2 S + ) + β; a′Q,i = γ ( aQ,i−µS√ σ2 S + ) + β;
LKL = DKL [ N (µS , σS)||N (µQ, σQ) ] return a′S,i = MetaNorm(aS,i); a′Q,i = MetaNorm(aQ,i); LKL
Meta-test: Input values of a over support set aS,i and query set aQ,i;
µS = 1 |S| |S|∑ i=1 f `µ(aS,i);
σS = 1 |S| |S|∑ i=1 f `σ ( (aS,i − µS)2 ) ;
a′Q,i = γ
( aQ,i−µS√
σ2 S +
) + β;
return a′Q,i = MetaNorm(aQ,i)
Algorithm 2 MetaNorm for Domain Generalization
Train: Input values of a over meta-source domain aS,i and meta-target domain aT,i; γ, β← Initialize parameters.
µS,i = f ` µ(aS,i); µS = 1|S| |S|∑ i=1 f `µ(aS,i);
σS,i = f ` σ ( (aS,i − µS,i)2 ) ; σS = 1|S| |S|∑ i=1 f `σ ( (aS,i − µS)2 ) ; µT,i = f ` µ(aT,i); σT,i = f `σ ( (aT,i − µT,i)2 ) ;
a′T,i = γ ( aT,i−µT,i√ σ2 T,i + ) + β;
LKL = DKL [ N (µS,〉, σS,〉)||N (µS , σS) ] return a′T,i = MetaNorm(aT,i); LKL
Test: Input values of a over test domain ai; µi = f ` µ(ai); σi = f ` σ ( (ai − µi)2 ) ;
a′i = γ ( ai−µi√ σ2i + ) + β; return a′i = MetaNorm(ai)

heading: B DATASETS
text: We conduct an extensive set of experiments on a total of 17 datasets containing more than 15 million images. All dataset details and settings are provided in this Appendix.
miniImageNet. The miniImageNet is originally proposed in (Vinyals et al., 2016) and has been widely used for evaluating few-shot learning algorithms. It consists of 60,000 color images from 100
Algorithm 3 MetaNorm for Few-Shot Domain Generalization
Meta-train: Input values of a over meta-source domain set aS,i and meta-target domain set aQ,i; γ, β← Initialize parameters.
µS = 1 |S| |S|∑ i=1 f `µ(aS,i); µQ = 1|Q| |Q|∑ i=1 f `µ(aQ,i);
σS = 1 |S| |S|∑ i=1 f `σ ( (aS,i − µS)2 ) ; σQ = 1|Q| |Q|∑ i=1 f `σ ( (aQ,i − µQ)2 ) ;
a′S,i = γ ( aS,i−µS√ σ2 S + ) + β; a′Q,i = γ ( aQ,i−µS√ σ2 S + ) + β;
LKL = DKL [ N (µS , σS)||N (µQ, σQ) ] return a′S,i = MetaNorm(aS,i); a′Q,i = MetaNorm(aQ,i); LKL
classes with 600 examples per class. The images have dimensions of 84×84 pixels. We follow the train/val/ test split introduced in (Ravi & Larochelle, 2017), which uses 64 classes for meta-training, 16 classes for meta-validation, and the remaining 20 classes for meta-testing.
Omniglot. Omniglot (Lake et al., 2015) is a few-shot learning dataset consisting of 1,623 handwritten characters (each with 20 instances) derived from 50 alphabets. We follow the pre-processing and training procedure defined in (Vinyals et al., 2016). We resize images to 28×28. The training, validation and test sets consist of a random split of 1,100, 100, and 423 characters.
PACS (Li et al., 2017a) contains a total of 9,991 images of the size 224×224 from 4 domains, i.e., photo, art-painting, cartoon and sketch, which demonstrate huge domain gaps. Images are from 7 object classes, i.e., dog, elephant, giraffe, guitar, horse, house, and person. We follow the “leave-one-out” protocol in (Li et al., 2017a; 2018b; Carlucci et al., 2019), where the model is trained on any three of the four domains, which we call source domains, and tested on the last (target) domain. The train-val-test splits are the same as in (Li et al., 2017a).
Office-Home (Venkateswara et al., 2017) also has 4 domains: art, product, clipart and real-world. For each domain, the dataset contains images of 65 object categories found typically in office and home settings. We use the same experimental protocol as for PACS.
DomainNet (Peng et al., 2019) contains 6 distinct domains, i.e., clipart, infograph, painting, quickdraw, real, and sketch for 345 categories. The categories are from 24 divisions, which are: Furniture, Mammal, Tool, Cloth, Electricity, Building, Office, Human Baby, Road Transportation, Food, Nature,
Cold Blooded, Music, Fruit, Sport, Tree, Bird, Vegetable, Shape, Kitchen, Water Transportation, Sky Transportation, Insect, Others.
Meta-Dataset (Triantafillou et al., 2020) is composed of ten (eight train, two test) existing image classification datasets. These are: ILSVRC-2012 (ImageNet, (Russakovsky et al., 2015)), Omniglot (Lake et al., 2015), Aircraft (Maji et al., 2013), CUB-200-2011 (Birds, (Wah et al., 2011)), Describable Textures (Cimpoi et al., 2014), Quick Draw, Fungi, VGG Flowr (Nilsback & Zisserman, 2008), Traffic Signs (Houben et al., 2013) and MSCOCO (Lin et al., 2014). Each episode generated in Meta-Dataset uses classes from a single dataset. Two of these datasets, Traffic Signs and MSCOCO, are fully reserved for evaluation, it means no classes from these sets participate in the training set. Except for Traffic Signs and MSCOCO, the remaining datasets contribute some classes to each of training, validation and test splits of classes. There are about 14 million images in total in Meta-Dataset.

heading: C FEW-SHOT DOMAINNET
text: To construct Few-shot DomainNet, we chose 200 random classes from DomainNet and used 140 for training, 20 for validation and the last 40 for testing. Note that the last 40 object classes were never seen during training. The dataset consists of 200,000 colour images of size 84×84 with each of the 200 classes having 1,000 examples. Please see Table 6, Table 7 and Table 8 for training, validation, and test classes.
D IMPLEMENTATION DETAILS.
In the few-shot learning task, MAML and ProtoNets use a simple CNN containing 4 convolutional layers, each of which is a 3×3 convolution with 32 filters, followed by MetaNorm, a ReLU nonlinearity, and finally a 2×2 max-pooling. VERSA uses a CNN containing 5 convolutional layers, each of which is a 3×3 convolution with 64 filters, followed by MetaNorm, a ReLU non-linearity, and finally a 2×2 max-pooling. In the domain generalization task, we rely on ResNet-18 as backbone for fair comparison with previous work. Each convolutional layer is followed by MetaNorm. The hypernetwork is a 3-layer MLP with 128 units per layer and rectifier nonlinearities. We implemented all models in the Tensorflow framework and tested on an NVIDIA Tesla V100. All code will be available at: https://github.com/YDU-AI/MetaNorm.
D.1 MAML EXPERIMENTS
For MAML experiments, we used the codebase by Finn (Finn, 2017). We use the Adam optimizer with default parameters, and a meta batch size of 4 tasks. The number of test episodes is set as 600. The number of training iterations is 60,000. We set λ=0.001. The other hyper-parameters we use are the default MAML parameters. No early stopping was used. We used the first-order approximation of MAML for the experiments.
D.2 PROTONETS EXPERIMENTS
For ProtoNets, we used the codebase by Fatir (Fatir, 2018). For miniImageNet, we used the following ProtoNets options: a learning rate of 0.001, 60,000 training iterations, 200 validation episodes, 600 test episodes and λ=0.0001. We choose the units of hidden layers and λ by cross-validation. For Meta-Dataset, we reproduce the code provided by CNAPS (Requeima et al., 2019) with TensorFlow. We simply replace its normalization method with our MetaNorm method and add the KL term to the final loss. We are consistent with the dataset configuration and follow the training process as specified in (Triantafillou et al., 2020). The number of training iterations is 80,000. We use a constant learning rate of 0.0001. We set λ=0.001. We follow TaskNorm’s (Bronskill et al., 2020) options: they do not use feature adaptation, and allow updates pre-trained feature extractor weights during meta-training stage.
D.3 VERSA EXPERIMENTS
For VERSA, we used the codebase by Gordon (Gordon, 2019). For the 5-way 5-shot model, we train using the setting of 8 tasks per batch for 100,000 iterations and use a constant learning rate of 0.0001, λ= 0.001. For the 5-way 1-shot model, we train with the setting of 8 tasks per batch for 150,000 iterations and use a constant learning rate of 0.00025, λ=0.01. We set validation episodes as 200, and test episodes as 600. The units of hidden layers and λ were chosen by cross-validation.

heading: E EXTRA RESULTS FOR EFFECT OF KL TERM
text: In this Appendix we consider extra results for the ablation on measuring the effect of the KL term. We report results for few-shot classification on miniImageNet with ProtoNets (Snell et al., 2017) and VERSA (Gordon et al., 2019) in Table 11. We also report domain generalization results on Office-Home in Table 12. In all cases the KL term is crucial.

heading: F SENSITIVITY TO ALGORITHM ON OMNIGLOT
text: The experiments on Omniglot for few-shot classification under the meta-learning settings of MAML, VERSA and ProtoNets are reported in Tables 13, 14 and 15. MetaNorm consistently outperforms both transductive and non-transductive normalization approaches.

heading: G SENSITIVITY TO DATASET
text: The complete set of results for each of the thirteen datasets in Meta-Dataset are provided in Table 16.

heading: H TRAINING SPEED
text: We plot the training loss versus training iterations by using the ProtoNets algorithm in Figure 4. MetaNorm achieves fastest training convergence. From Table 2 and Figure 4, MetaNorm achieves best classification accuracy and training efficiency, which demonstrates the benefit of leveraging meta-learning by MetaNorm for batch normalization.
Ta bl
e 16
:S en
si tiv
ity to
D at
as et
.F ew
-s ho
tc la
ss ifi
ca tio
n re
su lts
on M
et a-
D at
as et
us in
g Pr
ot oN
et s.
Th e ±
si gn
in di
ca te
s th
e 95
% co
nfi de
nc e
in te
rv al
ov er
ta sk
s. B
es t
pe rf
or m
in g
m et
ho ds
an d
an y
ot he
rr un
s w
ith in
th e
95 %
co nfi
de nc
e m
ar gi
n in
bo ld
.R es
ul ts
of ot
he rm
et ho
ds pr
ov id
ed by
B ro
ns ki
ll et
al .(
20 20
).
IL SV
R C
O m
ni gl
ot A
ir cr
af t
B ir
ds Te
xt ur
es Q
ui ck
D ra
w Fu
ng i
V G
G Fl
ow er
Tr af
fic Si
gn s
M SC
O C
O M
N IS
T C
IF A
R 10
C IF
A R
10 0
R an
k
T B
N 44
.7 ±
1. 0
90 .7 ±
0. 6
83 .3 ±
0. 6
69 .6 ±
0. 9
61 .2 ±
0. 7
75 .0 ±
0. 8
46 .4 ±
1. 0
83 .1 ±
0. 6
64 .0 ±
0. 8
38 .2 ±
1. 0
93 .4 ±
0. 4
64 .7 ±
0. 8
48 .0 ±
1. 1
4. 81
C B
N (I
of fe
& Sz
eg ed
y, 20
15 )
43 .6 ±
1. 0
77 .5 ±
1. 1
77 .0 ±
0. 7
67 .5 ±
0. 9
57 .7 ±
0. 7
62 .1 ±
1. 0
43 .6 ±
1. 0
82 .3 ±
0. 6
59 .5 ±
0. 8
36 .6 ±
1. 0
86 .5 ±
0. 6
57 .3 ±
0. 8
43 .1 ±
1. 0
9. 11
B R
N (I
of fe
,2 01
7) 43
.0 ±
1. 0
89 .1 ±
0. 7
84 .4 ±
0. 5
69 .0 ±
0. 9
58 .0 ±
0. 7
74 .3 ±
0. 8
46 .5 ±
1. 0
84 .5 ±
0. 6
65 .7 ±
0. 8
38 .4 ±
1. 0
91 .9 ±
0. 4
60 .1 ±
0. 8
43 .9 ±
1. 0
6. 23
L N
(B a
et al
., 20
16 )
33 .9 ±
0. 9
90 .8 ±
0. 6
73 .9 ±
0. 7
54 .1 ±
1. 0
55 .8 ±
0. 7
72 .5 ±
0. 8
33 .2 ±
1. 1
78 .3 ±
0. 8
69 .1 ±
0. 7
30 .1 ±
0. 9
94 .0 ±
0. 4
51 .5 ±
0. 8
34 .0 ±
0. 9
8. 19
IN (U
ly an
ov et
al .,
20 16
) 32
.5 ±
0. 9
83 .4 ±
0. 8
75 .0 ±
0. 6
50 .2 ±
1. 0
45 .3 ±
0. 7
70 .8 ±
0. 8
29 .8 ±
1. 0
69 .4 ±
0. 8
60 .7 ±
0. 8
27 .7 ±
0. 9
87 .4 ±
0. 5
50 .5 ±
0. 8
32 .1 ±
1. 0
10 .6 1 R N (N ic ho le ta l., 20 18 ) 45 .1 ± 1. 0 90 .8 ± 0. 6 80 .9 ± 0. 6 68 .6 ± 0. 9 64 .1 ± 0. 7 75 .4 ± 0. 7 46 .7 ± 1. 0 84 .4 ± 0. 7 66 .0 ± 0. 8 37 .3 ± 1. 0 93 .9 ± 0. 4 62 .3 ± 0. 8 47 .2 ± 1. 1 4. 73 M et aB N (B ro ns ki ll et al ., 20 20 ) 44 .2 ± 1. 0 90 .4 ± 0. 6 82 .3 ± 0. 6 68 .6 ± 0. 8 60 .5 ± 0. 7 74 .2 ± 0. 7 46 .5 ± 1. 0 86 .0 ± 0. 6 63 .2 ± 0. 8 38 .6 ± 1. 1 93 .9 ± 0. 4 63 .0 ± 0. 8 47 .0 ± 1. 0 4. 78 Ta sk N or m -r (B ro ns ki ll et al ., 20 20 ) 42 .7 ± 1. 0 88 .6 ± 0. 7 79 .6 ± 0. 6 64 .2 ± 0. 9 60 .8 ± 0. 7 73 .2 ± 0. 8 42 .3 ± 1. 1 81 .1 ± 0. 7 64 .9 ± 0. 8 35 .4 ± 1. 0 92 .5 ± 0. 4 61 .4 ± 0. 8 45 .2 ± 1. 0 7. 58 Ta sk N or m -L (B ro ns ki ll et al ., 20 20 ) 45 .1 ± 1. 1 90 .2 ± 0. 6 81 .2 ± 0. 6 68 .8 ± 0. 9 63 .4 ± 0. 8 75 .4 ± 0. 7 46 .5 ± 1. 0 82 .9 ± 0. 7 67 .0 ± 0. 7 39 .2 ± 1. 0 91 .9 ± 0. 4 66 .9 ± 0. 8 51 .3 ± 1. 1 4. 09 Ta sk N or m -I (B ro ns ki ll et al ., 20 20 ) 44 .9 ± 1. 0 90 .6 ± 0. 6 84 .7 ± 0. 5 71 .0 ± 0. 9 65 .9 ± 0. 7 77 .5 ± 0. 7 49 .6 ± 1. 1 83 .2 ± 0. 6 65 .8 ± 0. 7 38 .5 ± 1. 0 93 .3 ± 0. 4 67 .6 ± 0. 8 50 .0 ± 1. 0 3. 07 M et aN or m 45 .3 ± 1. 0 90 .8 ± 0. 5 83 .3 ± 0. 6 70 .6 ± 0. 8 66 .7 ± 0. 6 77 .6 ± 0. 6 51 .1 ± 0. 6 86 .3 ± 0. 7 68 .1 ± 0. 6 39 .7 ± 0. 9 92 .1 ± 0. 5 67 .1 ± 0. 7 52 .7 ± 0. 9 2. 35

Review summaries:
This paper describes a new method for normalizing few-shot learning episodes. The authors point out that the statistics of an episode are unreliable when the size of the episode is small or when the data distribution changes from episode to episode. To remedy this, the authors propose a method called ‘MetaNorm’ which uses a meta-learning approach to infer the means and variances to be used in the batch normalization layers that are employed in the feature extractor component. In particular, they meta-learn the parameters for a set of hypernetworks in an amortized fashion that learn to generate the means and variances of the batch normalization layers conditioned on the contents of the episode. The paper focuses entirely on the few-shot image classification scenario where MetaNorm is evaluated in various settings including standard few-shot classification and domain generalization (including a novel few-shot domain generalization setting).

This paper proposes to replace batch normalization statistics, which are typically computed as the batch moments during training or a fixed training average during testing, with the outputs of learned neural networks. These networks are trained to minimize the KL divergence between their output and the expected or desired batch statistics. In this way, the statistics computation is amortized and can hopefully generalize in the face of small batches and distribution shift.