Year: 2021

Abstract:
Model-agnostic meta-learning (MAML) is one of the most popular and widely adopted meta-learning algorithms nowadays, which achieves remarkable success in various learning problems. Yet, with the unique design of nested inner-loop and outer-loop updates which govern the task-specific and meta-model-centric learning respectively, the underlying learning objective of MAML still remains implicit and thus impedes a more straightforward understanding of it. In this paper, we provide a new perspective of the working mechanism of MAML. We discover that MAML is analogous to a meta-learner using a supervised contrastive objective function in classification, where the query features are pulled towards the support features of the same class and against those of different classes, in which such contrastiveness is experimentally verified via an analysis based on the cosine similarity. Moreover, we reveal that the vanilla MAML algorithm has an undesirable interference term originating from the random initialization and the cross-task interaction. We therefore propose a simple but effective technique, zeroing trick, to alleviate such interference. Extensive experiments are then conducted on both miniImagenet and Omniglot datasets to demonstrate the consistent improvement brought by our proposed technique, thus validating its effectiveness.

heading: 1 INTRODUCTION
text: Humans can learn from very few samples. They can readily establish their cognition and understanding to novel tasks, environments, or domains even with very limited experience in the corresponding circumstances. Meta-learning, a subfield of machine learning aims at equipping machines with such capacity to effectively accommodate new scenarios (Vilalta & Drissi, 2002; Grant et al., 2018). Machines learn to extract task-agnostic information so that their performance on unseen tasks can be improved (Hospedales et al., 2020).
One highly influential meta-learning algorithm is Model Agnostic Meta-Learning (MAML) (Finn et al., 2017), which has inspired numerous follow-up extensions (Nichol et al., 2018; Rajeswaran et al., 2019; Liu et al., 2019; Finn et al., 2019; Jamal & Qi, 2019; Javed & White, 2019). MAML estimates a set of model parameters such that an adaptation of the model to a new task only requires some updates to those parameters. We take the few-shot classification task as an example to review the algorithmic procedure of MAML. A few-shot classification problem refers to classifying samples from some classes (i.e. query data) after seeing a few examples per class (i.e. support data). In a meta-learning scenario, we consider a distribution of tasks, where each task is a few-shot classification problem and different tasks have different target classes. MAML aims to meta-train the base-model based on training tasks (i.e., the meta-training dataset) and evaluate the performance of the base-model on the testing tasks, sampled from a held-out unseen dataset (i.e. the meta-testing dataset). In meta-training, MAML follows a bi-level optimization scheme composed of the inner loop and the outer loop, as shown in Appendix A (please refer to Section 2 for detailed definition). In the inner loop (also known as fast adaptation), the base-model θ is updated to θ′ using the support set. In the outer loop, a loss is evaluated on θ′ using the query set, and its gradient is computed with respect to θ to update the base-model. Since the outer loop requires computing the gradient of gradient (as the update in the inner loop is included in the entire computation graph), it is called second-order MAML (SOMAML). To prevent computing the Hessian matrix, Finn et al. (2017) propose first-order MAML (FOMAML) that uses the gradient computed with respect to the inner-loop-updated parameters θ′ to update the base-model.
The widely accepted intuition behind MAML is that the models are encouraged to learn a generalpurpose representations which are broadly applicable not only to the seen tasks but also to novel tasks (Finn et al., 2017; Raghu et al., 2020; Goldblum et al., 2020). Raghu et al. (2020) confirm this perspective by showing that during fast adaptation, the majority of changes being made is in the final linear layers, while the convolution layers (as the feature encoder) remain almost static. This implies the models trained with MAML learn a good feature representation and that they only have to change the linear mapping from features to outputs during the fast adaptation. Similar ideas of freezing feature extractor during the inner loop have also been explored (Lee et al., 2019; Bertinetto et al., 2019; Liu et al., 2020), and have been held as an assumption in theoretical works (Du et al., 2021; Tripuraneni et al., 2020; Chua et al., 2021).
While this intuition sounds satisfactory, we step further and ask the following fundamental questions: (1) In what sense does MAML guide any model to learn general-purpose representations? (2) How do the inner and outer loops in the training mechanism of MAML collaboratively prompt to achieve so? (3) What is the role of the support and query data and how do they, if any, interact with each other? In this paper, we answer these questions and give new insights on the working mechanism of MAML, which turns out to be closely connected to supervised contrastive learning (SCL)1.
Here, we provide a sketch of our analysis in Figure 1. We consider a setting of (a) a 5-way 1-shot paradigm of few-shot learning, (b) the mean square error (MSE) between the one-hot encoding of groundtruth label and the outputs as the objective function, and (c) MAML with a single inner-loop update. At the beginning of the inner loop, we set the linear layer w0 to zero. Then, the inner loop update of w0 is equivalent to adding the support features to w0. In the outer loop, the output of a query sample q1 is actually the inner product between the query feature ϕ(q1) and all support features (the learning rate is omitted for now). As the groundtruth is an one-hot vector, the encoder is trained to either minimize the inner product between the query features and the support features (when they are from different classes, as shown in the green box), or to pull the inner product between the query features and the support features to 1 (when they have the same label, as shown in the red box). Therefore, the inner loop and the outer loop together manifest a SCL objective. Particularly, as the vanilla implementation of MAML uses non-zero (random) initialization for the linear layer, we will show such initialization leads to a noisy SCL objective which would impede the training.
In this paper, we firstly review a formal definition of SCL, present a more general case of MAML with cross entropy loss in classification, and show the underlying learning protocol of vanilla MAML as an interfered SCL in Section 2. We then experimentally verify the supervised contrastiveness of MAML, and propose to mitigate the interference with our simple but effective technique of the zeroinitialization and zeroing trick (cf. Section 3). In summary, our main contributions are three-fold:
• We show that MAML is implicitly an SCL algorithm in classification and that the noise comes from the randomly initialized linear layer and the cross-task interaction, interfering the training of the encoder.
• We verify the inherent contrastiveness of MAML based on the cosine similarity analysis.
• We experimentally validate our analysis and show that applying the zeroing trick induces a notable improvement in testing accuracy during training. We also show that during meta-testing, a pronounced increase in accuracy occurs when the zeroing trick is applied.

heading: 2 WHY MAML IS IMPLICITLY A NOISY SUPERVISED CONTRASTIVE ALGORITHM?

heading: 2.1 PRELIMINARY: SUPERVISED CONTRASTIVE LEARNING
text: In this work, we aim to bridge MAML and supervised contrastive learning (SCL) and attribute the success of MAML to SCL’s capacity in learning good representations. Thus, we would like to briefly introduce SCL.
1We use the term supervised contrastiveness to refer to the setting of using groundtruth label information to differentiate positive samples and negative samples (Khosla et al., 2020). This setting is different from (unsupervised/self-supervised) contrastive learning.
Supervised contrastive learning, proposed by Khosla et al. (2020), is a generalization of several metric learning algorithms, such as triplet loss and N-pair loss (Schroff et al., 2015; Sohn, 2016) and has shown the best performance in classification compared to SimCLR and CrossEntropy. In Khosla et al. (2020), SCL is described as “contrasts the set of all samples from the same class as positives against the negatives from the remainder of the batch” and “embeddings from the same class are pulled closer together than embeddings from different classes.” For a sample s, label information is leveraged to indicate positive samples (i.e., samples having the same label as sample s) and negative samples (i.e., samples having different labels to sample s). The loss of SCL is designed to increase the similarity (or decrease metric distance) of embeddings of positive samples and to decrease the similarity (or increase the metric distance) of embeddings of negative samples (Khosla et al., 2020). In essence, SCL combines supervised learning and contrastive learning and differs from supervised learning in that the loss contains a measurement of the similarity or metric distance between the embedding of a sample and embedding of its positive/negative sample pairs.
Now we give a formal definition of SCL. For a set of N samples drawn from a n-class dataset. Let i ∈ I = {1, ..., N} be the index of an arbitrary sample. Let A(i) = I \ {i}, P (i) be the set of
indices of all positive samples of sample i, and N(i) = A(i) \ P (i) be the set of indices of all negative samples of sample i. Let zi indicates the embedding of sample i.
Definition 1 Let Msim be a measurement of similarity (e.g., inner product, cosine similarity). Training algorithms that adopt loss of the following form belong to SCL:
LSCL = ∑ i ∑ p∈P (i) c−p,iMsim(zi, zp) + ∑ i ∑ n∈N(i) c+n,iMsim(zi, zn) + c (1)
where c−p,i < 0 and c + n,i > 0 for all n, p and i; and c is a constant independent of samples.
We further define that a training algorithm that follows Eq.(1), but with either (a) c+n,i < 0 for some n, i or (b) c is a constant dependent of samples, belongs to noisy SCL.

heading: 2.2 PROBLEM SETUP
text: We provide the detailed derivation to show that MAML is implicitly a noisy SCL, where we adopt the few-shot classification as the example application. In this section, we focus on the meta-training period. Consider drawing a batch of tasks {T1, . . . , TNbatch} from a meta-training task distribution D. Each task Tn contains a support set Sn and a query set Qn, where Sn = {(sm, tm)} Nway×Nshot m=1 , Qn = {(qm, um)} Nway×Nquery m=1 , sm, qm ∈ RNin are data samples, and tm, um ∈ {1, ..., Nway} are labels. We denote Nway the number of classes in each task, and {Nshot, Nquery} respectively the number of support and query samples per class. The architecture of our base-model comprises of a convolutional encoder ϕ : RNin → RNf (parameterized by φ), a fully connected linear head w ∈ RNf×Nway , and a Softmax output layer, where Nf is the dimension of the feature space. We denote the kth column of w as wk. Note that the base-model parameters θ consist of φ and w.
As shown in Appendix A, both FOMAML and SOMAML adopt a training strategy comprising the inner loop and the outer loop. At the beginning of a meta-training iteration, we sample Nbatch tasks. For each task Tn, we perform inner loop updates using the inner loop loss (c.f. Eq. (2)) evaluated on the support data, and then evaluate the outer loop loss (c.f. Eq. (3)) on the updated base-model using the query data. In the ith step of the inner loop, the parameters {φi−1,wi−1} are updated to {φi,wi} using the multi-class cross entropy loss evaluated on the support dataset Sn as
L{φi,wi},Sn = E (s,t)∼Sn Nway∑ j=1 1j=t[− log exp(ϕi(s)⊤wj i)∑Nway k=1 exp(ϕ i(s)⊤wki) ] (2)
After Nstep inner loop updates, we compute the outer loop loss using the query data Qn:
L{φNstep ,wNstep},Qn = E(q,u)∼Qn [− log exp(ϕ
Nstep(q)⊤wu Nstep)∑Nway
k=1 exp(ϕ Nstep(q)⊤wkNstep)
] (3)
Then, we sum up the outer loop losses of all tasks, and perform gradient descent to update the base-model’s initial parameters {φ0,w0}. To show the supervised contrastiveness entailed in MAML, we adopt an assumption that the Encoder ϕ is Frozen during the Inner Loop (the EFIL assumption) and we discuss the validity of the assumption in Section 2.6. Without loss of generality, we consider training models with MAML with Nbatch = 1 and Nstep = 1, and we discuss the generalized version in Section 2.6. For simplicity, the kth element of model output exp(ϕ(s) ⊤wk
0)∑Nway j=1 exp(ϕ(s) ⊤wj0) (respectively exp(ϕ(q) ⊤wk 1)∑Nway j=1 exp(ϕ(q) ⊤wj1) ) of sample
s (respectively q) is denoted as sk (respectively qk).

heading: 2.3 INNER LOOP AND OUTER LOOP UPDATE OF LINEAR LAYER AND ENCODER
text: In this section, we primarily focus on the update of parameters in the case of FOMAML. The full derivation and discussion of SOMAML are provided in Appendix B.
Inner loop update of the linear layer. In the inner loop, the linear layer w0 is updated to w1 with a learning rate η as shown in Eq. (4) in both FOMAML and SOMAML. In contrast to the example in Figure 1, the columns of the linear layer are added with the weighted sum of the features extracted
from support samples (i.e., support features). Compared to wk0, wk1 is pushed towards the support features of the same class (i.e., class k) with strength of 1 − sk, while being pulled away from the support features of different classes with strength of sk.
wk 1 = wk 0 − η ∂L{φ,w0},S
∂wk0 = wk 0 + η E (s,t)∼S (1k=t − sk)ϕ(s) (4)
Outer loop update of the linear layer. In the outer loop, w0 is updated using the query data with a learning rate ρ. For FOMAML, the final linear layer is updated as follows.
w′k 0 = wk 0 − ρ ∂L{φ,w1},Q
∂wk1 = wk 0 + ρ E (q,u)∼Q (1k=u − qk)ϕ(q) (5)
Note that the computation of qk requires the inner-loop updated w 1. Generally speaking, Eq. (5)
resembles Eq. (4). It is obvious that, in the outer loop, the query features are added weightedly to the linear layer, and the strength of change relates to the output value. In other words, after the outer loop update, the linear layer memorizes the query features of current tasks. This can cause a crosstask interference because in the next inner loop there would be additional inner products between the support features of the next tasks and the query features of the current tasks.
Outer loop update of the encoder. Using the chain rule, the gradient of the outer loop loss with respect to φ (i.e., the parameters of the encoder) is given by
∂L{φ,w1},Q ∂φ = E(q,u)∼Q ∂L{φ,w1},Q ∂ϕ(q) ∂ϕ(q) ∂φ +
E(s,t)∼S ∂L{φ,w1},Q ∂ϕ(s) ∂ϕ(s) ∂φ , where the second term can be neglected when FOMAML is considered. Below, we take a deeper look at the backpropagated error of one query data (q, u) ∼ Q. The full derivation is provided in Appendix B.2.
∂L{φ,w1},q ∂ϕ(q) = Nway∑ j=1 (qj − 1j=u)wj 0 + η E (s,t)∼S [−( Nway∑ j=1 qjsj) + su + qt − 1t=u]ϕ(s) (6)

heading: 2.4 MAML IS A NOISY CONTRASTIVE LEARNER
text: Reformulating the outer loop loss for the encoder as a noisy SCL loss. We can observe from Eq. (6) that the actual loss for the encoder (evaluated on a single query data (q, u) ∼ Q) is as the following.
L{φ,w1},q = Nway∑ j=1 (qj − 1j=u)wj 0⊤
stop gradient
ϕ(q) + η E (s,t)∼S [− Nway∑ j=1 qjsj + su + qt − 1t=u]ϕ(s) ⊤
stop gradient
ϕ(q) (7)
For SOMAML, the range of “stop gradient” in the second term is different:
L{φ,w1},q = Nway∑ j=1 (qj − 1j=u)wj 0⊤
stop gradient
ϕ(q) + η E (s,t)∼S [− Nway∑ j=1 qjsj + su + qt − 1t=u]
stop gradient
ϕ(s)⊤ϕ(q) (8)
With these two reformulations, we observe the essential difference between FOMAML and SOMAML is the range of stop gradient. We provide detailed discussion and instinctive illustration in Appendix B.5 on how this explains the phenomenon that SOMAML often leads to faster convergence. To better deliberate the effect of each term in the reformulated outer loop loss, we define the first term in Eq. (7) or Eq. (8) as interference term, the second term as noisy contrastive term, and the coefficients − ∑Nway j=1 qjsj + su + qt − 1t=u as contrastive coefficients.
Understanding the interference term. In the case of j = u, the outer loop loss forces the model to minimize (qj − 1)wj0⊤ϕ(q). This can be problematic because (a) at the beginning of training, w0 is assigned with random values and (b) w0 is added with query features of previous tasks as shown in Eq. (5). Consequently, ϕ(q) is pushed to a direction composed of previous query features or to a random direction, introducing an unnecessary cross-task interference or an initialization interference that slows down the training of the encoder. Noting that the cross-task interference also occurs at the testing period, since, at testing stage, w0 is already added with query features of training tasks, which can be an interference to testing tasks.
Understanding the noisy contrastive term. When the query and support data have the same label (i.e., u = t), e.g., class 1, the contrastive coefficients becomes − ∑Nway j=2 qjsj − q1s1 + s1 + q1− 1,
which is − ∑Nway
j=2 qjsj − (1 − q1)(1 − s1) < 0. This indicates the encoder would be updated to maximize the inner product between ϕ(q) and the support features of the same class. However, when the query and support data are in different classes, the sign of the contrastive coefficient can sometimes be negative. The outer loop loss thus cannot well contrast the query features against the support features of different classes, making this loss term not an ordinary SCL loss.
To better illustrate the influence of the interference term and the noisy contrastive term, we provide an ablation experiment in Appendix B.7. Theorem 1 below formally connects MAML to SCL.
Theorem 1 With the EFIL assumption, FOMAML is a noisy SCL algorithm. With assumptions of (a) EFIL and (b) a single inner-loop update, SOMAML is a noisy SCL algorithm.
Proof: For FOMAML, both Eq. (7) (one inner loop update step) and Eq. (26) (multiple inner loop update steps) follows Definition 1. For SOMAML, Eq. (8) follows Definition 1.
Introduction of the zeroing trick makes Eq. (7) and Eq. (8) SCL losses. To tackle the interference term and make the contrastive coefficients more accurate, we introduce the zeroing trick: setting the w0 to be zero after each outer loop update, as shown in Appendix A. With the zeroing trick, the original outer loop loss (of FOMAML) becomes
L{φ,w1},q = η E (s,t)∼S
(qt − 1t=u)ϕ(s) ⊤
stop gradient
ϕ(q) (9)
For SOMAML, the original outer loop loss becomes
L{φ,w1},q = η E (s,t)∼S (qt − 1t=u) stop gradient ϕ(s)⊤ϕ(q) (10)
The zeroing trick brings two nontrivial effects: (a) eliminating the interference term in both Eq. (7) and Eq. (8); (b) making the contrastive coefficients follow SCL. For (b), since all the predictive values of support data become the same, i.e., sk = 1Nway , the contrastive coefficient becomes qt − 1t=u, which is negative when the support and query data have the same label, and positive otherwise. With the zeroing trick, the contrastive coefficient follows the SCL loss, as summarized below.
Corollary 1 With mild assumptions of (a) EFIL, (b) a single inner-loop update and (c) training with the zeroing trick (i.e., the linear layer is zeroed at the end of each outer loop), both FOMAML and SOMAML are SCL algorithms.
Proof: Both Eq. (9) and Eq. (10) follow Definition 1.
The introduction of the zeroing trick makes the relationship between MAML and SCL more straightforward. Generally speaking, by connecting MAML and SCL, we can better understand other MAML-based meta-learning studies.

heading: 2.5 RESPONSES TO QUESTIONS IN SECTION 1
text: In what sense does MAML guide any model to learn general-purpose representations? Under the EFIL assumption, MAML is a noisy SCL algorithm in a classification paradigm. The effectiveness of MAML in enabling models to learn general-purpose representations can be attributed to the SCL characteristics of MAML.
How do the inner and outer loops in the training mechanism of MAML collaboratively prompt to achieve so? MAML adopts the inner loop and the outer loop to perform noisy SCL in a sequential manner. In the inner loop, the features of support data are memorized by w via inner-loop update. In the outer loop, the softmax output of the query data thus contains the inner products between the support features and the query feature.
What is the role of the support and query data and how do they, if any, interact with each other? We show that the original loss in MAML can be reformulated as a loss term containing the inner products of the embedding of the support and query data. In FOMAML, the support features act as the reference, while the query features are updated to move towards the support features of the same class and against those of the different classes.

heading: 2.6 GENERALIZATION OF OUR ANALYSIS
text: In Appendix C, we provide the analysis where Nbatch ≥ 1 and Nstep ≥ 1. For the EFIL assumption, it can hardly be dropped because the behavior of the updated encoder is intractable. Besides, Raghu et al. (2020) show that the representations of intermediate layers do not change notably during the inner loop of MAML, and thus it is understood that the main function of the inner loop is to change the final linear layer. Furthermore, the EFIL assumption is empirically reasonable, since previous works (Raghu et al., 2020; Lee et al., 2019; Bertinetto et al., 2019; Liu et al., 2020) yield comparable performance while leaving the encoder untouched during the inner loop.
With our analysis, one may notice that MAML is approximately a metric-based few-shot learning algorithm. From a high-level perspective, under the EFIL assumption, second-order MAML is similar to metric-based few-shot learning algorithms, such as MatchingNet (Vinyals et al., 2016), Prototypical network (Snell et al., 2017), and Relation network (Sung et al., 2018). The main difference lies in the metric and the way prototypes are constructed. Our work follows the setting adopted by MAML, such as using negative LogSoftmax as objective function, but we can effortlessly generalize our analysis to a MSE loss as had been shown in Figure 1. As a result, our work points out a new research direction in improving MAML by changing the objective functions in the inner and the outer loops, e.g., using MSE for the inner loop but negative LogSoftmax for the outer loop. Besides, in MAML, we often obtain the logits by multiplying the features by the linear weight w. Our work implies future direction as to alternatively substitute this inner product operation with other metrics or other similarity measurements such as cosine similarity or negative Euclidean distance.

heading: 3 EXPERIMENTAL RESULTS
text: In this section, we provide empirical evidence of the supervised contrastiveness of MAML and show that zero-initialization of w0, reduction in the initial norm of w0, or the application of zeroing trick can speed up the learning profile. This is applicable to both SOMAML and FOMAML.

heading: 3.1 SETUP
text: We conduct our experiments on the miniImagenet dataset (Vinyals et al., 2016; Ravi & Larochelle, 2017) and the Omniglot dataset (Lake et al., 2015). For the results on the Omniglot dataset, please refer to Appendix E. For the miniImagenet, it contains 84 × 84 RGB images of 100 classes from the ImageNet dataset with 600 samples per class. We split the dataset into 64, 16 and 20 classes for training, validation, and testing as proposed in (Ravi & Larochelle, 2017). We do not perform hyperparameter search and thus are not using the validation data. For all our experiments of applying MAML into few-shot classification problem, where we adopt two experimental settings: 5-way 1- shot and 5-way 5-shot, with the batch size Nbatch being 4 and 2, respectively (Finn et al., 2017). The few-shot classification accuracy is calculated by averaging the results over 400 tasks in the test phase. For model architecture, optimizer and other experimental details, please refer to Appendix D.1.

heading: 3.2 COSINE SIMILARITY ANALYSIS VERIFIES THE IMPLICIT CONTRASTIVENESS IN MAML
text: In Section 2, we show that the encoder is updated so that the query features are pushed towards the support features of the same class and pulled away from those of different classes. Here we verify this supervised contrastiveness experimentally. Consider a relatively overfitting scenario where there are five classes of images and for each class there are 20 support images and 20 query images. We fix the support and query set (i.e. the data is not resampled every iteration) to verify the concept that the support features work as positive and negative samples. Channel shuffling is used to avoid the undesirable channel memorization effect (Jamal & Qi, 2019; Rajendran et al., 2020). We train the model using FOMAML and examine how well the encoder can separate the data of different classes in the feature space by measuring the averaged cosine similarities between the features of each class. The results are averaged over 10 random seeds.
As shown in the top row of Figure 2, the model trained with MAML learns to separate the features of different classes. Moreover, the contrast between the diagonal and the off-diagonal entries of the heatmap increases as we remove the initialization interference (by zero-initializing w0, shown in the middle row) and remove the cross-task interference (by applying the zeroing trick, shown in the
bottom row). The result agrees with our analysis that MAML implicitly contains the interference term which can impede the encoder from learning a good feature representation. For experiments on semantically similar classes of images, the result is shown in Section D.3 and displays high consistency.

heading: 3.3 ZEROING LINEAR LAYER AT TESTING TIME INCREASES TESTING ACCURACY
text: Before starting our analysis on benchmark datasets, we note that the cross-task interference can also occur during meta-testing. In the meta-testing stage, the base-model is updated in the inner loop using support data S and then the performance is evaluated using query data Q, where S and Q are drawn from a held-out, unseen meta-testing dataset. Recall that at the end of the outer loop (in meta-training stage), the query features are added weightedly to the linear layer w0. In other words, at the beginning of meta-testing, w0 is already added with the query features of previous training tasks, which can drastically influence the performance on the unseen tasks.
To validate this idea, we apply the zeroing trick at meta-testing time (which we refer to zeroing w0 at the beginning of the meta-testing time) and show such trick increases the testing accuracy of the model trained with FOMAML. As illustrated in Figure 3, compared to directly entering meta-testing (i.e. the subplot at the left), additionally zeroing the linear layer at the beginning of each meta-testing time (i.e. the subplot at the right) increases the testing accuracy of the model whose linear layer is randomly initialized or zero-initialized (denoted by the red and orange curves, respectively). And the difference in testing performance sustains across the whole training session.
In the following experiments, we evaluate the testing performance only with zeroing the linear layer at the beginning of the meta-testing stage. By zeroing the linear layer, the potential interference brought by the prior (of the linear layer) is ignored. Then, we can fully focus on the capacity of the encoder in learning a good feature representation.

heading: 3.4 SINGLE INNER LOOP UPDATE SUFFICES WHEN USING THE ZEROING TRICK
text: In Eq. (4) and Eq. (21), we show that the features of the support data are added to the linear layer in the inner loop. Larger number of inner loop update steps can better offset the effect of interference brought by a non-zeroed linear layer. In other words, when the models are trained with the zeroing trick, a larger number of inner loop updates can not bring any benefit. We validate this intuition in Figure 4 under a 5-way 1-shot setting. In the original FOMAML, the models trained with a single inner loop update step (denoted as red curve) converge slower than those trained with update step of 7 (denoted as purple curve). On the contrary, when the models are trained with the zeroing trick, models with various inner loop update steps converge at the same speed.

heading: 3.5 EFFECT OF INITIALIZATION AND THE ZEROING TRICK
text: In Eq. (7), we observe an interference derived from the historical task features or random initialization. We validate our formula by examining the effects of (1) reducing the norm of w0 at initialization and (2) applying the zeroing trick. From Figure 5, the performance is higher when the initial norm of w0 is lower. Compared to random initialization, reducing the norm via down-scaling w0 by 0.7 yields visible differences. Besides, the testing accuracy of MAML with zeroing trick (the purple curve) outperforms that of original MAML.

heading: 4 CONCLUSION
text: In this paper, we perform an extensive study to demystify how the seminal MAML algorithm guides the encoder to learn a general-purpose feature representation and what is the interaction between the support and query set. Our analysis shows that MAML is implicitly a supervised contrastive learner using the support features as positive and negative samples to direct the update of the encoder. Moreover, we unveil an interference term hidden in MAML originated from the random initialization or from the cross-task interaction, which can impede the representation learning. Driven by our analysis, removing the interference term by a simple zeroing trick renders the model unbiased to seen or unseen tasks. With this zeroing trick, we show constant improvements in both the training and testing profiles, with experiments conducted on the miniImagenet and Omniglot datasets.

heading: APPENDIX

heading: A ORIGINAL MAML AND MAML WITH THE ZEROING TRICK
text: Algorithm 1 Second-order MAML Require: Task distribution D Require: η, ρ: inner loop and outer loop learning rates Require: Randomly initialized base-model parameters θ
1: while not done do 2: Sample tasks {T1, . . . TNbatch} from D 3: for n = 1, 2, . . . , Nbatch do 4: {Sn, Qn} ← sample from Tn 5: θn = θ 6: for i = 1, 2, . . . , Nstep do 7: θn ← θn − η∇θnLθn,Sn 8: end for 9: end for
10: Update θ ← θ − ρ ∑Nbatch
n=1 ∇θLθn,Qn 11: end while
Algorithm 2 First-order MAML Require: Task distribution D Require: η, ρ: inner loop and outer loop learning rates Require: Randomly initialized base-model parameters θ
1: while not done do 2: Sample tasks {T1, . . . TNbatch} from D 3: for n = 1, 2, . . . , Nbatch do 4: {Sn, Qn} ← sample from Tn 5: θn = θ 6: for i = 1, 2, . . . , Nstep do 7: θn ← θn − η∇θnLθn,Sn 8: end for 9: end for
10: Update θ ← θ − ρ ∑Nbatch
n=1 ∇θnLθn,Qn 11: end while
Algorithm 3 Second-order MAML with the zeroing trick Require: Task distribution D Require: η, ρ: inner loop and outer loop learning rates Require: Randomly initialized base-model parameters θ
1: Set w← 0 (the zeroing trick) 2: while not done do 3: Sample tasks {T1, . . . TNbatch} from D 4: for n = 1, 2, . . . , Nbatch do 5: {Sn, Qn} ← sample from Tn 6: θn = θ 7: for i = 1, 2, . . . , Nstep do 8: θn ← θn − η∇θnLθn,Sn 9: end for
10: end for 11: Update θ ← θ − ρ ∑Nbatch n=1 ∇θLθn,Qn 12: Set w← 0 (the zeroing trick) 13: end while

heading: B SUPPLEMENTARY DERIVATION
text: In this section, we provide the full generalization and further discussion that supplement the main paper. We consider the case of Nbatch = 1 and Nstep = 1 under the EFIL assumption. We provide the outer loop update of the linear layer under SOMAML in Section B.1. Next, we offer the full derivation of the outer loop update of the encoder in Section B.2. Then, we reformulate the outer loop loss for the encoder in both FOMAML and SOMAML in Section B.3 and Section B.4. Afterward, we discuss the main difference in FOMAML and SOMAML in detail in Section B.5. Finally, we show the performance of the models trained using the reformulated loss in Section B.6.

heading: B.1 THE DERIVATION OF OUTER LOOP UPDATE FOR THE LINEAR LAYER USING SOMAML
text: Here, we provide the complete derivation of the outer loop update for the linear layer. Using SOMAML with support set S and query set Q, the update of the linear layer follows
w′0k = wk 0 − ρ ∂L{φ,w1},Q ∂wk0 = wk 0 − ρ Nway∑ m=1 ∂wm 1 ∂wk0 · ∂L{φ,w1},Q ∂wm1
= wk 0 − ρ∂wk
1
∂wk0 · ∂L{φ,w1},Q ∂wk1 − ρ Nway∑ m ̸=k ∂wm 1 ∂wk0 · ∂L{φ,w1},Q ∂wm1
= wk 0 + ρ[I − η E (s,t)∼S (sk − s2k)ϕ(s)ϕ(s)T ] E (q,u)∼Q (1k=u − qk)ϕ(q)
+ ρη ∑ m ̸=k [ E (s,t)∼S (smsk)ϕ(s)ϕ(s)T ][ E (q,u)∼Q (1m=u − qm)ϕ(q)]
= wk 0 + ρ[I − η E (s,t)∼S skϕ(s)ϕ(s)T ] E (q,u)∼Q (1k=u − qk)ϕ(q)
+ ρη Nway∑ m=1 [ E (s,t)∼S (smsk)ϕ(s)ϕ(s)T ][ E (q,u)∼Q (1m=u − qm)ϕ(q)]
(11)
We can further simplify Eq. (11) to Eq. (12) with the help of the zeroing trick.
w′0k = ρ[I − η E (s,t)∼S skϕ(s)ϕ(s)T ] E (q,u)∼Q (1k=u − qk)ϕ(q) (12)
This is because the zeroing trick essentially turns the logits of all support samples to zero, and consequently the predicted probability (softmax) output sm becomes 1Nway for all channel m. Therefore, the third term in Eq. (11) turns out to be zero (c.f. Eq. (13)). The equality of Eq. (13) holds since the summation of the (softmax) outputs is one.
ρη
N2way Nway∑ m=1 [ E (s,t)∼S ϕ(s)ϕ(s)T ][ E (q,u)∼Q (1m=u − qm)ϕ(q)]
= ρη
N2way [ E (s,t)∼S ϕ(s)ϕ(s)T ] E (q,u)∼Q ϕ(q) Nway∑ m=1 (1m=u − qm) = 0
(13)

heading: B.2 THE FULL DERIVATION OF THE OUTER LOOP UPDATE OF THE ENCODER.
text: As the encoder ϕ is parameterized by φ, the outer loop gradient with respect to φ is given by ∂L{φ,w1},Q
∂φ = E(q,u)∼Q ∂L{φ,w1},Q ∂ϕ(q) ∂ϕ(q) ∂φ +E(s,t)∼S ∂L{φ,w1},Q ∂ϕ(s) ∂ϕ(s) ∂φ . We take a deeper look at the
backpropagated error ∂L{φ,w1},Q
∂ϕ(q) of the feature of one query data (q, u) ∼ Q, based on the following
form:
− ∂L{φ,w1},Q
∂ϕ(q) = wu 1 − Nway∑ j=1 (qjwj 1) = Nway∑ j=1 (1j=u − qj)wj 1
= Nway∑ j=1 (1j=u − qj)wj 0 + η Nway∑ j=1 [1j=u − qj ][ E (s,t)∼S (1j=t − sj)ϕ(s)]
= Nway∑ j=1 (1j=u − qj)wj 0 + η E (s,t)∼S [( Nway∑ j=1 qjsj)− su − qt + 1t=u]ϕ(s)
(14)

heading: B.3 REFORMULATION OF THE OUTER LOOP LOSS FOR THE ENCODER AS NOISY SCL LOSS.
text: We can derive the actual loss (evaluated on a single query data (q, u) ∼ Q) that the encoder uses under FOMAML scheme as follows:
L{φ,w1},q = Nway∑ j=1 (qj − 1j=u)wj0⊤
stop gradient
ϕ(q)− η E (s,t)∼S [( Nway∑ j=1 qjsj)− su − qt + 1t=u]ϕ(s) ⊤
stop gradient
ϕ(q) (15)
For SOMAML, we need to additionally plug Eq. (4) into Eq. (3).
L{φ,w1},q = Nway∑ j=1 (qj − 1j=u)wj0⊤
stop gradient
ϕ(q)− η E (s,t)∼S [( Nway∑ j=1 qjsj)− su − qt + 1t=u]
stop gradient
ϕ(s)⊤ϕ(q) (16)
B.4 INTRODUCTION OF THE ZEROING TRICK MAKES EQ. (7) AND EQ. (8) SCL LOSSES.
Apply the zeroing trick to Eq. (7) and Eq. (8), we can derive the actual loss Eq. (17) and Eq. (18) that the encoder follows.
L{φ,w1},q = η E (s,t)∼S
(qt − 1t=u)ϕ(s) ⊤
stop gradient
ϕ(q) (17)
L{φ,w1},q = η E (s,t)∼S (qt − 1t=u) stop gradient ϕ(s)⊤ϕ(q) (18)
With these two equations, we can observe the essential difference in FOMAML and SOMAML is the range of stopping gradients. We would further discuss the implication of different ranges of gradient stopping in Appendix B.5.

heading: B.5 DISCUSSION ABOUT THE DIFFERENCE BETWEEN FOMAML AND SOMAML
text: Central to the mystery of MAML is the difference between FOMAML and SOMAML. There are plenty of works dedicated to approximating or estimating the second-order derivatives in the MAML algorithm in a more computational-efficient or in an accurate manner (Song et al., 2020; Rothfuss et al., 2019; Liu et al., 2019). With the EFIL assumption and our analysis through connecting SCL to these algorithms, we found that we can better understand the distinction between FOMAML and SOMAML from a novel perspective. To better understand the difference, we can compare Eq. (7) with Eq. (8) or compare Eq. (9) with Eq. (10). To avoid being distracted by the interference terms, we provide the analysis of the latter.
The main difference between Eq. (9) and Eq. (10) is the range of gradient stopping and we will show that this difference results in a significant distinction in the feature space. To begin with, by chain rule, we have ∂L∂φ = E(q,u)∼Q ∂L ∂ϕ(q) ∂ϕ(q) ∂φ + E(s,t)∼S ∂L ∂ϕ(s) ∂ϕ(s) ∂φ . As we specifically want to know how the encoded features are updated given different losses, we can look at the terms ∂L∂ϕ(q) and ∂L ∂ϕ(s) by differentiating Eq. (9) and Eq. (10) with respect to the features of query data q and support data s, respectively.
FOMAML: ∂L
∂ϕ(q) = η E (s,t)∼S (qt − 1t=uϕ(s)
∂L
∂ϕ(s) = 0
(19)
SOMAML: ∂L
∂ϕ(q) = η E (s,t)∼S (qt − 1t=uϕ(s)
∂L
∂ϕ(s) = η E (s,t)∼S (qt − 1t=u)ϕ(q)
(20)
Obviously, as the second equation in Eq (19) is zero, we know that in FOMAML, the update of the encoder does consider the change of the support features. The encoder is updated to move the query features closer to support features of the same class and further to support features of different classes in FOMAML. On the contrary, we can tell from the above equations that in SOMAML, the encoder is updated to make support features and query features closer if both of them come from the same class and make support features and query features further if they come from different classes.
We illustrate the difference in Figure 6. For simplicity, we do not consider the scale of the coefficients but their signs. The subplot on the left indicates that this FOMAML loss guides the encoder to be updated so that the feature of the query data 1) moves towards the support feature the same class and 2) against the support features of the different classes. On the other hand, the SOMAML loss guides the encoder to be updated so that 1) when the support data and query data belong to the same class, their features move closer, and otherwise their features move further. This generally explains why models trained using SOMAML generally converges faster than those trained using FOMAML.

heading: B.6 EXPLICITLY COMPUTING THE REFORMULATING LOSS USING EQ. (7) AND EQ. (8)
text: Under the EFIL assumption, we show that MAML can be reformulated as a loss taking noisy SCL form. Below, we consider a setting of 5-way 1-shot miniImagenet few-shot classification task, under the condition of no inner loop update of the encoder. (This is the assumption that our derivation heavily depends on. It means that we now only update the encoder in the outer loop.) We empirically show that explicitly computing the reformulated losses of Eq. (7), Eq. (17) and Eq. (18) yield almost the same curves as MAML (with the EFIL assumption). Please note that the reformulated losses are used to update the encoders, for the linear layer w0, we explicitly update it using Eq. (5). Note that
although the performance models training using FOMAML, FOMAML with the zeroing trick, and SOMAML converge to similar testing accuracy, the overall testing performance during the training process is distinct. The results are averaged over three random seeds.

heading: B.7 THE EFFECT OF INTERFERENCE TERM AND NOISY CONTRASTIVE TERM
text: Reformulating the loss of MAML into a noisy SCL form enables us to further investigate the effects brought by the interference term and the noisy contrastive term, which we presume both effects to be negative. To investigate the effect of the interference term, we simply consider the loss in Eq. (7) with the interference term dropped (denoted as “ITF ×”). As for the noisy contrastive term, the noise comes from the fact that “when the query and support data are in different classes, the sign of the contrastive coefficient can sometimes be negative”, as being discussed in Section 2.4. To mitigate this noise, we consider the loss in Eq. (7) with the term−( ∑Nway j=1 qjsj)+ su dropped from the contrastive coefficient, and denote it as “nCTT ×”. On the other hand, we also implement a loss with “ITF ×, nCTT ×”, which is actually Eq. (9). We adopt the same experimental setting as Section B.6.
In Figure 8, we show the testing profiles of the original reformulated loss (i.e., the curve in red, labeled as “ITF ✓, nCTT ✓”), dropping the interference term (i.e., the curve in orange, labeled as “ITF ×, nCTT ✓”), dropping the noisy part of the contrastive term (i.e., the curve in green, labeled as “ITF ✓, nCTT ×”) or dropping both (i.e., the curve in blue, labeled as “ITF ×, nCTT ×”). We can see that either dropping the interference term or dropping dropping the noisy part of contrastive coefficients yield profound benefit.
To better understand how noisy is the noisy contrastive term, i.e., how many times the sign of the contrastive coefficient is negative when the query and support data are in different classes, we explicitly record the ratio of the contrastive term being positive or negative. We adopt the same experimental setting as Section B.6.
The result is shown in Figure 9. When the zeroing trick is applied, the ratio of contrastive term being negative (shown as the red curve on the right subplot) is 0.2, which is 1Nway where Nway = 5 in our setting. On the other hand, when the zeroing trick is not applied, the ratio of contrastive term being negative (shown as the orange color on the right subplot) is larger than 0.2. This additional experiment necessitates the application of the zeroing trick.

heading: C A GENERALIZATION OF OUR ANALYSIS
text: In this section, we derive a more general case of the encoder update in the outer loop. We consider drawing Nbatch tasks from the task distribution D and having Nstep update steps in the inner loop while keeping the EFIL assumption.
To derive a more general case, we use wki,n to denote the kth column of wi,n, where wi,n is updated from w0 using support data Sn for i inner-loop steps. For simplicity, the kth channel softmax predictive output exp(ϕ(s) ⊤wk i,n)∑Nway
j=1 exp(ϕ(s) ⊤wji,n)
of sample s (using wi−1,n) is denoted as si,nk .
Inner loop update for the linear layer We yield the inner loop update for the final linear layer in Eq. (21) and Eq. (22).
wk i,n = wk i−1,n − η ∂L{φ,wi−1,n},Sn
∂wki−1,n = wk i−1,n + η E (s,t)∼Sn (1k=t − si−1,nk )ϕ(s) (21)
wk Nstep,n = wk 0 − η Nstep∑ i=1 E (s,t)∼Sn (1k=t − si−1,nk )ϕ(s) (22)
Outer loop update for the linear layer We derive the outer loop update for the linear layer in SOMAML, with denoting I = {1, 2, ..., Nway}:
w′k 0 = wk 0 − ρ Nbatch∑ n=1 ∂L{φ,wkNstep,n},Qn ∂wk0
= wk 0 − ρ Nbatch∑ n=1 ∑ p0=k,p1∈I,...,pNway∈I [( Nstep−1∏ i=0 ∂wpi+1 i+1,n ∂wpi i,n ) ∂L{φ,wNstep,n},Qn ∂wpNstep Nstep,n ]
(23)
When it comes to FOMAML, we have
w′k 0 = wk 0 − ρ Nbatch∑ n=1 ∂L{φ,wkNstep,n},Qn ∂wkNstep,n = w0k + ρ Nbatch∑ n=1 E (q,u)∼Qn (1k=u − qNstep,nk )ϕ(q) (24)
Outer loop update for the encoder We derive the outer loop update of the encoder under FOMAML as below. We consider the back-propagated error of the feature of one query data (q, u) ∼ Qn. Note that the third equality below holds by leveraging Eq. (21).
− ∂L{φ,wNstep,n},Qn
∂ϕ(q) = wu Nstep,n − Nway∑ i=1 (qNstep,ni wi Nstep,n)
= Nway∑ i=1 (1i=u − qNstep,ni )wi Nstep,n
= Nway∑ i=1 (1i=u − qNstep,ni )[w 0 i + η Nstep∑ p=1 E (s,t)∼Sn (1i=t − sp−1,ni )ϕ(s)]
= Nway∑ i=1 (1i=u − qNstep,ni )w 0 i
+ η Nway∑ i=1 (1i=u − qNstep,ni ) Nstep∑ p=1 E (s,t)∼Sn (1i=u − sp−1,ni )ϕ(s)
= Nway∑ i=1 (1i=u − qNstep,ni )w 0 i
+ η E (s,t)∼Sn Nstep∑ p=1 [( Nway∑ j=1 qNstep,nj s p−1,n j )− s p−1,n u − q Nstep,n t + 1t=u]ϕ(s)
(25)
Reformulating the Outer Loop Loss for the Encoder as Noisy SCL Loss. From Eq. (25), we can derive the generalized loss (of one query sample (q, u) ∼ Qn) that the encoder uses under FOMAML scheme.
L{φ,wNstep,n},q = Nway∑ i=1 (1i=u − qNstep,ni )w 0 i ⊤
stop gradient
ϕ(q)
+ η E (s,t)∼Sn Nstep∑ p=1 [( Nway∑ j=1 qNstep,nj s p−1,n j )− s p−1,n u − q Nstep,n t + 1t=u]ϕ(s) ⊤
stop gradient
ϕ(q)
(26)

heading: D EXPERIMENTS ON MINIIMAGENET DATASET

heading: D.1 EXPERIMENTAL DETAILS IN MINIIMAGENET DATASET
text: The model architecture contains four basic blocks and one fully connected linear layer, where each block comprises a convolution layer with a kernel size of 3 × 3 and filter size of 64, batch normalization, ReLU nonlineartity and 2 × 2 max-poling. The models are trained with the softmax cross entropy loss function using the Adam optimizer with an outer loop learning rate of 0.001 (Antoniou et al., 2019). The inner loop step size η is set to 0.01. The models are trained for 30000 iterations (Raghu et al., 2020). The results are averaged over four random seeds, and we use the shaded region to indicate the standard deviation. Each experiment is run on either a single NVIDIA 1080-Ti or V100 GPU. The detailed implementation is based on Long (2018) (MIT License).

heading: D.2 THE EXPERIMENTAL RESULT OF SOMAML
text: The results with SOMAML are shown in Figure 10. Note that as it is possible that longer training can eventually overcome the noise factor and reach similar performance as the zeroing trick, the benefit of the zeroing trick is best seen at the observed faster convergence results when compared to vanilla MAML.

heading: D.3 COSINE SIMILARITY ANALYSIS ON SEMANTICALLY SIMILAR CLASSES VERIFIES THE IMPLICIT CONTRASTIVENESS IN MAML
text: In Figure 2, we randomly sample five classes of images under each random seed. Given the rich diversity of the classes in Mini-ImageNet, we can consider that the five selected classes as semantically dissimilar or independent for each random seed. Here, we also provide the experimental outcomes using a dataset composed of five semantically similar classes selected from the Mini-ImageNet dataset: French bulldog, Saluki, Walker hound, African hunting dog, and Golden retriever. Likewise to the original setting, we train the model using FOMAML and average the results over ten random seeds. As shown in Figure 11, the result is consistent with Figure 2. In conclusion, we show
that the supervised contrastiveness is manifested with the application of the zeroing trick even if a semantically similar dataset is considered.

heading: D.4 EXPERIMENTAL RESULTS ON LARGER NUMBER OF SHOTS
text: To empirically verify if our theoretical derivation generalizes to the setting where the number of shots is large, we conduct experiment of a 5-way 25-shot classification task using FOMAML with four random seeds where we adopt mini-ImageNet as the example dataset. As shown in Figure 12, we observe that models trained with the zeroing trick again yield the best performance, consistent with our theoretical work that MAML with the zeroing trick is SCL without noises and interference.

heading: D.5 THE ZEROING TRICK MITIGATES THE CHANNEL MEMORIZATION PROBLEM
text: The channel memorization problem (Jamal & Qi, 2019; Rajendran et al., 2020) is a known issue occurring in a non-mutually-exclusive task setting, e.g., the task-specific class-to-label is not randomly assigned, and thus the label can be inferred from the query data alone (Yin et al., 2020). Consider a 5-way K-shot experiment where the number of training classes is 5 × L. Now we construct tasks by assigning the label t to a class sampled from class tL to (t+1)L. It is conceivable that the model will learn to directly map the query data to the label without using the information of the support data and thus fails to generalize to unseen tasks. This phenomenon can be explained from the perspective that the tth column of the final linear layer already accumulates the query features from tLth to
(t + 1)Lth classes. Zeroing the final linear layer implicitly forces the model to use the imprinted information from the support features for inferring the label and thus mitigates this problem. We use the miniImagenet dataset and consider the case of L = 12. As shown in Figure 13, the zeroing trick prevents the model from the channel memorization problem whereas zero-initialization of the linear layer only works out at the beginning. Besides, the performance of models trained with the zeroing trick under this non-mutually-exclusive task setting equals the ones under the conventional few-shot setting as shown in Figure 5. As the zeroing trick clears out the final linear layer and equalizes the value of logits, our result essentially accords with Jamal & Qi (2019) that proposes a regularizer to maximize the entropy of prediction of the meta-initialized model.

heading: E EXPERIMENTS ON OMNIGLOT DATASET
text: Omniglot is a hand-written character dataset containing 1623 character classes, each with 20 drawn samples from different people (Lake et al., 2015). The dataset set is splitted into training (1028 classes), validation (172 classes) and testing (423 classes) sets (Vinyals et al., 2016). Since we follow Finn et al. (2017) for setting hyperparamters, we do not use the the validation data. The character images are resized to 28× 28. For all our experiments, we adopt two experimental settings: 5-way 1- shot and 5-way 5-shot where the batch size Nbatch is 32 and Nquery is 15 for both cases (Finn et al., 2017). The inner loop learning rate η is 0.4. The models are trained for 30000 or 20000 iterations using FOMAML or SOMAML, respectively. For channel memorization problem, the models are trained for 10000 iterations using FOMAML. The few-shot classification accuracy is calculated by averaging the results over 500 tasks in the test stage. The model architecture follows the architecture used to train on miniImagenet, but we substitute the convolution with max-pooling with strided convolution operation as in Finn et al. (2017). The loss function, optimizer, and outer loop learning rate are the same as the ones used in the experiments on miniImagenet. Each experiment is run on either a single NVIDIA 1080-Ti or V100 GPU. The results are averaged over four random seeds, and the standard deviation is illustrated with the shaded region. The models are trained using FOMAML unless stated otherwise. The detailed implementation is based on Deleu (2020) (MIT License).
We revisit the application of the zeroing trick at the testing stage on Omniglot in Figure 14 and observe the increasing testing accuracy, in which such results are compatible with the ones on miniImagenet (cf. Figure 3 in the main manuscript). In the following experiments, we evaluate the testing performance only after applying the zeroing trick.
In Figure 15, the distinction between the performance of models trained with the zeroing trick and zero-initialized models is less prominent as compared to miniImagenet (cf. Figure 5 in the main manuscript) in 5-way 1-shot setting. This result can be explained by the larger inner loop learning rate η = 0.4 used in Omniglot. We also show the testing performance of models trained using SOMAML in Figure 16, where there is little distinction in performance (in comparison to the results on miniImagenet, cf. Figure 10 in the main manuscript) between the models trained with the zeroing trick and the ones trained with random initialization.
For channel memorization task, we construct non-mutually-exclusive training tasks by assigning the label t (where 1 ≤ t ≤ 5 in 5-way setting) to a class sampled from class tL to (t + 1)L where L is 205 on Omniglot. The class-to-channel assignment is not applied to the testing tasks. The result is shown in Figure 17. For a detailed discussion, please refer to Section D.5 in the main manuscript.

Review summaries:
This work shows that, in the setting of few-shot classification, MAML is a (noisy) contrastive learner. Complementary theoretical and experimental results are provided. The theoretical results also lead to a (to my knowledge) new proposal, namely to zero out the linear layer after each MAML outer loop update. In their experiments, the authors show that making this small change to the MAML algorithm can lead to meaningful improvements in performance.

The paper analyzes MAML algorithms. Assuming in the inner loop the encoder is fixed and only last linear layer is updated, they analyze the gradient update and loss terms in the inner loop and outer loops (sec 2.3). Through this effort, the authors claim that there are noisy supervised contrastive term in the outer loop loss (eqn (7) and (8)). They further claim that there are additional interference terms which may degrade the performance of MAML at the beginning of training when the linear layer weights are largely random. To overcome this, they propose a simple zeroing trick by zeroing the initial linear layer weights after each outer loop update, essentially removing the interference terms (eqn (9) and (10)). They conduct experiments to support the contrastiveness in MAML and performance improvement using zeroing trick. 

In this paper, a new view of MAML under few-shot learning is proposed. The main result is that under the assumption that the inner loop updates are only applied on the top linear layer, MAML actually performs supervised contrastive learning (SCL). SCL shows that MAML learns the feature transformation that makes the intra-class feature distances small, meanwhile the inter-class feature distances large. The zeroing trick is proposed based on this result, showing performance gain in the experiments.