MAML is a Noisy Contrastive Learner in Classification
Authors: Chia Hsiang Kao
Wei-Chen Chiu
Pin-Yu Chen
Venue: ICLR 2022 Poster (2022)
Keywords: Meta learning
contrastive learning
few shot learning

Abstract: Model-agnostic meta-learning (MAML) is one of the most popular and widely adopted meta-learning algorithms, achieving remarkable success in various learning problems. Yet, with the unique design of nested inner-loop and outer-loop updates, which govern the task-specific and meta-model-centric learning, respectively, the underlying learning objective of MAML remains implicit, impeding a more straightforward understanding of it. In this paper, we provide a new perspective of the working mechanism of MAML. We discover that MAML is analogous to a meta-learner using a supervised contrastive objective in classification. The query features are pulled towards the support features of the same class and against those of different classes. Such contrastiveness is experimentally verified via an analysis based on the cosine similarity. Moreover, we reveal that vanilla MAML has an undesirable interference term originating from the random initialization and the cross-task interaction. We thus propose a simple but effective technique, the zeroing trick, to alleviate the interference. Extensive experiments are conducted on both mini-ImageNet and Omniglot datasets to validate the consistent improvement brought by our proposed method.

Published as a conference paper at ICLR 2022
MAML IS A NOISY CONTRASTIVE LEARNER IN CLASSIFICATION
Kaot
Wei-Chen Chiu
Pu Chen
National Yang Ming Chiao Tung University, Taiwan 'IBM Research
c m0004 edu.tw tw walonêcs auctueddu.
pin-yu. chene ibm com
ABSTRACT
Model-agnostic meta-learning (MAML) is one of the most popular and widely adopted meta-learning algorithms, achieving remarkable success in various learn- ing problems. Yet, with the unique design of nested inner-loop and outer-loop updates, which govern the task-specific and meta-model-centric learning, respec- tively, the underlying learning objective of MAML remains implicit, impeding a more straightforward understanding of it. In this paper, we provide new perspec- tive of the working mechanism of MAML. We discover that MAML is analogous meta-learner using supervised contrastive objective in classification. The query features are pulled towards the support features of the same class and against those of different classes. Such contrastiveness is experimentally verified via analysis based on the cosine similarity. Moreover, we reveal that vanilla MAML has an undesirable interference term originating from the random initialization and the cross-task interaction. We thus propose simple but effective technique, the zeroing trick, to alleviate the interference. Extensive experiments are con- ducted on both mini-ImageNet and Omniglot datasets to validate the consistent improvement brought by our proposed method.
INTRODUCTION
Humans can learn from very few samples. They can readily establish their cognition and understand- ing of novel tasks, environments, or domains even with very limited experience in the corresponding circumstances. Meta-learning, a subfield of machine learning. aims at equipping machines with such
capacity to accommodate new scenarios effectively (Vilalta Drissi] 2002 Grant et al. 2018). Ma- chines learn to extract task-agnostic information so that their performance on unseen tasks can be improved (Hospedales et al. 2020).
One highly influential meta-learning algorithm is Model Agnostic Meta-Learning (MAML) Finn ctt al. 2017). which has inspired numerous follow-up extensions (Nichol et al.) 2018 Rajeswaran al. 2019 Liu al. 2019| Finn et al., 2019; Jamal & Qil 2019 Javed & White 2019). MAML estimates a set of model parameters such that an adaptation of the model to a new task only requires some updates to those parameters. We take the few-shot classification task the algorithmic procedure of MAMI. A few-shot classification problem refers to classifying sam- ples from some classes (i.e. query data) after seeing few examples per class (i.e. support data). In meta-lcarning scenario, we consider distribution of tasks, where each task is few-shot clas- sification problem and different tasks have different target classes. MAML aims to meta-train the base-model based on training tasks (i.e., the meta-training dataset) and evaluate the performance of the base-model on the testing tasks sampled from held-out unseen dataset (i.e. the meta-testing dataset). In meta-training, MAML follows bi-level optimization scheme composed of the inner loop and the outer loop, as shown in Appendix A (please refer to Section 2 for detailed defini- tion). In the inner loop (also known as fas! adapiation), the base-model 0 is updated to O' using the support set. In the outer loop. loss is evaluated on 6' using the query set, and its gradient is computed with respect to 0 to update the base-model. Since the outer loop requires computing the gradient of gradient (as the update in the inner loop is included in the entire computation graph), called second-order MAML (SOMAML). To prevent computing the Hessian matrix, Finn et al.
example to review
"ooee available /ttttpt:/ggiieeeee com/ http://nnoooee/......... cooossycoottiio rasi wetaaveeeeeaeeee 1earner

---

Published as a conference paper at ICLR 2022
2017) propose first-order MAML (FOMAML) that uses the gradient computed with respect to the amer-loop-updated parameters 8' to update the base-model.
The widely accepted intuition behind MAML is that the models are encouraged to learn general- purpose representations which are broadly applicable not only to the seen tasks but also to novel tasks (Finn et al., 2017: Raghu et al., 2020; Goldblum al. 2020). Raghu al. (2020) confirm this perspective by showing that during fast adaptation, the majority of changes being made are in the final linear layers. In contrast, the convolution layers (as the feature encoder) remain almost static. This implies that the models trained with MAML learn a good feature representation and that they only have to change the linear mapping from features to outputs during the fast adaptation. Similar ideas of freezing feature extractors during the inner loop have also been explored (Lee et al.) 2019 Bertinetto et al. 2019 Liu al. 2020). and have been held as an assumption in theoretical works (Du et al. 2021 Tripuraneni et al. 2020 Chua et al., 202T).
While this intuition sounds satisfactory, we step further and ask the following fundamental questions: (1) In what sense does MAML guide any model to learn general-purpose representations? (2) How do the inner and outer loops in the training mechanism of MAML collaboratively prompt to achieve so? (3) What is the role of support and query data, and how do they interact with each other? In this paper, we answer these questions and give new insights on the working mechanism of MAML, which turns out to be closely connected to supervised contrastive learning (
Here, we provide a sketch of our analysis in Figure II We consider a setting of (a) a 5-way 1-shot paradigm of few-shot learning, (b) the mean square error (MSE) between the one-hot encoding of groundtruth label and the outputs as the objective function, and (c) MAML with a single inner-loop update. At the beginning of the inner loop, we set the linear layer w zero. Then, the inner loop update of w is equivalent to adding the support features to w In the outer loop, the output of a query sample qı is actually the inner product between the query feature ((91) and all support features (the learning rate is omitted for now). As the groundtruth is an one-hot vector, the encoder is trained to either minimize the inner product between the query features and the support features (when they are from different classes, as shown in the green box), or to pull the inner product between the query features and the support features to 1 (when they have the same label, as shown in the red box). Therefore, the inner loop and the outer loop together manifest SCL objective. Particularly, as the vanilla implementation of MAML uses non-zero (random) initialization for the linear layer, we will show such initialization leads to noisy SCL objective which would impede the training.
In this paper, we firstly review a formal definition of SCL. present a more general case of MAML with cross entropy loss in classification, and show the underlying learning protocol of vanilla MAML as an interfered SCL in Section 2 MAML and propose to mitigate the interference with our simple but effective technique of the zero- initialization and zeroing trick (ef. Section 3) In summary, our main contributions are three-fold:
We then experimentally verify the supervised contrastiveness of
We show MAML is implicitly an SCL algorithm in classification and the noise comes from the randomly initialized linear layer and the cross-task interaction.
We verify the inherent contrastiveness of MAML based on the cosine similarity analysis.
Our experiments show that applying the zeroing trick induces a notable improvement in testing accuracy during training and that that during meta-testing, a pronounced increase in the accuracy occurs when the zeroing trick is applied.
2 WHY MAML IS IMPLICITLY A NOISY SUPERVISED CONTRASTIVE ALGORITH??
2.1 PRELIMINARY: SUPERVISED CONTRASTIVE LEARNING
In this work, we aim to bridge MAML and supervised contrastive learning (SCL) and attribute the success of MAML to SCL's capacity in learning good representations. Thus, we would like to introduce SCL briefly.
We use the term supervised contrastiveness to refer to the setting of using ground truth label information to differentiate positive samples and negative samples (Khosla et al. 2020). This setting is different from sessspprrisedd/eef-uuerrvsssss contrastive learning.
2

---

Published as a conference paper at ICLR 2022
way 1-shot using MAMI with one inner-lopp update under MSF loss.
Output
Layer
Set linear layer zero.
Forward single support data 4.).
Image
MSE loss
Inner loop
Dulpur
Gradlent of
For single updated by adding support feature:
Overall, k nnd added support features of class & becomes w
(()(())))))
gradient
I. M nE (-(()))))
4. the outter loop, the loss computed query data (q)
MSB
pel 10-00--1 -41
output
As w contains features, the outer essentially supervised contrastive
Outer loop
Negative sample loss
Minimize the inner product query features and support features different bi, (y).
2
Guide the encoder leam disentangled features
Force the inner product of query features and support features the
sample loss
03
$ (5))
Figure 1: A step-by-step illustration showing the SCL objective underlying MAML. Assume the linear layer w to be zero, we find that, during the inner loop, the jth column of w is added with the support features of class In other words, the support features memorized by the linear layer during the inner loop. In the outer loop, the output of query sample is the inner product of ((q1) and w'. which is essentially the inner product of the query features and all the support features. The outer loop loss aims to minimize the MSE between the inner products and the one-hot label. Thus, MAMI. displays the characteristic of supervised contrastiveness. Besides, the support data acts as positive samples when the labels of support and query data match, and vice versa.
Supervised contrastive learning, proposed by Khosla et al. (2020), is a generalization of several metric learning algorithms, such triplet loss and N-pair loss (Schroff et al.) 2015 Sohn 2016). and has shown the best performance classification compared to SimCLR and CrossEntropy. In Khosla al. (2020), SCL described "contrasts the of all samples from the same class as positives against the negatives from the remainder of the batch" and "embeddings from the same class are pulled closer together than embeddings from different classes." " For sample s, the label information is leveraged to indicate positive samples (i.e., samples having the same label as sam- ple s) and negative samples (i.e., samples having different labels sample s). The loss of SCL is designed to increase the similarity (or decrease the metric distance) of embeddings of positive samples and to reduce the similarity (or increase the metric distance) of embeddings of negative samples (Khosla et al. 2020). In essence, SCL combines supervised learning and contrastive learn- ing and differs from supervised learning in that the loss contains a measurement of the similarity (or distance) between the embedding of sample and embeddings of its positive/negative sample pairs.
Now we give a formal definition of SCL. For a set of samples drawn from a n-class dataset. Let N//b be the index of arbitrary sample. Let A(i) I fi), P(t) be the set of
i E I = {1, N} indices of all positive samples of sample i, and N A(i) P(i) be the set of indices of all negative samples of sample Let z; indicates the embedding of sample

---

Published as a conference paper at ICLR 2022
Definition 1 Let Maim be measurement of similarity (e.g.. inner product, cosine similarity). Training algorithms that adopt loss of the following form belong to SCL:
LSCL -E> M. (2t, 2p) +
>>> ct C (Zi, Zn) t
(1)
PEP())
DEN()
where Cp. < 0 and c for all p and i; and is constant independent of samples.
We further define that a training algorithm that follows Eq. T. but with either (a) Cn < 0 for some
n, or (b) is constant dependent of samples, belongs noisy SCL.
2.2 PROBLEM SETUP
We provide the detailed derivation to show that MAML is implicitly a noisy SCL, where we adopt the few-shot classification as the example application. In this section, we focus on the meta-training period. Consider drawing a batch of tasks (Ti,..., ,TTT..e,T TNmeon from a meta-training task distribution fm)] M x Nohot
D. Each task Tn contains support set Sn and query set Qn, where S,, {55m tm)
On = {(ym, Um ) N x Nauery 8m, 9m E R Ne,in are data samples, and lm; Um E {I,.... N Nwayy are labels. We denote N Way the number of classes in each task, and {Nahhtt Nquery) respectively the number of support and query samples per class. The architecture of our base-model comprises of convolutional encoder o R N.. + R (parameterized by (p), fully connected linear head w E R. x and a Softmax output layer, where N, is the dimension of the feature space. We denote the 1.th column of w as Wk. Note that the base-model parameters 0 consist of yp and w.
As shown in Appendix A both FOMAML and SOMAML adopt a training strategy comprising the inner loop and the outer Toop. At the beginning of meta-training iteration, we sample Nistch tasks. For each task Tn, we perform inner loop updates using the inner loop loss (c.f. Eq. (2) evaluated on the support data, and then evaluate the outer loop loss (c.f. Eq. (3) on the updated base-model using the query data. In the 1t step of the inner loop, the parameters {spt-1 w.-l) are updated to (s0", w²) using the multi-class cross entropy loss evaluated on the support dataset S, as
Nuay
exp(x'(s) (w) Σ exp(0) wkr)
(ip Bet-wwl Y,88
Σ 1 log
(2)
(s,1)=
After Nstep inner loop updates, we compute the outer loop loss using the query data Qn:
exploMster W.. Notor)
L Nater, w M.
E
[- log Exers exp(áMMter wk Noter)
(3)
(q.n)-Q.
Then, we sum up the outer loop losses of all tasks, and perform gradient descent to update the base-model's 's initial parameters fsp w.).
To show the supervised contrastiveness. entailed in MAML, we adopt an assumption that the Encoder 6 is Frozen during the Inner Loop (the EFIL assumption) and we discuss the validity of the as- sumption Section 2.6] Without loss of generality, we consider training models with MAMI. with Nbatch = 1 and N stepp = 1, and we discuss the generalized version in Section 2.6 For simplicity,
the ht element of model output exp(d(s) wx.)
j=1 exp()(e) o) (respectively
of sample Nuay explo(y) )
exp(o(y) Wk
(respectively q) denoted as Sş (respectively 9g).
2.3 INNER LOOP AND OUTER LOOP UPDATE OF LINEAR LAYER AND ENCODER
In this section, we primarily focus on the update of parameters in the case of FOMAML. The full derivation and discussion of SOMAML are provided Appendix B
Inner loop update of the linear layer. In the inner loop, the linear layer w is updated to w/ with learning rate 77 shown in Eq. (4) in both FOMAML and SOMAML. In contrast the example in Figure []] the columns of the linear layer are added with the weighted sum of the features extracted from support samples (i.e., support features), Compared to wx", wk is pushed towards the support features of the same class (i.e.. class k) with strength of 1 Sk. while being pulled away from the support features of different classes with strength of Sk-
OL(.. Own
Wk wk.
Wk
+ E (1 Su)o(s) (6.t)~S
(4)
4

---

Published as a conference paper at ICLR 2022
Outer loop update of the linear layer. In the outer loop, w" is updated using the query data with learning p. For FOMAML, the final linear layer is updated as follows.
= Wk wl Wk Owk
+
E (1k=u - -4))(g) (a.))
(5)
Note that the computation of 9₁ requires the inner-loop updated w Generally speaking. Eq. ( resembles Eq. (4). It is obvious that, in the outer loop, the query features are added weightedly to the linear layer, and the strength of change relates to the output value. other words, after the outer loop update, the linear layer memorizes the query features of current tasks. This cause cross- task interference because in the next inner loop there would additional inner products between the support features of the next tasks and the query features of the current tasks.
Outer loop update of the encoder. Using the chain rule, the gradient of the outer loop loss with spect to yp (i.e., the parameters of the encoder) is given by 04 (u.=1 Eq AL (w. Be(a), + Bō(g) 3e
0e
aL ) de(s) where the second term can be neglected when FOMAML is considered.
E(s,)>SS de(e) 0ip
Below, we take deeper look at the backpropagated error of one query data (g, u) Q. The full derivation is provided Appendix B.2
aT b. 0②(g)
Nway
Nuus
(g 1j= +1
[
9,85) + + 9: 1==)((())
(6)
(a.t)~ 1=1
2.4 MAML IS A NOISY CONTRASTIVE LEARNER
Reformulating the outer loop loss for the encoder as noisy SCL loss. We can observe from Eq. 66 that the actual loss for the encoder (evaluated on a single query data (g, u) ~ Q) is as the following.
Nway
Nuay
[sg 3:1
(gp
1== Ww
((4) +
9,5y + Su + [d)) 2(4)
(5.0)~8
gradiemt
gradient
For SOMAMI, the range of "stop gradient" in the second term is different:
Nwaay
Nuay
[o.w)), =
( 1 DT ol)) +
9,8y + Su + 9: Lumul ((8)
((s)T((g) ((g))
(8.6)
(8)
stop gradiess
With these two reformulations, observe the essential difference between FOMAML and SO- MAML the range of stop gradient. We provide detailed discussion and instinctive illustration in Appendix [B.5 on how this explains the phenomenon that SOMAML often leads faster conver- gence. To better deliberate the effect of each term in the reformulated outer loop loss, we define the interference term, the second term as noisy contrastive term, and Σ 1 9,5y + Su + 4, 1;=u as contrastive coefficients.
first term in Eq. 7 or Eq. Naway
the coefficients
Understanding the interference term. In the case of j = u, the outer loop loss forces the model to Φ(g). This can be problematic because (a) at the beginning of training, w
minimize (9, 1)w,"' ( assigned with random values and (b) w is added with query features of previous tasks as shown in Eq. (5) Consequently, Φ(g) is pushed to a direction composed of previous query features or to a random direction, introducing an unnecessary cross-task interference or an initialization interference that slows down the training of the encoder. Noting that the cross-task interference also occurs at the testing period, since, at testing stage, w is already added with query features of training tasks, which can be an interference to testing tasks.
Understanding the noisy contrastive term. When the query and support data have the same label (i.e., u = t), e.g., class 1, the contrastive coefficients becomes - NNuy 4,8, S1 + S1 + q - 1, which - Σ Nway 8, - (1 91 - 51) < 0. This indicates the encoder would updated to maximize the inner product between ((q) and the support features of the same class. However, when the query and support data are in different classes, the sign of the contrastive coefficient can
5

---

Published as a conference paper at ICLR 2022
sometimes be negative. The outer loop loss thus cannot well contrast the query features against the support features of different classes, making this loss term not an ordinary SCL loss.
To better illustrate the influence of the interference term and the noisy contrastive term, provide an ablation experiment in Appendix B.7 Theorem below formally connects MAML to SCL.
Theorem With the EFIL assumption, FOMAML a noisy SCL algorithm. With assumptions of (a) EFIL and (b) single inner-loop update, SOMAML is noisy SCL algorithm.
Proof: For FOMAML, both Eq. (one inner loop update step) and Eq. (26) (multiple inner loop update steps) follows Definition For SOMAML, Eq. (8) follows Definition 1.
Introduction of the zeroing trick makes Eq. 7 and Eq. (8) SCL losses, To tackle the interference
term and make the contrastive coefficients more accurate, we introduce the zeroing trick: setting the w to be zero after each outer loop update, as shown in Appendix A original outer loop loss (of FOMAML) becomes
With the zeroing trick, the
L (ggwh) !J
(q. 1I=))((s) p(g)
(s.t)= stop gradicut
(9)
For SOMAML, the original outer loop loss becomes
[ =
(q. 1== ((s) (b() (s.t)=St
E
(a))
(10)
gradions
The zeroing trick brings two nontrivial effects: (a) eliminating the interference term in both Eq. 7 and Eq. (8): (b) making the contrastive coefficients follow SCL. For (b), since all the predictive
values of support data become the same, i.c., S& = Neas' the contrastive coefficient becomes 11=u. which is negative when the support and query data have the same label, and positive otherwise.
With the zeroing trick, the contrastive coefficient follows the SCL loss, as summarized below.
Corollary With mild assumptions (a) EFIL, (b) single inner-loop update and (c) training with the zeroing trick (i.e., the linear layer is zeroed at the end of each outer loop), both FOMAML. and SOMAML are SCL algorithms.
Proof: Both Eq. 9 and Eq. 10 follow Definition 1.
The introduction of the zeroing trick makes the relationship between MAML and SCL more straight- forward. Generally speaking, by connecting MAML and SCL, we can better understand other
MAML-based meta-leaming studies.
2.5 RESPONSES TO QUESTIONS IN SECTION
In what sense does MAML guide any model to learn general-purpose representations? Under the EFIL assumption, MAML is a noisy SCL algorithm a classification paradigm. The effective- ness of MAML in enabling models to learn general-purposs representations can be attributed to the SCL characteristics of MAML.
How do the inner and outer loops in the training mechanism of MAML collaboratively prompt to achieve so? MAML adopts the inner and outer loops to perform noisy SCL sequentially. In the inner loop, the features of support data memorized by via inner-loop update. In the outer loop, the softmax output of the query data thus contains the inner products between the support features and the query feature.
What is the role of support and query data, and how do they interact with each other? We show that the original loss in MAML can be reformulated as loss term containing the inner products of the embedding of the support and query data. In FOMAML, the support features act as the reference, while the query features are updated to move towards the support features of the same class and against those of the different classes.
2.6 GENERALIZATION OF OUR ANALYSIS
In Appendix C we provide the analysis where Nhatch 2 1 and Nstep 2 1. For the EFIL assumption, it can hardly be dropped because the behavior of the updated encoder is intractable. Besides, Raghu

---

Published as a conference paper at ICLR 2022
et al. (2020) show that the representations of intermediate layers do not change notably during the inner loop of MAML, and thus it is understood that the main function of the inner loop is to change the final linear layer. Furthermore, the EFIL assumption is empirically reasonable, since previous works (Raghu et al., 2020) Lee et al. 2019 Bertinetto et al. 2019: Liu et al. 2020) yield comparable performance while leaving the encoder untouched during the inner loop.
With our analysis, one may notice that MAML is approximately a metric- few-shot learning al- gorithm. From a high-level perspective, under the EFIL assumption, second-order MAML similar to metric-based few-shot learning algorithms, such as MatchingNet (Vinyals et al. 2016), Proto- typical network (Snell et al., 2017), and Relation network (Sung et 20188). The main difference lies in the metric and the way prototypes are constructed. Our work follows the setting adopted by MAMI., such as using negative LogSoftmax as objective function, but we can effortlessly generalize our analysis to a MSE loss as had been shown in Figure [ As result, our work points out new research direction in improving MAML by changing the objective functions in the inner and the outer loops, e.g., using MSE for the inner loop but negative LogSoftmax for the outer loop. Besides, in MAML, we often obtain the logits by multiplying the features by the linear weight w. Our work implies future direction as to alternatively substitute this inner product operation with other metrics or other similarity measurements such as cosine similarity or negative Euclidean distance.
3 EXPERIMENTAL RESULTS
In this section, we provide empirical evidence of the supervised contrastiveness of MAML and show that pero-initialization of w", reduction in the initial norm of w or the application of zeroing trick can speed up the learning profile. This is applicable to both SOMAML and FOMAML.
3.1 SETUP
We conduct our experiments on the mini- Imaniiit dataset (Vinyals et al. 2016 Ravi & Larochelle 2017) and the Omniglot dataset (Lake et al. 2015). For the results on the Omniglos dataset, please refer to Appendix [E] For the mini-ImageNNet, it contains 84 x 84 RGB images of 100 classes from the ImageNet dataset with 600 samples per class. We split the dataset into 64, 16 and 20 classes for training, validation, and testing as proposed in (Ravi & Larochelle| 2017). We do not perform hyperparameter search and thus are not using the validation data. For all our experiments of applying MAML into few-shot classification problem, where we adopt two experimental settings: 5-way 1- shot and 5-way 5-shot, with the batch size Nbatch being 4 and 2, respectively (Finn al. [017]) The few-shot classification accuracy is calculated by averaging the results over 400 tasks in the test phase. For model architecture, optimizer and other experimental details, please refer to Appendix D.I
3.2 COSINE SIMILARITY ANALYSIS VERIFIES THE IMPLICIT CONTRASTIVENESS IN MAML
In Section 2 we show that the encoder is updated so that the query features are pushed towards the support features of the same class and pulled away from those of different classes. Here we verify this supervised contrastiveness experimentally. Consider a relatively overfitting scenario where there are five classes of images and for each class there are 20 support images and 20 query images. We fix the support and query set (i.e. the data is not resampled every iteration) to verify the concept that the support features work as positive and negative samples. Channel shuffling is used avoid the undesirable channel memorization effect (Jamal & Qi, 2019; Rajendran et al. 2020). We train the model using FOMAML and examine how well the encoder can separate the data of different classes in the feature space by measuring the averaged cosine similarities between the features of each class. The results are averaged over 10 random seeds.
As shown in the top row of Figure 2 the model trained with MAML learns to separate the features of different classes. Moreover, the contrast between the diagonal and the off-diagonal entries of the heatmap increases
we remove the initialization interference (by zero-initializing w", shown in the middle row) and remove the cross-task interference (by applying the zeroing trick, shown in the bottom row). The result agrees with analysis that MAML implicitly contains the interference term which can impede the encoder from learning a good feature representation. For experiments on semantically similar classes of images, the result is shown in Section D.3
7

---

Published as a conference paper at ICLR 2022
outer loop update
10 outer loop updates
100 outer loop uodates
Midels with Innar layer randomly intiralined
Models with Inner liver aroointiiliod
class 5 query
cless 4 query class query cless 2 query class 1
Models trained with the
class copoprt
Support
class giory
Glep query clape claun rivery
Figure 2: The supervised contrastiveness entailed in MAML manifests when zero initialization or the zeroing trick applied. The value the heatmap is calculated by averaging the pair- wise cosine similarities between query features between query features and support features. We consider the setting of having randomly initialized linear layer (top), zero-initialized linear layer (middle), and the zeroing trick (bottom), and experiment with various numbers of outer loop up- dates. The readers are encouraged to compare the results between different rows.
3.3 ZEROING LINEAR LAYER TESTING TIME INCREASES TESTING ACCURACY
Before starting our analysis on benchmark datasets, we that the cross-task interference can also occur during meta-testing. In the meta-testing stage, the base-moder is updated in the inner loop using support data S and then the performance is evaluated using query data Q, where S and Q are drawn from a held-out, unseen meta-testing dataset. Recall that at the end of the outer loop (in meta-training stage), the query features are added weightedly to the linear layer w In other words, at the beginning of meta-testing, w is already added with the query features of previous training tasks, which can drastically influence the performance on the unseen tasks.
To validate this idea, we apply the zeroing trick at meta-testing time (which we refer to zeroing w at the beginning of the meta-testing time) and show such trick increases the testing accuracy of the model trained with FOMAML. As illustrated in Figure compared to directly entering meta-testing (i.e. the subplot at the left), additionally zeroing the linear layer at the beginning of each meta-testing time (i.e. the subplot at the right) increases the testing accuracy of the model whose linear layer is randomly initialized or zero-initialized (denoted by the red and orange curves, respectively). And the difference in testing performance sustains across the whole training session.
In the following experiments, we evaluate the testing performance only with zeroing the linear layer the beginning of the meta-testing stage. By zeroing the linear layer, the potential interference brought by the prior (of the linear layer) is ignored. Then, we can fully focus on the capacity of the encoder in learning good feature representation.
Original
w/ zeroing trick applied testing stage
0.48
0.48
Testho acoras
0.46
0.45
0.42
0.42
randomly intraline
0.38
0.38
0.36
0.35
(earding tric at traning staget
4200
10000120001-6052
Figure 3: Using the zeroing trick meta-testing stage improves performance, The left/right subplot shows the performance of models without/with w zeroed at the beginning of meta-testing time. The curves in red: w is randomly initialized. The curves in yellow: w is zeroed initial- ization. The curves in green: the models trained with training trick applied in the training stage.
8

---

Published as a conference paper at ICLR 2022
3.4 SINGLE INNER LOOP UPDATE SUFFICES WHEN USING THE ZEROING TRICK
In Eq. (4) and Eq. (21). we show that the features of the support data are added to the linear layer in the inner loop. Larger number of inner loop update steps can better offset the effect of interference brought by a non-zeroed linear layer. In other words, when the models are trained with the zeroing trick, a larger number of inner loop updates can not bring any benefit. We validate this intuition in Figure A under a 5-way I-shot setting. In the original FOMAML, the models trained with a single inner loop update step (denoted as red curve) converge slower than those trained with update step of 7 (denoted as purple curve). On the contrary, when the models are trained with the zeroing trick, models with various inner loop update steps converge at the same speed.
Original FOMAML
w/ zeroing trick applied in training slage
0.48
0.48
0.64
0.44
number stops
0.40
0.40
0.38
0.38
0.36
2000
4000 6000
8000 106001200014000
2000
4000 6000 8000 100001000040000
heration touter loog)
Iteration (outer toop!
Figure 4: With the zeroing trick, a larger number of inner loop update steps is not necessary. In original MAML, larger number of inner loop update steps is preferred as it generally yields better testing accuracy even with zeroing trick applied in the mela-testing stage (refer to the left figure). However, models trained using the zeroing trick do not show this trend (refer the right figure).
3.5 EFFECT OF INITIALIZATION AND THE ZEROING TRICK
In Eq. (7). we observe an interference derived from the historical task features or random initializa- tion. We validate our formula by examining the effects of (1) reducing the norm of w at initializa- tion and (2) applying the zeroing trick. From Figure 5 the performance is higher when the initial norm of w is lower. Compared to random initialization, reducing the norm via down-scaling w by 0.7 yields visible differences. Besides, the testing accuracy of MAML with zeroing trick (the purple curve) outperforms that of original MAML.
5-way 1-shot
5-way S-shot
0.48
3.625
Trotng 3c0/n0
0.16
0.600
0.575
0.42
sotting trear layer
0.40
rendenty rild ou (x0.5) pioo intiiized
0.38
0.175
aaker (reeinn tick nomiiingg stor!
0.36
0.450
2000
aoDo 190001260014600
12500 15000 hopp
Figure 5: Effect of initialization and the zeroing trick on testing performance. Both reducing the norm of w and zeroing w" each outer loop (i.c., the zeroing trick) increase the testing accuracy. The curves in red: models with w randomly initialized. The curves orange/green: reducing the value of w at initialization by factor of 0.7/ 0.5. The curve in blue: w is zero-initialized. The curve in blue: models trained with the zeroing trick.
4 CONCLUSION
This paper presents an extensive study to demystify how the seminal MAML algorithm guides the encoder to learn a general-purpose feature representation and how support and query data inter- act. Our analysis shows that MAML is implicitly supervised contrastive learner using the support features as positive and negative samples to direct the update of the encoder. Moreover, we un- veil an interference term hidden in MAML originated from the random initialization or cross-task interaction, which can impede the representation learning. Driven by our analysis, removing the interference term by a simple zeroing trick renders the model unbiased to seen or unseen tasks. Fur- thermore, we show constant improvements in the training and testing profiles with this zeroing trick, with experiments conducted on the mini-ImageNet and Omniglot datasets.
9

---

Published as a conference paper at ICLR 2022
REFERENCES
Antreas Antoniou, Harrison Edwards, and Amos Storkey. How to train your maml. In International Conference on Learning Representations (ICLR), 2019.
Luca Bertinetto, Joao F Henriques, Philip HS Torr, and Andrea Vedaldi. Meta-leaming with differ- entiable closed-form solvers. In International Conference on Learning Representations (ICLR), 2019.
Kurtland Chua, Lei, and Jason Lee. How fine-tuning allows for effective meta-learning. arXiv preprint arXiv:2105.02221/ 2021.
Tristan Deleu. Model-agnostic meta-learning.
bttpos:/gghhubb.om//ttttttttttt
pytorch-man maml] 2020.
Simon Du, Wei Hu, Sham M Kakade, Jason D Lee, and Qi Lei. Few-shot learning via learning the representation, provably. In International Conference Learning Representations (ICLR),
2021.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference Learning Representations (ICIRR), 2017.
Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta-learning. In International Conference on Machine Learning (ICML), 2019.
Micah Goldblum, Steven Reich, Liam Fowl, Renkun Ni, Valeriia Cherepanova, and Tom Goldstein. Unraveling meta-learning: Understanding feature representations for few-shot tasks. In Interna- tional Conference on Machine Learning (ICML), 2020.
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient- based meta-learning as hierarchical bayes. In International Conference on Learning Representa- tions (ICLR), 2018.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey, arXiv preprint urXiv:2004.05433, 2020.
Muhammad Abdullah Jamal and Guo-Jun Qi. Task agnostic meta-learning for few-shot learning. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2019.
Khurram Javed and Martha White. Meta-learning representations for continual learning. In Ad- vances Neural Information Processing Systems (NeurIPS), 2019.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In Advances in Neural Information Processing Systems (NeuriPS), 2020.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 2015.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), 2019.
Chen Liu, Chengming Xu, Yikai Wang. Li Zhang, and Yanwei Fu. An embarrassingly simple baseline to one-shot learning. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2020.
Hao Liu, Richard Socher, and Caiming Xiong. Taming maml: Efficient unbiased meta- reinforcement learning. In International Conference Machine Learning (ICML),
Liangqu Long. Myml-pytorch implementation.
https///ggLhbbb.omm/rraag....
MAML- MAML-Pytorch 2018.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv: 00...0003.99990 2018
10

---

Published as a conference paper at ICLR 2022
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning feature reuse? towards understanding the effectiveness of maml. In International Conference on Learning Rep- resentations (ICLR), 2020.
Janarthanan Rajendran, Alex Irpan, and Eric Jang. Meta-learning requires meta-augmeenattion. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Aravind Rajeswaran, Chelsea Finn, Sham Kakade, and Sergey Levine. Meta-learning with implicit gradients. In Advances Neural Information Processing Systems (NeurIPS). 2019.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In International Conference Learning Representations (ICLR), 2017.
Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. Promp: Proximal meta-policy search. In International Conference on Learning Representations (ICLR), 2019.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2015.
Jake Snell, Kevin Swersky, and Richard S Zemel. Prototypical networks for few-shot learning. Advances in Neural Information Processing Systems (NeurIPS), 2017.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
Xingyou Song, Wenbo Gao, Yuxiang Yang. Krzysztof Choromanski, Aldo Pacchiano, and Yunhao Tang. Es-maml: Simple hessian-free meta learning. In International Conference on Learning Representations (ICLR), 2020.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning compare: Relation network for few-shot learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Nilesh Tripuraneni, Chi Jin, and Michael Jordan. Provable meta-lcarning of linear representations. arXiv preprint arXiv:2002.11684 2020.
Ricardo Vilalta and Youssef Drissi. A perspective view and survey of meta-learning. Artificial Intelligence Review, 2002.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. Advances in Neural Information Processing Systems (NeurIPS), 2016.
Mingzhang Yin, George Tucker, Mingyuan Zhou, Sergey Levine, and Chelsea Finn. Meta-learning without memorization. In International Conference on Learning Representations (ICLR), 2020.

---

Published as a conference paper at ICLR 2022
APPENDIX
A ORIGINAL MAML AND MAML WITH THE ZEROING TRICK
Algorithm Second-order MAML
Require:
Task distribution D
Require: ft inner loop and outer loop learning rates
Require: Randomly initialized base-model parameters 0
while not done do
2: Sample tasks (Ti....T Nunten from D
3: for 1, 2.......
Notthh do
4: {Sp, Qn] sample from Ta
5:
On = 0
6:
for i 1, 1,2.... steep do
7:
8n + on 17 0. S0
8: end for
9: end for
Update - 0 P D NNaeot
Noateh VoLen.Q
11: end while
Algorithm 2 First-order MAMI.
Require: Task distribution D
Require:
7). Pt inner loop and outer loop learning rates
Require: Randomly initialized base-model parameters
1: while not done do
2: Sample tasks (Ti,... TNachh from D
3: for = 1.2..... Notth do
4:
{Sn, Q=) t sample from Tn
5:
On = 0
6:
for i 1,2, .., Nstep do
1.2....
7:
On + On Vo. S......
8:
end for
9: end for
10: Update - 0 - P Nantch Ve, Lon.Q.
11: end while
Algorithm Second-order MAML with the zeroing trick
Require: Task distribution D
Require: 1). p: inner loop and outer loop learning rates
Require: Randomly initialized base-model parameters 0
1: Set +- (the zeroing trick)
2: while not done do
3: Sample tasks (T)... T Noncen)) from D
4: for 2, Notch do
5: {Sn, Qn3 sample from Tx
6:
A, = 0
7:
for i 1.2..... Nstep do
8:
O, +- On Vo, 0, Sn
9:
end for
10:
end for
11:
Update 6-6 0 - p > E
Nonteh V VoLg, Qn
12: Set w +- 0 (the zeroing trick)
13:
end while
12

---

Published as a conference paper at ICLR 2022
B SUPPLEMENTARY DERIVATION
In this section, we provide the full generalization and further discussion that supplement the main paper. We consider the case of Nbutch 1 and Netep provide the outer loop update the linear layer under SOMAML Section [3.1 Next, we offer the full derivation of the outer loop update of the encoder in Section [..2 Then, we reformulate the outer loop loss for the encoder in both FOMAML and SOMAML in Section [B.3] and Section B.4
under the EFIL assumption. We
Afterward, discuss the main difference in FOMAMI and SOMAMI. in detail in Section B.5
Finally, we show the performance of the models trained using the reformulated loss in Section B.6
B. THE DERIVATION OF OUTER LOOP UPDATE FOR THE LINEAR LAYER USING SOMAML
Here, we provide the complete derivation of the outer loop update for the linear layer. Using SO- MAML with support set S and query set Q, the update of the linear layer follows
ÖT Olqow')). Wk Owk"
Nway
Owm Owk"
wk = wk
Owm
Owk'
OL (l.
Neas
Owm (o
=wk
"o
Owk"
Ow
8wm
mapth
.w + pll
(St {)))()))(s)}}
(1k=u ())()))
(a.1)~
(q.u)-Q
(11)
+ pn Σ mplk
Sm (3m)))(())(())) (4.1)~8
"
E (1 - 9m)
Wk + pl1 -
E (s))s((s)))
E (1kmn (x))(a)
Nwap
+ pn
E
(Sm
S (o())(())
E (1m=u - -m (0 (am)m
We can further simplify Eq. 11 to Eq.
with the help of the zeroing trick.
wk pll 1
E {Rd())((s))) (a.t)~S
E (1x=u 9a)((4)
(12)
(q.0)~Q
This is because the zeroing trick essentially turns the logits of all support samples to zero, and con- sequently the predicted probability (softmax) output sm becomes Neay for all channel m. Therefore, the third term in Eq. (I1) turns out to be zero (c.f. Eq. (33 ). The equality of Eq. (13) holds since the summation of the (softmax) outputs is one.
1_
Nway
E
((s)d(s)))
E (1m=u - )p())
N2 Kang
(g.u)
(13)
Nway
M
(s)
(1m=u 4m
Naag
(6.1)
(q.0)~Q
B.2 THE FULL DERIVATION THE OUTER LOOP UPDATE THE ENCODER.
As the encoder & is parameterized by s, the outer loop gradient with respect to yp is given by OL( E(qn)~Q 0l de(g) + E(,,t)~
at aL...wl). ap(s))
Dip We take a deeper look the
3olg) dG
Be(s)
8L (L of the feature of one query data (q. u) Q. based on the following
backpropagated error delg)
13

---

Published as a conference paper at ICLR 2022
form:
Naaay
Naaay
Wu Σ
(q,wg wy
Σ
11 y- - [wj
ab(g)
Neav
Nuuy
[1j=u - -q,ww3" +
1==
q;ll, E (Ij=t - 8;)((s)]
(14)
(a.1)~
Nway
Nuaa
Σ
[1j=
g,ww3
+
E (1.1)08
9,z;) - Su -9 lt=)(((s)
B.3 REFORMULATION OF THE OUTER LOOP LOSS FOR THE ENCODER AS NOISY SCL Loss.
We can derive the actual loss (evaluated single query data (q. u) Q) that the encoder uses under FOMAML scheme follows:
Naay
N Kuu
( 2.4
1j=u Ws ST
((g) -
E
9,s;) - Su
Lt= [d()) Φ(2) (15)
lt=u
(a.1)~
gradion
stop gradient
For SOMAMI, we need to additionally plug Eq.
into Eq.
Nway
Nuay
g - -y- oT
((g)
E
9,s,) -u
luwul (d() ((2) (16)
stpp gradiens
stop gradient
B.4 INTRODUCTION OF THE ZEROING TRICK MAKES EQ.
AND EQ. 8 SCL LOSSES.
Apply the zeroing trick to Eq. and Eq. ( we can derive the actual loss Eq. and Eq. (8) that the encoder follows.
Lepww!). =
(q. l==))((s)) ((g)
(a.1)-
(17)
stop gradiem
L L =1
(q. ( (b44)
(a.0)~S)
(18)
stop gradient
With these two equations, we can observe the essential difference in FOMAML and SOMAML is the range of stopping gradients. We would further discuss the implication of different ranges of gradient stopping in Appendix B.5
B.5 DISCUSSION ABOUT THE DIFFERENCE BETWEEN FOMAML AND SOMAML
Central to the mystery of MAML is the difference between FOMAML and SOMAML. Plenty of work is dedicated to approximating or estimating the second-order derivatives in the MAML algo- rithm in a more computaationll-effccennt or accurate manner (Song al.) 2020, Rothfuss al. 2019 Liu et al. 2019). With the EFIL assumption and our analysis through connecting SCL to these algo- rithms, we found that we can better understand the distinction between FOMAML and SOMAML
from novel perspective. To better understand the difference, we can compare Eq. (7) with Eq. (8) or compare Eq. (9) with Eq. (10). To avoid being distracted by the interference terms, we provide the analysis of the latter.
The main difference between Eq. and Eq. (10 is the range of gradient stopping and we will show that this difference results in significant distinction in the feature space. To begin with, by chain de(s) rule. we have 0p Eqq.)~2Q Belg) 0p + E(s,)~S Bo(s) ap As we specifically want to know how the encoded features updated given different losses,
aL de(g)
look at the terms aL and Tel BL by differentiating Eq. (9) and Eq. [00 with respect the features of query data & and support
Da(e) data 8, respectively.
14

---

Published as a conference paper at ICLR 2022
FOMAML:
0L E (q: 1c=))((s)
06(g)
(s,())
(19)
0L 0
06(s)
SOMAMI:
aL 06(4)
E (q: - 1 1==))(())
(s.t)mu5
(20)
0L
E (q. 1, 11=1)d(a)
0o(s)
(a.t)r~5
Obviously, as the second equation in Eq 19 zero, we know that FOMAML, the update of the encoder does consider the change of the support features. The encoder is updated to move the query features closer to support features of the same class and further to support features of different classes in FOMAMI. On the contrary, we can tell from the ahove equations that in SOMAMI, the encoder is updated to make support features and query features closer if both come from the same class and make support features and query features further if they come from different classes.
We illustrate the difference in Figure 6 For simplicity, we do not consider the scale of the coeffi- cients but their signs. The subplot on the left indicates that this FOMAML loss guides the encoder to be updated so that the feature of the query data moves I) towards the support feature of the same class, and 2) against the support features of the different classes. On the other hand, the SOMAML loss guides the encoder to be updated so that 1) when the support data and query data belong to the same class, their features move closer, and otherwise, their features move further. This generally explains why models trained using SOMAML generally converge faster than those trained using FOMAML.
First-order MAML
Second-order MAML (s,))
(s1)
Feature space
(Qq)
((q)
Φ (5a)
(S²)
(S2))
($22)
Feature space
Support data: 5,,52,
Query data:
Gradient from posilive sample (sarne wulor)
Encoder:
Gradient from negative sample (diff. color)
Figure 6: Illustration of the distinction of FOMAML and SOMAML. Conceptually speaking, the objective function of FOMAML aims to change the features of the query data; in contrast, that of SOMAML seeks to change the query's features and support data simultaneously. In this figure, the support data and query data features are plotted as solid and hollow circles, respectively. The different colors represent different classes. The solid and hollow arrows indicate the gradient calculated from positive and negative samples, respectively. Note that this is in the feature space, not the weight space.
B.6 EXPLICITLY COMPUTING THE REFORMULATING LOss USING EQ 7 AND EQ.
Under the EFIL assumption, we show that MAML can be reformulated as a loss taking noisy SCL form. Below, we consider setting 5-way 1-shot mini-ImageNNe few-shot classification task, under the condition of no inner loop update of the encoder. (This is the assumption that our derivation heavily depends on. It means that we now only update the encoder the outer loop.) We empirically show that explicitly computing the reformulated losses of Eq. 7 Eq. (17) and Eq. (18) yield almost
the same curves as MAML (with the EFIL assumption). Please note that the reformulated losses are used update the encoders, for the linear layer w explicitly update it using Eq. (5). Note that
15

---

Published as a conference paper at ICLR 2022
although the performance models training using FOMAML, FOMAML with the zeroing trick, and SOMAML converge to similar testing accuracy, the overall testing performance during the training process is distinct. The results are averaged over three random seeds.
0.48
Tezting accuracy
0.46
0.44
0.42
FOMAML, random-inttalized
FOMAMI, random-ntuulized (contrastive)
0.40
FOMAMI, inre-rillisoedd
0.38
FOMAML zere-iritskzed (contrastive)
0.36
SOMAML zero initalized (contrastive)
2000
4000
6000
8000 10000 12000 14000
Iteration (outerloop)
Figure Updating the encoder using the reformulated outer loop loss. We experimentally validate that the testing accuracy of models trained using MAML (with inner loop update of encoder) consists with that of models using their corresponding supervised contrastive losses, i.c., Eq. 77. Eq. 17 and Eq. (8).
B.7 THE EFFECT OF INTERFERENCE TERM AND NOISY CONTRASTIVE TERM
Reformulating the loss of MAML into a noisy SCL form enables us to further investigate the effeets brought by the interference term and the noisy contrastive term, which we presume both effects to be negative.
To investigate the effect of the interference term, simply consider the loss adopted by first- order MAML as Eq. (7) but with the interference term dropped (denoted as "nı x"). As for the noisy contrastive term, the noise comes from the fact that "when the query and support data are different classes, the sign of the contrastive coefficient can sometimes be negative", as being discussed in Section 2.4 To mitigate this noise, consider the loss in Eq.
7 with the term -EE 9585) + S, dropped from the contrastive coefficient, and denote it as "n2 x". On the other hand, we also implement loss with "n1, X, 722 x", which is actually Eq. (9. We adopt the same experimental setting as Section
show the testing profiles of the original reformulated loss (i.e., the curve in red, labeled as "Mı V, 12 v"), dropping the interference term (i.e., the curve in orange, labeled as "ni x, 122 v"),, dropping the noisy part of the contrastive term (i.e., the curve in green, labeled as "n V, n22 x"" or dropping both (i.e., the curve in blue, labeled as "nı x, n2 x"). We can see that either dropping the interference term dropping dropping the noisy part of contrastive coefficients yield profound benefit.
In Figure
Testing eccuracy
0.42
0.40
Pt. 7.0g (original
0.38
0. 7.0,
0.36
(zereing trick)
2000
4000 6000 8000 10000 12000 14000 Iteration (outerloog)
Figure 8: The effect of the interference term and the noisy contrastive term. We perform ablation study of the reformulated loss in Eq. (7) by dropping the interference term (denoted as "n ") or dropping the noisy part in the noisy contrastive term (marked as "n²2) ").
To better understand how noisy is the noisy contrastive term, i.c., how many times the sign of the contrastive coefficient is negative when the query and support data in different classes, explicitly record the ratio the contrastive term being positive or negative. We adopt the same experimental setting Section B.6
16

---

Published as a conference paper at ICLR 2022
The result is shown in Figure 9 When the zeroing trick is applied, the ratio of contrastive term being negative (shown as the red curve on the right subplot) 0.2, which Neay where Nway = 5 in our setting. On the other hand, when the zeroing trick is not applied, the ratio of contrastive term being negative (shown as the orange color on the right subplot) larger than 0.2. This additional experiment necessitates the application of the zeroing trick.
2000 1000 5266 8300 70000:76001 1000
2000 4502 6003 B03E 11600174001110 teartin (outer cop)
Beradon router laaps
Figure 9: The ratio of the contrastive term being positive negative during training. With the zeroing trick, the MAML becomes a SCL. algorithm, so its ratio of negative contrastive term 0.2. The original MAML, however, is noisy SCL algorithm, and thus its ratio negative contrastive term is larger than 0.2
17

---

Published as a conference paper at ICLR 2022
C A GENERALIZATION OF OUR ANALYSIS
In this section, we derive a more general case of the encoder update in the outer loop. We consider drawing Nortch tasks from the task distribution D and having Natep update steps in the inner loop while keeping the EFIL assumption.
To derive more general case, we Wk" 6. denote the hth column of w where wan is updated from w using support data Sn for inner-loop steps. For simplicity, the t channel exp(éla) "w" (") of sample & (using wi- is denoted as ss"
softmax predictive output Naay exp(o(s) wyjn))
i.n.
Σz-1
Inner doop update for the linear layer We yield the inner loop update for the final linear layer in Eq. 21 and Eq. 22
ØL (g.w Wk
Wk
Wk
1.n 1)
1-1.
+
(Ik=tt
S 11")))))(
(21)
Öwk" 1-11
(s,f)r-B
Natep
Wk Natepstt WW 7
E (s,f)maa
(La=t - St )L))))))))))
(22)
Outer loop update for the linear layer We derive the outer loop update for the linear layer in SOMAML, with denoting = {1,2, Nway):
Naotch 0L (p.w ".
Wk Wk
Owk
(23)
Noatoh
Natep
0Wp++1
i+1m
Σ
Σ
Wk
P
Owp," im
dw PWe
Notapi"
n=1
A. EX.....PWurs El
x=0
When it comes to FOMAML, we have
Nbatch
Nbutch
Wk
Wk
0
Own. Nateyon
Wk +pp
E (q.u)Qn
(1k=u
94
(24)
n=l
Outer loop update for the encoder We derive the outer loop update of the encoder under FOMAML as below. We consider the back-pren pack-propagated error of the feature of one query data (4, u) ~ Qn- that the third equality below holds by leveraging Eq. 21
Note
ÖL, ( Netep:
Nway
Natopsr
(9.
Notepi Natex:")
06(g)
Neav
(lizu Netep. ywy Nacepin
Nway
Nater
(1emu Natopill /wP +
(lim - ((--))))(())
(1.1)-8a
pml
Neav
(lizu Netapa
Jw
Nway
Nater
+7
(1izu Natapint
E
(lizu - 1aan-sp-ling (p-1in)((s)
(a.))~pp
Nway
(Iimn
Nolepa ww
Nater Nwux
Natep S g-l,m)
sp-1. q ((s)
Natap
(p.0
1=1
(25)
18

---

Published as a conference paper at ICLR 2022
Reformulating the Outer Loop Loss for the Encoder Noisy SCL Loss. From Eq. (25)
can derive the generalized loss (of one query sample (g, u) Qn) that the encoder uses under FOMAML scheme.
Nuay
L "ro
[1amu
No Jw
((q)
stop
Natep
Niaay
(26)
+
E
9
No Notepi 1,11,0
Sy 1p-1.n Natep.n + +eme)o(s)) x(g)
(e,
stop gradient
D EXPERIMENTS ON MINI-IMAGENET DATASET
D.I EXPERIMENTAL DETAILS IN MINI-IMAGENET DATASET
The model architecture contains four basic blocks and one fully connected linear layer, where each block comprises a convolution layer with kernel size of 3 x and filter size of 64, batch normal- ization, ReLU nonlineartity and 2 2 max-poling. The models are trained with the softmax cross entropy loss function using the Adam optimizer with an outer loop learning rate of 0,001 (Antoniou et al., 2019 The inner loop step size is set to 0.01. The models trained for 30000 itera- tions (Raghu et al. (020). The results are averaged over four random seeds, and we use the shaded region indicate the standard deviation. Each experiment is on either single NVIDIA 1080-Ti or V100 GPU. The detailed implementation is based on Long 2018) (MIT License).
D.2 THE EXPERIMENTAL RESULT OF SOMAML
The results with SOMAML are shown in Figure TO Note that as it is possible that longer training eventually overcome the noise factor and reach similar performance the zeroing trick. the benefit of the zeroing trick is best seen at the observed faster convergence results when compared to vanilla MAML.
5-way 1-shot setting
5-way Sishot setting
244
0/40
0.522
0.75
0.379
838
0.50
8559
0.500
2009 4000 0283 8900 000-10001000140
2500 0253 800 0001000020000000
2000 AXCC (291 3X0E 10000199344330
2000 4290 100
Figure 10: Both FOMAML and SOMAML benefit from the zeroing trick. We examine if re- ducing or removing the interference can increase the testing performance in models trained with FOMAML and SOMAML. The results suggest that SOMAML also suffers from the interference term. Note that the second subplots from the right shows lower testing performance of models trained with the zeroing trick as compared to the pero-initialized model. This may result from the overfitting problem. The curves in red: models trained with original MAML. The curve orange: w is zero-initialized. The curve in green: models trained with the zeroing trick.
D.3 COSINE SIMILARITY ANALYSIS ON SEMANTICALLY SIMILAR CLASSES VERIFIES THE IMPLICIT CONTRASTIVENESS MAML
In Figure 2 we randomly sample five classes of images under each random seed. Given the rich diversity of the classes in mini-ImageNet, we can consider that the five selected classes as seman- tically dissimilar or independent for each random seed. Here, we also provide the experimental outcomes using a dataset composed of five semantically similar classes selected from the mini- ImageNet dataset: French bulldog, Saluki, Walker hound, African hunting dog, and Golden retriever. Likewise to the original setting, we train the model using FOMAML and average the results over ten random seeds. As shown in Figure IT the result is consistent with Figure 2 In conclusion,
19

---

Published as a conference paper at ICLR 2022
show that the supervised contrastiveness is manifested with the application of the zeroing trick even a semantically similar dataset is considered.
outer toop updates
outer updates
100 suter kep updatess
Models with linear layer randomiy initialized
100
Models with linear layer zero initialized
toop updates
outer spdates
100 sutrr kop update
Models applied on linear layer each loop
query
clss quay
Figure 11: The supervised contrastiveness is verified even using dataset composing of semanti- cally similar classes of images. Considering a dataset composing of different species of dogs, we again observe the tendency that the supervised contrastiveness is manifested when we zero-initialize the linear weight and apply the zeroing trick.
D.4 EXPERIMENTAL RESULTS ON LARGER NUMBER OF SHOTS
To empirically verify if our theoretical derivation generalizes to the setting where the number of shots is large, we conduct experiment of 5-way 25-shot classification task using FOMAML with four random seeds where we adopt mini-ImageNet as the example dataset. As shown in Figure 12 we observe that models trained with the zeroing trick again yield the best performance, consistent with our theoretical work that MAML with the zeroing trick is SCL without noises and interference.
5-way 25-shot
0.70
Testing accuracy
0.65
0.60
0.55
setting
0.50
MAML
MAML zero initializstion
0.45
0
2000
4000
6000
8000
Iteration loop)
Figure 12: The zeroing trick works when it comes to larger number of shots. To examine if our work generalized to the scenario when number shots increases, we perform a 5-way 25-shot classification task. The result agrees with results of 5-way 1-shot and 5-way 5-shot.
D.5 THE ZEROING TRICK MITIGATES THE CHANNEL MEMORIZATION PROBLEM
The channel memorization problem (Jamal & Qi 2019 Rajendran al. 2020 a known issue oc- curring in non-mounne iutuully--exclssiv task setting, c.g., the task-specint blass-to-laide label is not randomly assigned, and thus the label can be inferred from the query data alone (Yin et al. 2020). Consider 5-way K-shot experiment where the total number of training classes is 5 x L. Now we construct tasks by assigning the label to a class sampled from class iL to (t + 1)L. It is conceivable that the model will learn directly map the query data to the label without using the information of the support data and thus fails to generalize to unseen tasks. This phenomenon can be explained from the perspective that the t column of the final linear layer already accumulates the query fea- tures from tL th to (t + I)L classes. Zeroing the final linear layer implicitly forces the model to the imprinted information from the support features for inferring the label and thus mitigates
20

---

Published as a conference paper at ICLR 2022
this problem. We use the mini-ImageNee dataset and consider the case of L 12. As shown in Figure 13 the zeroing trick prevents the model from the channel memorization problem whereas pecooimitializztioo of the linear layer only works out at the beginning. Besides, the performance of task setting equals the ones
models trained with the zeroing trick under this non-mutually-exe under the conventional few-shot setting shown in Figure As the zeroing trick clears out the final linear layer and equalizes the value of logits, our result essentially accords with Jamal & Qil (2019) that proposes a regularizer maximize the entropy of prediction of the meta-initialized model.
5wway 1-shot
5 5-way 5-shat
Testing accuracy
0.40
0.46
0.50
0.44
0.42
0.33
0.40
sitting for Iree laya iridd ceved eary soer kee
0:38
0.50
0.45
6000 Iteraton (outer loost
12000 14000
5000
15020
25000 30000
korstion loop)
Figure 13: The performance of the models trained on non-mutually exclusive tasks. The mod- els are trained under non-mutually exclusive tasks setting where there is one-to-one assignment between class and channel. Under this circumstance, the zeroing trick tackles the channel memo- rization problem and yields performance similar to conventional few-shot settings.
E EXPERIMENTS ON OMNIGLOT DATASET
Omniglot is hand-written character dataset containing 1623 character classes, each with 20 drawn samples from different people (Lake et al.) 2015). The dataset set is splitted into training (1028 classes). validation (172 classes) and testing (423 classes) sets (Vinyals al. 2016). Since we follow Finn et al. (2017) for setting hyperparamters, we do not use the the validation data. The character images are resized to 28 x 28. For all our experiments, we adopt two experimental settings: 5- way I-shot and 5-way 5-shot where the batch size Nbutch is 32 and Nouery is 15 for both cases (Finn et al. 2017). The inner loop learning rate 7) is 0.4. The models are trained for 3000 iterations using FOMAML SOMAML. The few-shot classification accuracy is calculated by averaging the results over 1000 tasks in the test stage. The model architecture follows the architecture used to train on mini-ImageNNe, but operation as in Finn et al. (0017). The loss function, optimizer, and outer loop learning rate are the same as those used in the experiments on mini- mmageNe.... Each experiment is run on either single NVIDIA 1080-Ti. The results are averaged over four random seeds, and the standard deviation is illustrated with the shaded region. The models are trained using FOMAML unless stated otherwise. The detailed implementation is based on Deleu (2020) (MIT License).
substitute the convolution with max-poolinn max-pooling with strided convolution
We revisit the application of the zeroing trick the testing stage on Omniglot in Figure 14 and observe the increasing testing accuracy, in which such results are compatible with the ones on mini- ImageNet (cf. Figure [3] in the main manuseript). In the following experiments, we evaluate the testing performance only after applying the zeroing trick.
In Figure [5] the distinction between the performance of models trained with the zeroing trick and iiro-initialized models is prominent, sharing remarkable similarity with the results in mini- ImageNet (cf. Figure the main manuscript) in both 5-way 1-shot and 5-way 5-shot settings. We also show the testing performance of models trained using SOMAML in Figure 16 under 5-way 5-shot setting, where there little distinction in performance (in comparison to the results on mini- ImageNet, ef. Figure [0 in the main manuscript) between the models trained with the zeroing trick and the ones trained with random initialization.
For channel memorization task, construct non-mutually-exxhusii training tasks by assigning the label & (where 1 < 1 < 5 in a few-shot 5-way setting) to a class sampled from class tL. to (t + 1)I. where I. is 205 on Omniglot. The toss-nnl- assignment is not applied to the testing tasks. The result is shown in Figure I7 For a detailed discussion, please refer to Section D.5
21

---

Published as a conference paper at ICLR 2022
Original FOMAMI FOMAML
zeroing trick applied tosting stage
23332
1415
Figure 14: Zeroing the final linear layer before testing time improves the testing accuracy on Omniglot. The two subplots on the left: original testing setting. The two subplots the right: the final linear layer is zeroed before testing time. The curves in red: the models whose linear layer is randomly initialized. The curves in yellow: the models whose linear layer is zeroed at initialization. The curves in green: the models whose linear layer is zeroed after each outer loop at training stage.
5-way 1-shot
S-way Sishot
1.000
0.975
0.050
0.925
0.00
0.875
randeer raliaieed o.o
curs arsentiaied one
beeaann trik volting stapi
20
60
100
20
40
60
Iteradion (outer loop!
Eeration (outer
Figure 15: Effect of initialization and the zerving trick in testing accuracy Omniglot. The test performance of the models with reducing the initial norm of the weights of final linear layer is similar to that with the final linear layer being zero-initialized. The distinction in performance between models trained using the zeroing trick and zero-initialized model is more prominent in 5- way 5-shot setting.
FOMAMLL
SOMMML
sutting for Fruor laysr randarbl Inllukee
20
40
60
80
100
2G2
40
60
80
100
Iteration (outer loop!
Beration (outter loog)
Figure 16: The effect of the zeroing trick on models trained using FOMAML and SOMAML on Omniglot. The results suggest that both zero-initialization and zeroing trick mitigate the inter- ference empirically.
5-way 1-shot
5-way 5-shot
0.95
0.90
0.85
0.00
1eaing
0.75
0.70
aeeeet earry 21 stoy.
0.65
20
40
20
100
Iteratio
Itoration
Figure 17: The performance of the models trained non-mutually exclusive tasks Om- niglot. The results are compatible to those on mini- ImageNe (cf. Figure[13]in the main manuscript), suggesting that the zeroing trick alleviates the channel memorization problem.
22


Reviews:
[{"review_id": "vQbDF6uZNuJ", "reviewer": "Reviewer_DKb7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The SCL view proposed in the paper shows the link between MAML and the metric-based approaches for few-shot learning, such as matching network and prototypical network. In my view, this is interesting and might inspire future improvement of MAML for few-shot learning.\n\nWeaknesses:\n- The analysis in the paper is restricted. The SCL view seems to be only valid for few-shot learning. It is also over-simplified to assume that the inner-loop update does not affect the feature backbone if the task is not few-shot learning. I think a better title for the paper is \"MAML is approximately a noisy contrastive learner for few-shot learning\". \n\n- There are no comparisons of MAML with the zeroing trick to metric-based few-shot learning methods in the experiments.  In my view, the SCL view of MAML indeed says that MAML is similar to metric-based approaches, especially when the zeroing trick is applied. The difference only lies in that metric-based approaches explicitly introduce metrics, while MAML with zeroing trick does perceptron-like learning on the top layer. I am surprised to see that the metric-based approaches are not even mentioned in the paper. It would make the results in the paper more insightful if this connection is explored in depth.", "summary_of_the_paper": "In this paper, a new view of MAML under few-shot learning is proposed. The main result is that under the assumption that the inner loop updates are only applied on the top linear layer, MAML actually performs supervised contrastive learning (SCL). SCL shows that MAML learns the feature transformation that makes the intra-class feature distances small, meanwhile the inter-class feature distances large. The zeroing trick is proposed based on this result, showing performance gain in the experiments. \n\n", "summary_of_the_review": "To summarize, I think the SCL view proposed in the paper would be interesting for researchers of few-shot learning. While the result is restricted to few-shot learning, meanwhile no exploration is done to analyse the connection between MAML and metric-based approaches. These drawbacks make the paper less insightful as I expected.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "mCHhZWiGZU_", "reviewer": "Reviewer_XCch", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths \n=======\n\nThe analysis is quite interesting. Their efforts make it explicit the interaction between the support and query set in MAML, and how MAML learns feature encoding. They discover a noisy supervised contrastive loss term in the outer loop loss using the support features as positive and negative samples. Interestingly, they discover another interference term in the outer loop loss which may degrade performance of MAML if last layer linear weights are randomly initialized.  \n\nTo remove the interference term, and also to remove the noise in the supervised contrastive loss, they propose a simple zeroing trick to set the initial weights to be zero after each outer loop update. The trick is simple and reasonable given their analysis results.  \n\nThey conduct experiments to verify the contrastiveness in MAML and improvements using the zeroing trick.  \n\nOverall, the paper is quite clear and easy to follow. \n\nQuestion and weakness \n===================\n\nComparison between eqn (7) and (8) is not clear. Can the authors discuss further the different range of stop gradient? \n\nThe results in Figure 2 are nice but limited, not sure if these can be observed for other classes. What are the 5 classes in the experiment? Can the authors experiment other classes, especially some semantically similar classes? \n\nFurther to the analysis in Figure 2, I am not sure if they really \"verify the supervised contrastiveness\". In particular, the results show that support and query features of same classes become similar as training progresses. I do not think this is particular for supervised contrastiveness. For example, authors can try standard transfer learning and fine-tuning and see if such pattern of consine similarities can be observed. \n\nI am ok with the assumption of freezing the encoder in the inner loop and update only linear layer. But with that, I thought some remarks of the paper sound trivial. For example, “In the inner loop, the features of support data are preserved in the linear layer via inner loop update. In the outer loop, the softmax output of the query data thus contains the inner products between the support features and the query feature.” If you can update only the last linear layer in the inner loop then of course support data feature can only reside there, and therefore in the outer loop it would be inner products between support features and query input as the last layer is a linear layer. Did I miss anything? \n\nThe analysis focuses on classification. Inner product and softmax are critical components in their analysis. But MAML has been applied to other problems, e.g. regression. Perhaps the focus on classification should be reflected in the paper title. \n\n ", "summary_of_the_paper": "The paper analyzes MAML algorithms. Assuming in the inner loop the encoder is fixed and only last linear layer is updated, they analyze the gradient update and loss terms in the inner loop and outer loops (sec 2.3). Through this effort, the authors claim that there are noisy supervised contrastive term in the outer loop loss (eqn (7) and (8)). They further claim that there are additional interference terms which may degrade the performance of MAML at the beginning of training when the linear layer weights are largely random. To overcome this, they propose a simple zeroing trick by zeroing the initial linear layer weights after each outer loop update, essentially removing the interference terms (eqn (9) and (10)). They conduct experiments to support the contrastiveness in MAML and performance improvement using zeroing trick. ", "summary_of_the_review": "Several concerns are listed above. I would be happy to re-consider my recommendation if the responses are reasonable.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "gW4J88o5nHy", "reviewer": "Reviewer_S6WC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I like the simplicity of the zeroing trick, and I found it useful to see it motivated via your theoretical arguments / perform well in your numerical experiments.\n\nIn my view, Section 2.1 could have done a better job of motivating why being a supervised contrastive learner is so appealing. This seems critical given that the rest of the paper provides conditions and theoretical/numerical results supporting that MAML is a (noisy) contrastive learner under certain conditions.\n\nYour theoretical analysis focuses on a specific setting: few-shot classification tasks, softmax output, frozen convolutional encoder during inner loop To what extent do your findings generalize beyond these settings? I had been hoping that the discussion in Section 2.6 (\"Generalization of our Analysis\") would shed some light on this, but it only seemed to address one specific aspect of your analysis, namely the fact that you took $N_{batch}=N_{step}=1$.\n\nAlong the lines of the above, I think that the title and abstract are overly general (neither explicitly referencing your focus on few-shot classification problems, for example) given what you actually showed in the paper.\n\nSorry if I missed this, but did you freeze the encoder in the inner loop for your simulations? If so, did you also evaluate what happens when you didn't freeze it? This would help to give some experimental indication of how general your theoretical findings might be.", "summary_of_the_paper": "This work shows that, in the setting of few-shot classification, MAML is a (noisy) contrastive learner. Complementary theoretical and experimental results are provided. The theoretical results also lead to a (to my knowledge) new proposal, namely to zero out the linear layer after each MAML outer loop update. In their experiments, the authors show that making this small change to the MAML algorithm can lead to meaningful improvements in performance.", "summary_of_the_review": "I found the proposed zeroing trick (and the supporting theory/experiments) to be interesting. I would, however, encourage the authors to work to better motivate their study in the intro of the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper connects MAML to contrastive learning under some simplifying assumptions and with slight modifications in the setting. 
Specifically, the authors show that if the inner loop updates are only applied on the top linear layer, MAML is equivalent to supervised contrastive learning (SCL). This means that MAML learns a feature transformation that brings in-class representations closer and representations across classes far away. The zeroing trick the authors propose seems to give some performance gain in the experiments and is an actionable insight from their theory. 

Overall the paper is very interesting and (as far as I know) novel. The proposed zeroing trick is supported by theory and experiments and is justifies the previous theoretical narrative. 

Some reviewers raised concerns on motivation and numerous clarification questions that the authors have addressed to a large extent in my opinion.
Decision: Accept (Poster)