To Learn Effective Features: Understanding the Task-Specific Adaptation of MAML
Authors: Zhijie Lin
Zhou Zhao
Zhu Zhang
Huai Baoxing
Jing Yuan
Venue:  (2021)
Keywords: Meta-Learning
Few-Shot Learning
Meta-initialization
Task-specific Adaptation

Abstract: Meta learning, an effective way for learning unseen tasks with few samples, is an important research
area in machine learning.
Model Agnostic Meta-Learning~(MAML)~(\cite{finn2017model}) is one of the most well-known gradient-based meta learning algorithms, that learns
the meta-initialization through the inner and outer optimization loop.
The inner loop is to perform fast adaptation in several gradient update steps with the support datapoints, 
while the outer loop to generalize the updated model to the query datapoints.
Recently, it has been argued that instead of rapid learning and adaptation, the learned meta-initialization through MAML
has already absorbed the high-quality features prior, where the task-specific head at training 
facilitates the feature learning.
In this work, we investigate the impact of the task-specific adaptation of MAML and discuss the general formula for
other gradient-based and metric-based meta-learning approaches.
From our analysis, we further devise the Random Decision Planes~(RDP) algorithm to find a suitable linear classifier
without any gradient descent step and the Meta Contrastive Learning~(MCL) algorithm to exploit the inter-samples relationship
instead of the expensive inner-loop adaptation. 
We conduct sufficient experiments on various datasets to explore our proposed algorithms.

Under review as a conference paper at ICLR 2021
TO LEARN EFFECTIVE FEATURES: UNDERSTANDING THE TASK-SPECIFIC ADAPTATION OF MAML
Anonymous authors Paper under double-blind review
ABSTRACT
Meta learning, an effective way for learning unseen tasks with few samples, is an important research area machine learning. Model Agnostic Meta- Learning (MAML) (Finn et al. (017)) is one of the most well-known gradient- based meta learning algorithms, that learns the meta-initialization through the in- ner and outer optimization loop. The inner loop is to perform fast adaptation in several gradient update steps with the support datapoints, while the outer loop to generalize the updated model to the query datapoints. Recently, it has been ar- gued that instead of rapid learning and adaptation, the learned meta-initialization through MAML has already absorbed the high-quality features prior, where the task-specific head at training facilitates the feature learning. In this work, we in- vestigate the impact of the task-specific adaptation of MAML and discuss the gen- eral formula for other gradient-based and metric-based meta-learning approaches. From our analysis, we further devise the Random Decision Planes (RDP) algo- rithm to find a suitable linear classifier without any gradient descent step and the Meta Contrastive Learning (MCL) algorithm to exploit the inter-samples relation- ship instead of the expensive inner-loop adaptation. We conduct sufficient experi- ments on various datasets to explore our proposed algorithms.
INTRODUCTION
Few-shot learning, aiming learn from few labelled examples, is a great challenge for modern machine learning systems. Meta learning, an effective way for tracking this challenge, enables the model to learn general knowledge across a distribution of tasks. Various ideas of meta learning have been proposed to address the few-shot problems. Gradient-based meta learning (Finn et al.
(017 Nichol et al. (2018) learns the meta-parameters that can be quickly adapted to new tasks by few gradient descent steps. Metric-based meta learning (Koch et al. (2015); Vinyals et al. (2016); Snell et al. (2077) proposes to learn a metric space by comparing different datapoints. Memory- based meta learning (Santoro et al. (2016)) can rapidly assimilate new data and leverage the stored information to make predictions.
Model Agnostic Meta-Learning (MAML) (Finn al. (207)) is one of the most well-known gradient-based meta learning algorithms, that learns the meta-initialization parameters through the inner optimization loop and the outer optimization loop. For a given task, the inner loop is to per- form fast adaptation in several gradient descent steps with the support datapoints, while the outer loop generalize the updated model to the query datapoints. With the learned meta-initialization, the model can be quickly adapted to the unseen tasks with few labelled samples. Following the MAML algorithm, many significant variants (Finn et al.) 2018); Rusu et al. (2018); Oreshkin et al.
2018 : Bertinetto et al. (2018); Lee et al. (209bb) are studied under the few-shot setting.
To understand how the MAML works, Raghu al. 2019) conduct a series of experiments and claim that rather than rapid learning and adaptation, the learned veca-initialization has already absorbed the high-quality features prior, thus the representations after fine-tuning are almost the same for the coming unseen tasks. Also, the task specific head of MAML at training facilitates the learning of better features. In this paper, we further design more representative experiments and present formal argument to explain the importance of the task specific adaptation. Actually, the multi-step task- specific adaptation, making the body and head have similar classification capabilities, can provide better gradient descent direction for the features learning of body. We also notice that for both the

---

Under review as a conference paper at ICLR 2021
gradient-based based methods (e.g. MAML (Finn al. (2017)) MetaOptNet (Lee et al. (2019b)) and metric-based methods (e.g. Prototypical Networks Snell et al. (2017)) that attempt to learn a task-
specific head using the support datapoints, the adaptation is a common mode for features learning of body but varied in different methods.
Based on our analysis, first propose a training paradigm to find decision plane (linear classifier) for guidance with no gradient descent step during the inner loop and get more support- ing conclusions. Moreover, we devise another training paradigm that removes the inner loop and trains the model with only the query datapoints. Specifically, inspired by contrastive representation learning (Oord et al. (2018;; Chen et al. (2020): He al. (2020), we exploit the inter-samples re- lationship of query set to find guidance for the body across different tasks. This meta contrastive learning algorithm even achieves competitive results comparable to some state-of-the-ar methods. In total, our contributions can be listed as follows:
1. We present sufficient experiments and formal argument to explore the impact of the task- specific adaptation for body features learning and discuss the general formula for other gradient- based and metric-based meta-learning approaches.
2. We devise training algorithm to obtain decision plane with no gradient descent step during the inner loop, named as Random Decision Planes (RDP), and get more supporting conclusions.
3. Unlike prior gradient-based methods, we propose the Meta Contrastive Learning (MCL) algorithm to exploit the inter-samples relations instead of training a task-specific head dur- ing the inner loop. Even without the task-specific adaptation for guidance, our algorithm still achieve better results with even less computation costs.
4. We empirically shows the effectiveness of the proposed algorithm with different backbones on four benchmark datasets: minilmageNet (Vinyals et al. (2016)), tieredImageNet (Ren et al. 20188), CIFAR-FS (Bertinetto et al. (2018) and FCI00 (Oreshkin et al. (2018)).
RELATED WORKS
MAMI (Finn al. (017)) is a highly influential gradient-based meta learning algorithm for few- shot learning. The amazing experiment results on several public few-shot datasets have proved its effectiveness. Following the core idea of MAML, there are numerous works to handle the data in- sufficiency problem in few-shot learning. Some works (Oreshkin et al. 2018); Vuorio et al. (2019) introduce the task-dependent representations via conditioning the feature extractor on the specific task to improve the performance. [Sun al.) (2019) also employ the meta-learned scaling and shift- ing parameters for transferring from another large-scale dataset. Others (Grant et al. (2018); ; Finn let al. (2018); Lee al.] (2019a)) study this problem from the perspective of Bayesian approach. Un- like prior methods, we provide two training paradigms, one with no gradient descent step during the inner loop and another removing the inner loop and exploiting the inter-sample relations for training.
Recent works also explore the key factors that makes the meta-learned model perform better than others at few-shot tasks. Chen et al. (2019) discovers that deeper backbone has a large effect on the success of meta learning algorithm. while Goldblum et al. (2020) finds that the meta learning tends to cluster object classes more tightly in feature space for those methods that fix the backbone during the inner loop (Bertinetto et al. (2018) Rusu et al. (2018) 8)). A very recent work (Raghu et al. (2019) argues that the meta-trained model can be applied to new task due to the high-quality features prior learned by the meta- meta-initialized parameters rather than rapid learning. In this paper, we further study the impact of the task-specific adaptation for feature learning. Based on the analysis, we devise two algorithms, Random Decision Planes (RDP) and Meta Contrastive Learning (MCL) requiring less computation cost but still with competitive performance.
3 MODEL-AGNOSTIE META LEARNING (MAML)
The MAML aims to learn the meta-initialized parameters 0 for the coming unseen tasks through the inner optimization loop and the outer optimization loop. Under the N-way-K-sha setting, for a task T3 sampled from the task distribution P(T),, we have support set of N x K examples Tg and
2

---

Under review as a conference paper at ICLR 2021
Table I: The evaluation results of 5-way-K-shhe learning for methods with different training regimes on the MinilmageNet and TieredImageNet datasets.
Method
MinilmageNett--wayy
Tieredlmageeet
Multi-Head((
38,66 = 0.34
31.78 0.37
Multi-To (1
40.14 + 0.38
33,62 = 0.38
MAML (201)((1)
49.31 + 0.40
52.45 = 0.48
ANIL (Almost No Inner (1op)(1)
50.23 + 0.42
52.69 + 0.47
BOHI (Body Outer loop, Head Inner Loop)(1)
50.61 + 0.43
53.60 + 0.48
Multi-Head(5)
48.99 + 0.33
41,48 + 0.38
Multi- (asti
50.82 + 0.35
44.94 + 0.39
MAML (5017)(5
64.77 + 0.36
67.66 = 0.42
ANIL (Almost No Inner (5)
65.98 + 0.38
67.44 = 0.43
BOHI (Body Outer loop, Head Inner L5op)(5)
66. + 0.37
68.39 + 0.42
query set TV. where N is the number of sampled class and K is the number of instances for each class. During the inner loop, with the support set Tj., we perform fast adaptation in several gradient descent steps and obtain the task-specific parameters 0r where t is the number of gradient descent steps, given by:
8% = -a 01 aV
Co,
(1)
where a is the step size for inner loop and Crş (0%;') denoted as the loss on the support set Tī after t steps. With the query set Tg. we compute the meta loss on the task-specific parameters 0% and backward to update the meta-initialized parameters 0, given by
B
0= 0- BV Cr
(2)
b=1
where / is the learning rate and B is the number of sampled tasks in a batch.
IMPACT OF TASK-SPECIFIC ADAPTATION
4.1 THE MULTI-STEP TASK-SPECIFIC ADAPTATION IS IMPORTANT.
To explore the effectiveness of MAML. Raghu et al. (0199) have conducted sufficient experiments, indicating that the network body (the representation layers) has already absorbed the high-quality features prior. During meta-testing, instead of fine tuning on the network head (the classifier), simply building the prototypes with the support set can achieve comparable performance MAMI.. Raghu et al. (019) also shows that the task specificity of head at training can facilitate feature learning and ensure good representation learning in the network body. our work, we show that besides the task specificity of head, the multi-step adaptation is also essential, and further study the role of network body and head during meta-training. We devise several methods using different training regimes: (1) Maskk,-Takk where all the tasks simply share one common head and the model is trained in traditional way without inner loop adaptation; (2) Multi-Head, where different tasks are equipped with different heads for task specificity and the model is trained in a traditional way without inner loop adaptation; (3) Almost No Inner Loop (ANII.), where the network body is fixed during the inner loop; (4) Body Outer Loop, Head Inner Loop (BOHI), where the network body is updated only by the outer loop and the head is adapted only during the inner loop, making the head's meta- intta-iiiiialzeed parameters unchanged. More algorithms' details be found in Appendix B and implementation details can be found in Appendix
C.I
Following Raghu et al. 22019), employ the cosine similarities between prototypes and the query datapoints to evaluate the quality of features learned. As Table [ shows, even equipped with task- specific head, the Multi-Hean training still performs worse than the standard MAMI. algorithm by
3

---

Under review as a conference paper at ICLR 2021
30
23
Aooumccy
Accuras
-3
Adgaass
Abcaat sicen Dep 1
ASqeddion 23p
Abd aicn Deg
Alo Dee
Ado 201c Deep 7
30
15
Task ID
Task ID
(a) Body Test Accuracy
(b) Head Test Accuracy
Figure 1: The adaptation of the random initialized model for the sampled tasks in different steps.
Acsurit
Delre Asapadio
5abee Aocutatce Smp
60.2
Adbutrian
Adudean 11a
Acaptaion 2143
A0854.00 am Skp
Ac Acnpadionn S1a3
Adtrtii Smp
Body Test Accuracy
(b) Head Test Accuracy
Figure 2: The adaptation after 5,000 iterations for the sampled tasks in different steps.
a large margin, indicating the multi-step adaptation of MAML is helpful for features learning. The results of Multi-Head and Multi- Task show the importance of multi-step task-specific adaptation.
As the results shown in Table II the ANIL training remains effective comparable to the standard MAML algorithm, indicating that the task-specific adaptation of network body is unnecessary to learn good features. More interestingly, the BOHI training that keeps the meta-initialization of head unchanged even performs better than MAML, further demonstrating that good features learning depends on the multi-step task-specific adaptation of head during inner loop more than updating the meta-initialization of head in outer loop. Also, the ANIL and BOHI have similar performance, indicating that compared with learned prior knowledge head, the inner loop adaptation, as a guidance, contributes more to the features learning. More experimental results be found in Appendix
4.2
WHY IS MULTI-STEP TASK-SPECIFIC ADAPTATION IMPORTANT?
Having observed that the MAML algorithm outperforms the Multi-Task training by large margin and the multi-step task-specific adaptation important for features learning. we extend our analysis to explore the reason why the inner loop adaptation is essential for MAML at different stages of meta training. Specifically, we freeze the initialized MAML model and model at 5,000 iterations, sample validation tasks from the task distribution, and record the test accuracy of model in different inner loop steps. Both the body accuracy based on prototypes construction and head accuracy based on fine-tuning are given in Figure [ and Figure [ where "Task ID" stands for different tasks. As the results shows, at different stages of meta training, the head accuracy increases significantly in the first few adaptation steps since the model has learnt the correspondence between sample and label. However, at the beginning of training, there is only a small improvement on the body accuracy after first adaptation step. In Figure 2 the model converges, the body accuracy decreases in the first few adaptation steps. In the following steps, with the taek-specific adaptation of head, the network body then learns better representations, further demonstrating that the multi-
4

---

reasonable direction for the network body to minimize the target loss La is to make the representation Under review as a conference paper at ICLR 2021
Algorithm The Random Decision Planes (RDP) Algorithm for N-way-K-shot learning
Input:
Network Body fs. Learning Rate 3, Task Distribution P(T)
Perform the Gram-Schmidt method on random metrices to get the classifier set P {
while not done do
Sample batch of tasks (Th) where To P(T)
for b E {1.... B) do
Sample the support set Т = [< )= Nx
and query set Tg = {(] y/ Nx from task Tb.
i=1
for each sample a in {T TT) do
== ==]]]()>]||
end for
define CrossEntropy H, D) as the cross entropy loss on the features representations set D with head H.
W = arg min LooysEutropyyLoss W, [(=], ot) WEP
L. = CoossEntropyLoss W {((= (V*(( 1))
end for
0 = 0 - OV 1 Em Lb
end while
step task-specific adaptation, making the body and head have similar classification capabilities, can be regarded as guidance to provide better gradient descent direction for the feature learning of body.
To understand this intuitive argument better, we consider a sample (æx y) for few-shot classification where the cross entropy loss is employed, formulated as:
exp(w, h) Le -log( Σn exp(w, h) -w h + log( Σ exp(w h))
(3)
where [ W2.... wR is the weights of the classifier head, h is the body representation of æ. The gradients loss La with respect to the body representation are denoted by,
dhc 0h
Ee waexp(wF h)) Σ. expx(w
wy
Wy 2
(4)
where v is exactly the weighted average of the weights { {WI, W2, .... w... As shown in Equation
h closer to the corresponding class weight wy, given by h h + X(w, w.. As the model converges, in the first few adaptation steps, there a significant margin between the performance of head and body, and the classifier weights contain little knowledge about correspondence between samples and labels and differences between different classes. With the low low-performance head, this updating rule for body may lead to a decline in the quality of features, which also explains why the simpler BOHI, ANIL even performs better than MAML in Table [] After several adaptation steps during the inner loop, the body then receives the useful guidance for features learning from the task- specific head since wy can better express its corresponding class. The formulation above shows that the multi-step task-specific adaptation, making the body and head have similar classification capabilities, can provide better gradient descent direction for the features learning of body.
4.3 TASK-SPECIFIC ADAPTATION IN OTHER META-LEARNING ALGORITHMS
Having noticed that the multi-step task-speeific adaptation of MAML, which promotes the perfor- mance of head, can facilitate the features learning of body. It works similarly for other gradient- based methods that use end-to-end fine-tuning, such as Reptile [Nichol et al. (2018))
the case of meta-learning methods that fix the network body and only update the head during the inner loop, such as MetaOptNet (Lee et al. (2019b)) and R2-D2 (Bertinetto et al. (2018). ). the convex optimiza- tion of head also aims to provide classifier with better classification capabilities. For nntticc-bsse methods, such as Prototypical Networks (Snell et al. (2017)), the adaptation of head is actually con- ducted through the nearest neighbor algorithm. In conclusion, the adaptation common mode but
5

---

Under review as a conference paper at ICLR 2021
Table 2:
The evaluation results of 5-way-K-shot learning for the standard MAMI. and Random Decision Planes (RDP) with different backbones.
Method
Backbone
MinilmaggeNet-S-wayy
TieredlmageNet--wwayy
FC100-5-way
MAML (1)
Conv4
49.31 = 0.40
52.45 = 0.48
34.75 = 0.39
RDP(I)
Conv4
46.12 + 0.38
47.63 = 0.44
36,63 = 0.38
RDP(1)
ResNet12
51.16 = 0.43
51.37 = 0.46
37.54 0.40
MAML (5)
Conv4
64.77 + 0.36
67.66 = 0.42
43.90 0.38
RDP(5)
Conv4
63.34 + 0.36
65.19 0.42
49.46 = 0.39
RDP(5)
ResNet12
65.72 + 0.36
66.31 + 0.41
50.29 0.38
65
RDP S-shot
Test Accurapy
RDP
Test Accuracy
45
RDP 1-shol
55
30
64 Number of decision planes
64 Number of decision planes
32
128
512
(a) MinilmageNet 5-way
FC100 5-way
Figure 3: The effect of the number of decision planes the minilmageNet and FC100 datasets.
varied different methods. These meta-learnigg algorithms reveal a general formula that the inner loop is for building task-specific head that matches the classification capabilities of body and the outer loop for task-independent features learning.
5 THE RANDOM DECISION PLANES AL.ORRITHM
As discussed above, the multi-step adaptation based gradient descent during the inner loop aims to provide guidance for features learning of body. From this consideration, we suppose that if a suitable linear classifier is given, the feature learning the inner loop. From this considcration, we devise such an algorithm named Random Decision Planes (RDP), where classifier is chosen from predefined set according the target loss the support set. The predefined set of classifier P consists of np different orthonormal matrices that method from random matrices. During the inner loop,
be facilitated even without gradient descent during
are generated through the Cram-Schmin without gradient descent, we directly choose a most suitable classifier as the network head which minimizes the cross entropy loss on the support set. In the outer loop, compute the loss based on the chosen head and run backward to update the network body. A formal description of RDP is The implementation details can be found in Appendix C.I
presented in Algorithm [
The overall evaluation results on three datasets are presented in Table Note that we also remove the head and construct the prototypes from the body network fo for predictions during meta-testing. The proposed RDP algorithm performs comparably to the standard MAML method on three datasets, especially on the FC100 dataset. Without any task-specific adaptation for the network body, best performing classifier chosen from set of randomly generated subspaces can also be guidance to facilitate the features learning, further suggesting that a head with better classification capabilities, key factor learn good representations even if the chosen approximate head performs worse than a gradient-basee head, and the main purpose of task-specific adaptation is to adjust the low- performance head for features learning of body.

---

Under review as a conference paper at ICLR 2021
Algorithm 2
The Meta Contrastive Learning (MCL) Algorithm for N-way learning
Input:
Network Body fe. Projection Layer Ié, Learning Rate B,
Constant T. Task Distribution P(T)
while not done do
Sample batch of tasks (Th) where TB ~ P(T)
for E (1, .1., B] do
Sample the query set 7g from task T3 where Tg = {(a, 1!))
and 1 = 1228 where N..
for i [1..... 2N do
g6 (ff
end for for i [1...., 2 and j E {1,... 2N do
Si,j = 2 ž (1] Zi 11
end for
define [(i,j) )) = -log( ΣズN [xxx) //
exp(sL,/7)
Lo = 2N Σ 11(2 - 1, 2k) + 12k, 2k- - 1)]
end for = 0 V // Σ Li
/V $ Σ Lb
end while
Also, we conduct experiments to explore the impact of the number of decision planes. Results are shown in Fo two datasets. With a small set of decision planes, it can be more difficult find a suitable head to guide the features learning, while with enough decision planes, the performance then reaches the upper limit.
THE META CONTRASTIVE LEARNING ALGORITH
We have already seen that the multi-step task-specific adaptation improve the classifier head can essentially facilitate the features learning of body. In total, prior maudient-based methods based on the cross-entropy loss proposes to learn the correspondence between samples and assigned labels for different tasks, thus requiring the task-specifie adaptation for the classifier head during inner loop. Since the task-specific head also serves for features learning of body, we wonder if we can remove the inner loop or adaptation, and make full use of the labels information in other way to be a guidance for features learning. From this consideration and inspired by recent works (Chen) al. (2020); He et al. (2020)) about self-supervised contrastive learning, we further devise the Meta Contrastive Learning (MCL) algorithm that directly removes the inner loop and exploits the inter-sample relationship with only the query set.
Specifically, rather than using cross entropy loss for specifcc adaptation, we simply impose that normalized representations from the same class closer together than representations from different classes. For N-way few-shot learning, we sample two examples per class to build the query set. Next, for given anchor example, the meta contrastive loss pulls closer the point of same class while pushes the anchor farther away from the negative examples of other classes. Following Chen
(2020), we also employ small neural network projection layer that maps the body features to the space where contrastive loss is applied. A formal description of MCL is presented in Algorithm 2 The implementation details can be found in Appendix C.II
During meta-testing, we discard the projection layer Is and construct the prototypes from the body network fo for predictions. The overall evaluation results on the MinilmageNet, magred and FC100 datasets are presented in Table 3, Note that TADAM (Oreshkin et al. (2018)) employs a extra task embedding network (TEN) block to predict element-wise scale and shift vectors, and MetaOptNet (Lee et al. (2019b)) proposes to learn linear support vector machine (SVM) as clas- sifier head during the inner loop. Unlike those methods, our MCL method is arguably simpler. By exploiting the relationship between different samples, contains a complex adaptation process, and devise a contrastive loss to train the network body di- rectly. As the results shows, our method outperforms almost previous well-designed methods and
able remove the inner loop which

---

Under review as a conference paper at ICLR 2021
Table 3: The evaluation results of 5-way-K-shot learning for the Meta Contrastive Learning (MCL) and other baselines with different backbones.
Method
Backbone
MinilmageNNt--5-wyy
TieredImageNee-5-wayy
FC100-5-way
MAML (1)
Conv4
49.31 = 0.40
52.45 = 0.48
34.75 + 0.39
MCL(1)
Conv4
50.73 = 0.43
53.12 0.48
37.73 0.38
TADAM (2018
(I)
12
58.50 = 0.30
40.10 + 0.40
MetaOptNer (2019b)(1)
ResNet12
62.64 + 0.61
65.99 0.72
41.10 0.60
MCL(T)
ResNet12
62.14 0.43
65.98 0.50
41.38 + 0.40
MAML (2017)(5)
Conv4
64.77 + 0.36
67.66 0.42
43.90 0.38
(CL(5)
Conv4
66.25 + 0.36
69.31 + 0.41
51.29 + 0.39
TADAM (2018 (5)
ResNet12
76.70 + 0.30
56.10 0.40
MetaOptNet (2019b)
ResNet12
78.63 + 0,46
81.56 + 0.53
55.50 + 0.60
MCL())
ResNet12
78.34 0.33
81.09 0.37
56.64 0.39
75
65
MCL 1-shot
MCL 1-shot
70
5-ahot
MCL 5-shot
Test T Accuracy
66
Test Accuracy
80
65
50
45
40
55
64
128
256
512
None
64
128
256
512
None
output dimension
The output dimension of
(a) MinilmageNet 5-way, Conv4
(b) MinilmageNet 5-way, ResNet12
Figure 4: The effect of output dimension of do on the MinilmageNet dataset.
also achieves results comparable to MetaOptNet. More experimental results time-efficiency analysis can be found in Appendix C.2 and C.3
We also study the impact of the projection layer 9a- Figure shows the evaluation results with different output dimensions. Note that "None" means that there projection layer for loss com- putation. As the results show, for a deeper ResNet12 backbone, the projection layer facilitates the features learning lot (>6% for 5-shot, >5% for 1-shot). We conjecture that the projection layer is trained to extract task specifc information useful for the contrastive loss, while the body represen- tations h learns general information. More analysis can be found in Appendix C.4
CONCLUSION
In this paper, based on the hypothesis that feature reuse is the dominant factor for the success of algorihh,,, we further study the impact of task-specific adaptation and devise several training regimes including BOHI, Multi-Head and so on. Also, we provide a more formal argument from the perspective of gradient descent optimization. Based on analysis above, we find that the multi- step task-specific adaptation, making the body and head have similar classification capabilities, can provide better gradient descent direction for the features learning body. We further connect our results to other meta-leaming algorithm, showing the adaptation is common mode but varied in different methods. From our consideration, we devise the RDP algorithm where a suitable linear classifier is chosen without gradient descent and get more supporting conclusions. We also build the
8

---

Under review as a conference paper at ICLR 2021
MCL algorithm that removes the inner loop and exploit the inter-sample relationship, and achieve results comparable to some state-of-the-arr methods.
REFERENCES
Luca Bertinetto, Joao F Henriques, Philip HS Torr, and Andrea Vedaldi. Meta-learning with differ- entiable closed-form solvers. arXiv preprint
2018.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05700 2020.
Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look few-shot classification. arXiv preprint arXiv:1904.04332, 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. arXiv preprint wrXiv:/703.0344 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. Ad- vances Neural Information Processing Systems, pp. 9516-9527, 2018.
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4367- 4375, 2018.
Micah Goldblum, Steven Reich, Liam Fowl, Renkun Ni, Valeriaa Cherepanova, and Tom Gold- stein. Unraveling meta-learming: Understanding feature representations for few-shot tasks. arXiv preprint S10.:::2002206753,, 2020.
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient- based meta-leariig hierarchical bayes. arXiv preprint arXiv:1801.0892 08930, 2018.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, volume 2. Lille, 2015.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 10 (canadian institute for advanced re- search). URL http://www. es. toronto. edu/kriz/cifar. html, 5, 2010.
Hae Beom Lee, Hayeon Lee, Donghyun Na, Sachoon Kim, Minseop Park, Eunho Yang, and Sung Ju Hwang. Learning balance: Bayesian meta-learning for imbalanced and -uttt--distritttinn 2019a.
tasks. arXiv preprint 19X.v:190S.12911
Kwonjoon Lee, Subhransu Maji. Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10657-10665, 2019b.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv:1803 02999,99999.. 2018
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprini arXiv: 187.03880 2018.
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In Advances in Neural Information Processing Systems, pp. 721-731, 2018.
Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan L Yuille. Few-shot image recognition by predicting parameters from activations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7229-7238, 2018

---

Under review as a conference paper at ICLR 2021
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning feature reuse? towards understanding the effectiveness of maml. arXiv preprint arXiv:1909.09157, 2019.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. 2016.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard S Zemel. Mcta-learning for semi-supervised few-shot classifica- tion. arXiv preprint arXiv: 00676, 2018.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):2112555 2015.
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osin- dero, and Raia Hadsell. Meta-learning with latent embedding optimization. arXiv preprint /807.0/807.085 2018.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with memory- -uggneee neural networks. In International conference on machine learn- ing. pp. 1842-1850, 2016.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems, pp. 4077-4087, 2017.
Qianru Sun. Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few-shot learning. Proceedings of the IEFE conference on computer vision and pattern recognition, pp- 403-412, 2019.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, pp. 3630-3638, 2016.
Risto Vuorio, Shao-Hua Sun, Hexiang Hu, and Joseph J Lim. Multimodal aggost- meta- learning via task-aware modulation. In Advances Neural Information Processing Systems, pp.
2019.
A FEW-SHOT IMAGE CLASSIFICATION DATASETS
In this section, we introduce four benchmark datasets often used for few-shot image classification: the minilmageNet (Vinyals et al. (2016)), mageed (Ren et (2018)) CIFAR-FS (Bertinetto) et al. )0188) and FC100 (Oreshkin et al. (2018)).
The minilmageNet (Vinyals et al. (016)) dataset standard benchmark for few-shot image clas- sification, comprises 100 classes randomly chosen from the original ImageNet (Russakovsky et al. 2015 dataset, where 64 classes is used for meta-training. 16 classes for meta-validation and 20 classes for meta-testing. Each class contains 600 images of size 84 84. Since the original class splits are unavailable, we use the commonly-used split proposed in Ravi Larochelle (2016).
The Ieteed (Ren et al. (2018) dataset is another larger subset of ImageNet (Russakovsky al. (2015)). This dataset contains 608 classes that are grouped into 34 high-level categories, where 20 categories (351 classes) are used for meta veea-trainnng, 6 categories (97 classes) for meta-validation and categories(16) 160 classes) for meta-testing. All images are also size of 84 x 84.
The CIFAR-FS (Bertinetto et al. (2018) (1) dataset is a few-shot image classification benchmark, con- sisting of all 100 classes from CIFAR- 100 (Krizhevsky al. (210)). These classes are randomly split into 64, 16, and 20 separately for meta-training, vecaa-validtion and meta-testing, Each class
contains 600 images of size 32 x 32.
The FC100 (Oreshkin et al. (2018)) dataset is another benchmark derived from CIFAR- 100 Krizhevsky et al. (2010)). This dataset comprises 100 classes that are grouped into 20 high- level categories, where 12 categories (60 classes) are used for meta-training, 4 categories (20 classes) for meta- volaavaaid and categories (20 classes) for meta-testing. Each class contains 600 images
of size 32 x 32.
10

---

Under review as a conference paper at ICLR 2021
Table 4: The evaluation results of 5-way-K-shot learning the MinilmageNet dataset.
Method
Conv4, K=1
Conv4, K=5
ResNet12, K=1
ResNet12, K=5
MAML (Fin et al. (2017))
49.31 + 0.40
64.77 + 0.36
57.45 0.47
72.70 + 0.35
Almost No Inner Loop (ANIL)
50.23 = 0,42
65.98 + 0.38
59.43 = 0.44
73.28 + 0.33
Body Inner, Head Outer (BOH)
50.61 = 0.43
66.14 0.37
59.69 = 0.46
73.42 + 0.37
MetaOptNet (Lee al. (2019b)
62.64 = 0.61
78.63 + 0.46
Meta Contrastive Learning (MCL.
50.73 + 0.43
66.25 + 0.36
62.14 + 0.43
78.34 + 0.33
B MORE DETAILS ABOUT ALGORITTM
In this section, we provide further details about the training regimes and algorithms mentioned above. Note that we denote the meta parameters of the network as 0 in previous sections. Consider- ing the network composed the body (feature extractor) and head (classifier), we further rewrite 0 as 0 [0y.0cl where Or, 0g is the parameters of body and head respectively. For given task Tb = {Tz. T.). the meta-initian mnta-initialization updating of MAML can be expressed as follows:
0 8 a a Lrg (00-1100-11 0¹ -0 aV oo (8) .6
(S)
, 0 Vo, v, Crg (0,, 02), 0. = Oc v. Crg (0,, 00)
where is the step size of the inner loop, 3 is the learning rate. In work, devise eral methods using different training regimes including Multi-Task, Multi-Head, Almost No Inner Loop (ANIL) and Body Outer Loop, Head Inner Loop (BOHI) study the role of network body and head during meta-training.
The updating rules of Multi-Task can be expressed follows
f = f - Lrg (0y, 0z), 0g = 3o (0,,
(6)
For different tasks, the Multi-head has different heads for task specificity, given by
0f = f - Ve, CTx )t:), 0 0td - BV on (0, 6%)
where 8 is the specific parameters for task Tg. The network body is fixed during the inner loop for ANIL, given by
0. = 0% a av 0:- LT, (0, 0 8, 8, V Cr 8.), O. 0. SV Cog
For BOHI, the network body is updated only by the outer loop and the head is adapted only during the inner loop, making the head's metaa-nitialized parameters unchanged, given by
8 86-1 a V CTg (07-1 1,0--1)
0, 0y e, (0,, 0%)
C MORE EXPERIMENTAL DETAILS
C.I IMPLEMENTATION DETAILS
For all training regimes, RDP and MCL, we use the Adam optimizer with weight decay of 5e-4 and the learning rate is set to le-3, For 4-layer convolution network with 64 filters, we flatten the output feature map of the network body, and obtain 1600-d features for minilmageNet and tieredImageNet, while 256-d features for CIFAR-FS and FC100. For ResNet12 network, we employ global max pooling layer on the output feature map of the network body, and obtain 512-d features for four public datasets. During meta-training, adopt horizontal flip, random crop and color (brightness,
11

---

Under review as a conference paper at ICLR 2021
Table 5: The evaluation results of 5-way-K-shot learning the Nagee dataset.
Method
Conv4, K=1
Conv4, K=5
ResNet12, K=1
ResNet12, K=5
MAML (Finn et al. (2017))
52.45 + 0.48
67.66 + 0.42
63.05 0.50
77.01 + 0.40
Almost No Inner Loop (ANIL)
52.69 = 0.47
67.44 + 0.42
63.03 = 0.49
76.98 + 0.41
Body Inner, Head Outer (BOH)
53.60 = 0.48
68.39 + 0.42
63.20 = 0.51
77.11 0.40
MetaOptNet (Lee al. (2019b)
65.99 = 0.72
81.56 : 0.53
Meta Contrastive Learning (MCL.
53.12 + 0.48
69.31 + 0.41
63.98 + 0.50
81.09 + 0.37
Table 6: The evaluation results of 5-way-K-shot learning on the CIFAR-FS dataset.
Method
Conv4, K=1
Conv4, K=5
ResNet12, K=1
ResNet12, K=5
MAML (Finn et al. (007))
61.96 + 0.51
76.16 + 0.39
67.41 + 0.51
80.94 + 0.37
Almost No Inner Loop (ANIL)
63.27 + 0.52
77.25 + 0.38
69.23 = 0.50
81.85 + 0.36
Body Inner, Head Outer (BOHI)
63.58 + 0.52
77.11 + 0.38
68.41 = 0.52
80.86 + 0.37
MetaOptNet (Lee et al. (2019b)
72.00 + 0.70
84.20 + 0.50
Meta Contrastive Learning (MCL)
64.59 + 0.52
77.24 + 0.38
71.17 + 0.49
84.18 + 0.35
Table 7: The evaluation results of 5-way-K-shon learning on the FC100 dataset.
Method
Conv4, K=1
Conv4, K=5
ResNet12, K=1
ResNet12, K=5
MAML (inn al. (017))
34.75 + 0.39
43.90 + 0.38
38.43 + 0.39
50.85 + 0.38
Almost No Inner Loop (ANIL)
37.49 + 0.38
49.58 + 0.39
38.60 = 0.40
50.70 + 0.39
Body Inner, Head Outer (BOHI)
37.15 1 0.41
48.68 + 0.39
38.63 = 0.40
50.65 + 0.38
MetaOptNet (Lee et al. (2019b))
41.10 + 0.60
55.50 + 0.60
Meta Contrastive Learning (MCL)
37.73 + 0.38
51.29 + 0.39
41.38 0.40
56.64 + 0.39
contrast, and saturation) jitter data augmentation as proposed in Gidaris & Komodakis 20 Qiao al. (2018). We train all models 100 epochs and take 500 batches per epoch. For MAML, BOHI and ANIL, both models are trained using 5 gradient steps of size a = 0.01 for Conv4 and a 0.1 for R2 For the Random Decision Planes algorithm, the number decision planes np is set to 64. For the Meta Contrastive Learning (MCL) algorithm, we apply a two-layer nonlinear projection layer with hidden size of 512. Also, the query datapoints come from different classes for each sampled task, which is helpful for accelerating model convergence.
C.2 MORF RESULTS FOR BOHI, ANIL,, MAML., MCL.
In this section, we provide complete experimental results for BOHI, ANIL, MAMI and MCL. with different backbones on four datasets. The complete results on four datasets are presented in Table T Table and Table respectvepyt The results can further verify description mentioned above. Good features learning depends on the multi-step task-specific adaptation of head during the inner loop more than updating the meta-initialization of head in outer loop. With the low- performance head, the update of body may even lead to a decline in the quality of features. In addition, the results on four datasets further demonstrate the effectiveness of our proposed MCL algorithm.
C.3 THE TIME-EFFICIENCY ANALYSIS FOR BOHI, ANIL, MAML, MCL
is obvious that ANIL BOHI and MCL can speeds up training. The results about the comparison of computation time are presented in Table We implement our methods based on PoTcrc and the
12

---

Under review as a conference paper at ICLR 2021
Table 8: The computation time of different (tthods.(tasks//se
Method
Conv4
ResNet12
MAML (Finn et al. (017))
10.60
2.24
Random Decision Planes (RDP)
31.24
6.88
Almost No Inner Loop (ANIL)
30,98
6,86
Body Inner, Head Outer (BOHI)
31.12
6.88
Meta Contrastive Learning (MCL)
63.76
30.96
training of models is run on two NVIDIA 1080Ti GPU. Notice that our MCL can run much faster than BOHI and ANIL while achieves better evaluation results. The training speedups also illustrate the significant computational benefit of MCL and prove its effectiveness.
Table 9: The evaluation results about the quality of features extracted by the network body and projection layer. (5-way-5-shot on MinilmageNet, ResNet12)
Hidden size of 9ø
Network Body fo
Projection Layer Js
64
77.69 + 0.33
61.83 + 0.37
256
77.98 + 0.34
63.46 + 0.38
512
78.34 = 0.33
66.27 + 0.40
C.4 ABOUT THE PROJECTION LAYER OF MCL.
We have found that with a deeper backbone, the features learning can be facilitated a lot by the projection layer. We further evaluate the quality of features extracted by the network body and the projection layer. The evaluation results are given in Table Even the contrastive loss is applied to the projection layer, the network body learns better and general representations. We conjecture that during the meta- training the projection layer may absorb more taec-fiec information while the backbone tends to learn task-independent representations.
13


Reviews:
[{"review_id": "vsXDm2uaW8", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper adds to a series of papers that look at understanding how and why MAML works as well as it does, and what exactly is going on in the adaptation process. There are many open questions in this regard, and so I think the topic of the paper is interesting and of interest to the community. However I found the paper somewhat difficult to read and I'm not entirely sure what to take away from the paper. It was often not clear to me why certain things were introduced or why certain experiments were conducted. Overall I think that the contribution is not strong enough for publication at ICLR.\n\nI'll structure my questions according to the contributions listed in the introduction, and the corresponding main sections in the paper:\n\nContribution 1, Section 4:\n- If I understand this point correctly, then you're arguing that the task-specific adaptation in the inner loop is *necessary* such that the outer loop can learn a good feature extractor, is that correct? But doesn't the fact that you can do MLP contradict this statement, because here you've basically removed the task-specific adaptation.\n- Section 4.1.: You compare four methods here, to show that multi-step task adaptation is important. \n - What do you mean with \"Muti-step\" adaptation? Are you focusing on the fact that you need multiple *gradient* updates in the inner loop and that one is not enough? \n - (Model 1) How is the multi-task model trained? You say that all tasks use the same head, so how do you add the task-specific information to the classifier? How does this work in the few-shot learning setting? How do you adapt the model at test time? \n - (Model 2) How is the multi-head model trained? I can see how you can do this for a small, fixed number of training tasks, where you just learn one head per task over the entire course of meta-training. But the what happens at test time? You say there is no inner loop, but the test accuracy is larger than random. So do you re-initialise a random new head and perform gradient updates on that?\n - (Model 3) Add a citation here to make clear that ANIL is from Raghul et al. 2019\n - (Model 4) Is this your own method, or is it taken from prior work? If I understand correctly a similar model is investigated in concurrent work: https://arxiv.org/abs/2008.08882 (which is not a problem; but you might be interested in what they find and it would be good to just add a brief mention of this concurrent work somewhere in the paper).\n- Section 4.2: In this section you ask *why* multi-step adaptation is important.\n - In Figure 1 and 2, how come the accuracy before adaptation is significantly larger than random?\n- What do you mean with \"We present _sufficient_ experiments\"? Sufficient in what sense? \n\nContribution 2, Section 5:\n- It's kind of cool that the suggested algorithm of using (random) pre-defined classifiers (last layer of the network) instead of adapting the last layer using gradient descent in the inner loop works well on few-shot image classification (on two datasets worse, on one dataset better than MAML). I think this illustrates a similar point as ANIL (Raghu et al. 2019), namely that only adapting the last layer is sufficient on some problems.\n-  In Figure 3 you show the number of \"decision planes\" in the set of classifiers that you use. I don't understand how using 1 decision plane can give such high (~30-55%) accuracy, can you explain this? If I have only one decision plane to choose from in the inner loop, then I always have to choose the same one, so adaptation isn't really possible. Shouldn't I just get random performance, so 20% on the 5-way problem then?\n- I think it would help to explain in this section why you are proposing this algorithm. Is it to drive home an observation about MAML? If so what does this example do that wasn't shown in ANIL (Raghu et al. 2019)? And/or is it to propose a new method that people should use - in which case it would be good to argue why. Current meta-learning methods perform much better than what RDP can achieve and evaluating the accuracy on all available heads might be a computational overhead. But maybe there are situations where we'd rather do that than gradient updates? \n- Do you have any intuition as to why RDP works better on the FC100 dataset compared to MAML, but not on the other datasets? Does that dataset have different properties? I think the reader could benefit from your insights here.\n- Why does the accuracy drop slightly when 512 decision planes are used compared to using 128?\n\nContribution 3, Section 6:\n- Do I understand correctly that this method basically does something like NIL (Raghu et al. 2019) but then explicitly trains the features with a contrastive loss such that features from different classes get pushed further apart? If that is the case, shouldn't you compare to NIL here? \n- There are a few other methods that use features to compare classes, or contrastive losses, and I'm surprised you don't compare to these. E.g. https://arxiv.org/1807.02872 https://arxiv.org/2008.09942 https://arxiv.org/abs/1606.04080\n- You write that \"out method outperforms almost previous [sic] well-designed methods and also achieves results comparable to MetaOpNet\". But current methods SOTA that aren't listed in your comparison get much higher (+20 percentage points) accuracies (https://paperswithcode.com/sota/few-shot-image-classification-on-mini-1). Could you explain why you chose to compare to the methods in Table 3?\n- I think I have the same question as my last point above for [2]. It's not clear to me what the purpose of the proposed method is - illustrating a point or proposing a useful new algorithm?\n\n---------------------------------------------------\n---------------------------------------------------\nUPDATE:\n\nI have read the other reviews and the author's response. \n\nThank you for the thorough answers - this cleared a lot of things up and I understand much better how the multi-head, multi-task and random decision planes work. I'll increase my score to a 5 because I've gotten more insight now with the additional information, and think that the paper raises some interesting points. Overall, I still tend towards rejection - even with the updated version, I still find the contributions of the paper difficult to tease out and evaluate, and not all claims in the paper are sufficiently backed up / analysed by experiments. I would encourage the authors to try and centre the entire paper more clearly around *one single* central message in the future, and present all experiments in this light, making sure that every claim is sufficiently backed up empirically. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "uvRdnjX-495", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary:** \n* This paper conducts analysis of the task-specific adaptation in the MAML algorithm, building on recent work analysing the effectiveness of MAML. The paper explores different variants of MAML, and provides empirical analysis to argue that the multi-step task-specific adaptation of a network head in the MAML algorithm is important in learning good representations and thus enabling effective few-shot learning performance.  \n* The paper then proposes two algorithms: one that removes gradient-based updates for a network head (instead using random matrices for classification) and another one that uses contrastive learning (no inner loop adaptation). These are evaluated on benchmark datasets, and shown to have improved computational speed.\n\n\n**Overall Comments:** This paper consists of some analysis on task-specific adaptation, and presentation of two related few-shot learning methods that build on some of the insights from the analysis. As specified in the detailed comments, I am unconvinced about some of the experimental evaluation in the analysis section, and I am not sure the comparisons are fair. \nRelatedly, one of the two algorithms presented is also unconvincing to me (with the presented evaluation, it doesn't appear to strengthen the claim made by the analysis). \nIn contrast, the second algorithm presented is clear and interesting, with notable computational benefits and performance improvements over baselines. \nA stronger version of this paper could either focus on the analysis, making the claims clearer and strengthening the evaluation (see thoughts below), or present a shortened version of the analysis and then focus more on the MCL algorithm, stressing its performance improvements and computational benefits. In its current form, due to a weaker analysis section and limited analysis of MCL, I do not think this paper is strong enough for acceptance.\n\n\n**Detailed Comments:** \n* The analysis about the importance of multi-step adaptation is a bit unclear at points, and the experimental evaluation may not support the claim that the multi-step task specific adaptation is important. If the claim is the multi-step adaptation in MAML is specifically the important aspect, then a fairer comparison would be MAML/ANIL/BOHI with 1 inner adaptation step for the head vs MAML/ANIL/BOHI with 5 inner adaptation steps for the head, and comparing how these affect feature learning.  \n* As explored in Raghu et al (2019), there is an alignment problem that could make the multi task setting perform poorly, so this result is not as surprising.\n* One aspect of the multi head setting is a bit unclear -- for every task, do we initialise a new 5-way head and then perform only 1 gradient step to update these parameters? It seems that a fairer comparison would be to perform more than 1 update step. From a randomly initialised head, I would expect > 1 step to be important to help classification performance. Again though, if the claim is specifically about multi-step adaptation being important for feature learning in MAML, then the 1 inner step vs 5 inner steps on MAML/ANIL/BOHI is a more relevant comparison, as the multi head situation is fundamentally different. \n* Section 4.2 is a more convincing argument (both the mathematical statement and the plots) of the important of task specific adaptation of the head. I think this analysis as a whole could be summarised as how good gradient signal for the network body is only obtained when the head of the network is (at least somewhat) effective at classification, and a bad head will lead to poorer learning signal. I also want to note that in Fig 2, the improvements in the body's representations is quite small overall (a delta of < 0.3 percent). Minor comment: it would be good to specify in the text/figure caption some more experimental details about Figs 1/2 (dataset, N way K shot, etc).\n* Section 4.3 appears to extend the statement made in Section 6 of Raghu et al (2019) to newer methods, but is still an interesting observation. Minor comment: The method in Lee et al (2019b) is MetaOptNet, not MetaOpNet.\n* The RDP algorithm is an interesting investigation, but I am not convinced of its utility or that it supports the overall argument of the paper. Even though FC100 performance is improved, the performance on Mini/TieredImageNet is appreciably worse than MAML, especially for 1 shot settings. Further evaluation would help this claim.\n* The MCL algorithm section was interesting and I was impressed by the results, and the computational speedups that this method allows for. I think that this method itself is a good contribution. Table 8 in the appendix is in my opinion interesting enough that it would be good to put it in the main text. It is a nice idea to apply the insights from contrastive learning for good representations to FSL, especially as prior work has demonstrated good representation learning to be very important for good FSL performance.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "SVDp2_-8Cn0", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \n\nIn this paper, the authors investigate the inner-loop optimization mechanism of meta-learning algorithms. The analysis shows the effectiveness of the multi-step adaptation and (1) the key of meta-learning is how to design a well-differentiated classifier. They then propose Random Decision Planes (RDP) and Meta Contrastive Learning (MCL) and achieve comparable performance with existing methods.\n\n\nPros:\n1. Empirically investigating the performance w.r.t. the change of optimization mechanism on both head and body is very important in the meta-learning field.\n\nCons:\n1. My major concern is about the contribution of this paper. \n - The discussion is interesting. But most of the analysis results is widely accepted. For example, adapting the head layer improves performance. One claim  \"as the model converges, the body accuracy even decreases in the first few adaptation steps\" is not well-explained.\n - The final goal of this paper is to design a better metric for meta-learning (feature mapping function + differentiated metric design). I think the goal is the same as metric-based meta-learning, which is not new for me. \n - In addition, the baselines and experiments are not sufficient to support the goal. Especially, the proposed methods (both MCL and RDP) only show comparable performance compared with metric-based methods (e.g., MetaOptNet). It would also be better if the authors can add more metric-based baselines for comparison (e.g., [1],[2]). Moreover, they should also involve metric-based methods in the efficiency comparison section (Appendix C.3).\nThus, I feel the overall contributions are not enough to be accepted. \n\n2. Besides the Cons 1, if the authors focus on gradient-based meta-learning. Similar to ANIL, more experiments on different types of applications are supposed to conduct. For example, the experiments on regression and reinforcement learning tasks.\n\n3. The paper is not well-written. Here are some comments:\n - It would be more clear to formally formulate the MCL and RDL with more notations and equations.\n - Most figures are not clear. It is better to replot them with larger font size in legend, thicker lines, etc.\n\n[1] Ye et al., Few-Shot Learning via Embedding Adaptation with Set-to-Set Functions. CVPR 2020.\n\n[2] Wang, Yan, et al. \"Simpleshot: Revisiting nearest-neighbor classification for few-shot learning.\" arXiv preprint arXiv:1911.04623 (2019).", "extra_scores": {}, "extra_text": {}}, {"review_id": "FOzn_m5LnuI", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n\n1.\tBased on previous work (Raghu et al. 2019), this paper provides some analysis to explain why multi-step task-specific adaptation is important. \n\n2.\tThe idea of introducing contrastive learning is reasonable and leads to performance improvement in experiments.\n\nWeaknesses:\n\n1.\tThe proposed methods have little improvements over base models; Random Decision Planes (RDP) actually obtains worse results than the basic MAML. Although it is reasonable for the proposed method to use contrastive learning, its novelty is quite limited. Some questions are listed below, which would be great if they can be answered.\n\n2.\tAnalyses in some section are of heuristic nature, and thus not very convincing (please see the detailed comments below).\n\n3.\tThe presentation of the paper needs some improvement. Since the proposed algorithms are mainly based on the idea of learning better feature extraction, the analysis on why multi-step adaptation is important, thus, contributes little to the understanding of the proposed methods, which can be directly motivated by results in Raghu et al. 2019.\n\nDetailed Comments & Questions:\n\n1.\tWould it be possible to provide some more details on the implementation of ‘Multi-Head’ in section 4.1? How to make the head parameters task-specific to randomly sampled tasks? Are the heads re-initialized at every iteration? Or simply use n heads for a batch of n tasks, and then apply the updated n heads to the next batch of n tasks? I think this might be important.\n\n2.\tI do agree that low-accuracy head may provide bad signals for the model body. But the heuristic nature of the analysis is not convincing enough. Firstly, the body is updated based on the head’s prediction accuracy.  Obviously, there will be a significant gap between the body accuracy and the head accuracy. Secondly, the parameters in the body might not change “efficiently” as the head parameters, because the gradient with respect to the body parameters (especially the low-level features) might be small. It will be helpful if we can know the averaged norm of elements changes (or gradients) in the body and the head, respectively, or the averaged spectral norm of the matrix changes (or gradients) of the body and the head, respectively. Also, what if different step sizes are used in the body and the head? Overall, question about the importance of multi-step adaptation is still not well-answered.\n\n3.\tIn Table 2, results of RDP are shown, which are worse than  the basic MAML. If the analysis in previous sections are correct, why do not simply use ANIL or BOHI, which totally avoid the influence of the low-accuracy heads on the model body and  the results after adaptation, and obtain much better experimental results? What is the benefit of using RDP?\n \n4.\tIs it possible to provide more details on meta-testing for MCL? Is the classification based on RDP or using a new fully connected layer (as head)? Will the body also be updated during meta-testing? What will the results be if using cosine similarity as in Raghu et al. 2019, which might be better to illustrate the quality of features.\n\n5.\tIs it possible to visualize the features using algorithms like t-SNE? This will allow us to see how the contrastive learning leads to features that can better distinguish/discriminate samples from different classes.\n\n6.\tI agree that contrastive learning could benefit feature extraction, and consequently helps the final results after adaptation. It might be interesting to see whether multiple datasets can be used for pre-training, e.g. train the model by contrastive loss using FC100, CIFARFS, and ImageNet 32\\times 32, and then test on only one of them.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: MAML is a well-known gradient-based bi-level optimization to learn a good initialization over a set of relevant tasks. This paper investigate different variants of MAML, providing empirical analysis of two new algorithms (RDP and MCL). Reviewers agree that it is interesting to see what the change of optimization mechanism on both head and body brings to us in the MAML framework. This is done by only empirical analysis. However, all reviewers have concerns that the current version (or even revised one after the author responses) does not contain substantial contributions over existing work in the sense that: (1) experiments do not support well what's been claimed; (2) writing should be much improved to clearly explain the formulation of RDP and MCL, as well as figures in experiments; (3) the analysis about the importance of multi-step adaptation is not clear (Section 4); (4) the proposed method has little improvements over baseline methods.  Without any positive feedback from reviewers, I do not have choice but to suggest rejection. 




Decision: Reject