BOIL: Towards Representation Change for Few-shot Learning
Authors: Jaehoon Oh
Hyungjun Yoo
ChangHwan Kim
Se-Young Yun
Venue: ICLR 2021 Poster (2021)

Abstract: Model Agnostic Meta-Learning (MAML) is one of the most representative of gradient-based meta-learning algorithms. MAML learns new tasks with a few data samples using inner updates from a meta-initialization point and learns the meta-initialization parameters with outer updates. It has recently been hypothesized that representation reuse, which makes little change in efficient representations, is the dominant factor in the performance of the meta-initialized model through MAML in contrast to representation change, which causes a significant change in representations. In this study, we investigate the necessity of representation change for the ultimate goal of few-shot learning, which is solving domain-agnostic tasks. To this aim, we propose a novel meta-learning algorithm, called BOIL (Body Only update in Inner Loop), which updates only the body (extractor) of the model and freezes the head (classifier) during inner loop updates. BOIL leverages representation change rather than representation reuse. A frozen head cannot achieve better results than even a random guessing classifier at the initial point of new tasks, and feature vectors (representations) have to move quickly to their corresponding frozen head vectors. We visualize this property using cosine similarity, CKA, and empirical results without the head. Although the inner loop updates purely hinge on representation change, BOIL empirically shows significant performance improvement over MAML, particularly on cross-domain tasks. The results imply that representation change in gradient-based meta-learning approaches is a critical component.

Published as a conference paper at ICLR 2021
BOIL: TOWARDS REPRESENTATION CHANGE FOR FEW-SHOT LEARNING
Jaehoon Oh"l, Hyungjun You"!, ChangHwan Kim & Se-Young Yun?
1 Graduate School of Knowledge Service Engineering, KAIST
"Gaduaaee School of Artificial Intelligence, KAIST (jaehoon. oh, yooh kimbob, yunseyoung) @kaist. ac.kr
ABSTRACT
Model Agnostic Meta-Learning (MAML) is one of the most representative of gradient-based meta-learning algorithms. MAML learns new tasks with a few data samples using inner updates from mntta-initialization point and learns the intaainitializatiinn parameters with outer updates. It has recently been hypothesized that representation reuse, which makes little change in efficient representations, is the dominant factor in the performance of the meta-initiallzzed model through MAML in contrast representation change, which causes significant change in representations. In this study, we investigate the necessity of representation change for the ultimate goal of few-shot learning, which is solving domain-agnostic tasks, To this aim, we propose a novel meta-learning algorithm. called BOIL (Body Only update in Inner Loop). which updates only the body (extractor) of the model and freezes the head (classifier) during inner loop updates. BOII. leverages represen- tation change rather than representation reuse. This is because feature vectors (representations) have to move quickly to their corresponding frozen head vectors. We visualize this property using cosine similarity. CKA, and empirical results with- out the head. BOIL empirically shows significant performance improvement over MAML, particularly on change in gradient- based meta-learnin approaches is critical component.
tasks. The results imply that representation
INTRODUCTION
Meta-learning, also known as "learning to learn," is methodology that imitates human intelligence that can adapt quickly with even a small amount of previously unseen data through the use of previous learning experiences. To this aim, meta-learning with deep neural networks has mainly been studied using metric- and gradient-bart based approaches. Metric-based meta-learning (Koch) 2015 Vinyals et al. 2016 Snell et al., 2017; Sung et al., 2018) compares the distance between feature embeddings using models as mapping function of data into an embedding space, whereas gradient- based meta-learning (Ravi Larochelle] 2016 Finn et al. 2017 Nichol al. 2018) quickly learns the parameters to be optimized when the models encounter new tasks.
Model-agnostic meta-learning (MAML) (Finn al. 2017 is the most representative gradient-based meta-learning algorithm. MAML algorithm consists of two optimization loops: an inner loop and an outer loop. The inner loop learns task-specific knowledge, and the outer loop finds a universally good meta-initialized parameter allowing the inner loop to quickly learn any task from the initial point with only a few examples. This algorithm has been highly influential in the field of meta-learning, and
numerous follow-up studies have been conducted (Oreshkin et al. 2018
Rusu et al.) 2018 Zintgral
al. 2018 Yoon et al. 2018 Finn et al.
2018 Triantafillou al.
2019 Sun al.
2019 Na et al.
2019
Tseng et al.) 2020).
Very recent studies (Raghu et al. 2020; Arnold et al. 2019) have attributed the success of MAML to high-quality features before the inner updates from the meta-initialized parameters. For instance, Raghu al. (2020) claimed that MAML learns new tasks by updating the head (the last fully connected layer) with almost the same features (the output of the penultimate layer) from the meta- initialized network. In this paper, we categorize the learning patterns as follows: A small change in the representations during task learning is named representation reuse, whereas a large change is named representation change] Thus, representation reuse was the common belief of MAML.
* 'The authors contribute equally to this paper.
In our paper, representation reuse and representation change correspond to feature reuse and rapid learning (Raghu al. 2020), respectively. To prevent confusion from terminology, we re-express the terms.

---

Published as a conference paper at ICLR 2021
(a) MAML/ANIL
(b) BOIL.
Figure 1: Difference in task-specific (inner) updates between MAMI/ANII, and BOIL. In the figure, the lines represent the decision boundarics defined by the head (classifier) of the network. Different shapes and colors mean different classes. (a) MAML mainly updates the head with a negligible change in body (extractor); hence, representations on the feature space are almost identical. ANIL does not change in the body during inner updates, and they therefore identical. However, (b) BOIL updates only the body without changing the head during inner updates; hence, representations on the feature space change significantly with the fixed decision boundaries. We visualize the representations from various data sets using UMAP (Uniform Manifold Approximation and Projection for dimension reduction) (Meinnes et
2018
Appendix B
Herein, we pose intriguing question: Is representation reuse sufficient for meta-learning? ? We believe that the key to successful meta-learning is closer to representation change than to repre- sentation reuse. More importantly, representation change is crucial for cross-domain adaptation, which is considered the ultimate goal of meta-learning. By contrast, the MAML accomplished with representation reuse might be poorly trained for cross-domain adaptation since the success of representation reuse might rely heavily on the similarity between the source and the target domains.
To answer this question, we propose a novel meta-learning algorithm that leverages representation change. Our contributions can be summarized as follows:
We emphasize the necessity of representation change for meta-learning through cross- domain adaptation experiments.
We propose a simple but effective meta-learning algorithm that learns the Body (extractor) of the model Only in the Inner Loop (BOIL). We empirically show that BOIL improves the performance over most of benchmark data sets and that this improvement is particularly noticeable in fine-gr fine-grained data sets or cross-domain adaptation.
We interpret the connection between BOIL and the algorithm using preconditioning gradients (leennerhag et al., 2020) and show their compatibility, improving performance.
We demonstrate that the BOIL. algorithm enjoys representation layer reuse on the low-/mid- level body and representation layer change the high-level body using the cosine similarity and the Centered Kernel Aligmment (CKA). We visualize the features between before and after adaptation, and empirically analyze the effectiveness of the body of BOIL through an ablation study on eliminating the head.
For ResNet architectures, we propose a disconnection trick that removes the back- propagation path of the last skip connection. The disconnection trick strengthens rep- resentation layer change on the high-level body.
2 PROBLEM SETTING
2.1 META-LEARNING FRAMEWORK (MAML)
The MAMI algorithm (Finn et al. 2017) attempts to meta-learn the best initialization of the parame- ters for task leakner It consists of two main optimization loops: inner loop and outer loop. First, we sample a batch of tasks within a data set distribution. Each task т, consists of a support set S,, and query set Q When sample support set for each task, we first sample labels from the label set and then sample i instances for each label. Thus, each support set contains n x instances. For a query set, sample instances from the same labels with the support set.
With these tasks, the MAML algorithm conducts both meta- trattatrriinng and meta-testing. During meta- training, we first sample meta-batch consisting of B tasks from the meta-training data set. In the
2

---

Published as a conference paper at ICLR 2021
inner loops, we update the meta- veta-initialized parameters 0 to task-specific parameters 0- using the task-specific loss Ls., (fo), where fe is a neural network parameterized by 8, as follows
0 = -- (fo)
(1)
Using the query set of the corresponding task, we compute the loss LQq, (Je..) based on each inner updated parameter. By summing all these losses, the meta-loss of each meta-batch, Lieea (0), is computed. The intta-iitiiliize parameters are then updated using the meta-los in the outer loop through a gradient descent.
B 0' - --0-0V (V where metet (0) = Σ La. (fo..)
(2)
In meta-testing, the inner loop, which can be interpreted task-specific learning, is the same in meta-training. However, the outer loop only computes the accuracy using query set of tasks and does not perform a gradient descent; thus, it does not update the meta-initialization parameters.
2.2 EXPERIMENTAL SETUP
We used two backbone networks, 4conv network with 64 channels from Vinyals et al. (2016) and ResNet-12 starting with channels and doubling them after every block from cremhknen et al. (2018). For the batch normalization, testing. following the original MAML (Finn et al. 2017). We trained 4conv network and ResNet- 12 for 30,000 and 10,000 epochs, respectively, and then used the model with the best accuracy meta- validation data set to verify the performance. We applied an inner update once for both meta-training and meta-testing. The outer learning rate set to 0.5 and 0.3 for 4cony network and ResNet- 12, respectively. All results were reproduced by our group and reported as the average and standard deviation of the accuracies over 5 1,000 tasks, and the values in parentheses in the algorithm name column of the tables are the number of shots. We validated both MAML/ANIL and BOIL on two general data sets, minilmageNeet (Vinyals et al. 2016) and tieredlmageNett (Ren et al. 2018), and two specific data sets, Cars (Krause et al. 2013) and CUB /Welinder et al.) 2010). Note that our algorithm is not for state-of-the-arr performance but for proposal of new learning scheme for meta-learning. Full details on the implementation and data sets are described in Appendix A In addition, the results of the other data sets at size of x 32 and using the 4conv network with 32 channels from Finn et al. (2017) (i.e., original setting) are reported in Appendix C
used batch statistics instead of the running statistics during meta-
0.001 and 0.0006 and the inner learning rate was
and Appendix D respectively.
BOIL (BODY ONLY UPDATE IN INNER LOOP)
3.1 THE ULTIMATE GOAL OF
DOMAIN-AGNOSTIC ADAPTATION
Recently, Raghu et al. (2020) proposed two opposing hypotheses, representation reuse and repre- sentation change, and demonstrated that representation reuse is the dominant factor of MAML. We can discriminate two hypotheses according to which part of the neural network, body or head, is mostly updated through the inner loop. Here, the body indicates all convolutional layers, and the head indicates the remaining fully connected layer. other words, the representation change hypothesis attributes the capability of MAML to the updates on the body, whereas the representation reuse hypothesis considers that the network body is already universal to various tasks before the inner loops. To demonstrate the representation reuse hypothesis of MAML, the authors proposed the ANIL (Almost No Inner Loop) algorithm, which only updates the head in the inner loops during training and testing, and showed that ANIL has a performance comparable to that of MAML. This implies that the representation trained by MAML/ANIL, even before updated task task-specifically. is sufficient
2 Although the inner loop(s) can be applied through one or more steps. for simplicity, we consider only the case of a single inner loop.
3 'll implementations are based on Torchmeta (Deleu et al. 2019 except for WarpGrad, and all results were reproduced according to our details. These results are not the highest for MAML/ANIL because our setting is more fitted to BOII.. However, under more suitable hyperparameters for each algorithm, the best performance of BOIL is better than that of JAMI.
3

---

Published as a conference paper at ICLR 2021
to achieve the desired performance. Furthermore, they proposed the NIL-testing (No Inner Loop) algorithm, which removes the head and performs unseen tasks using only the distance between the representations of support set and those of query set during testing to identify the capability of representation reuse. -testtgg of MAMI. also achieves a performance comparable to MAMI.. Based these results, it was claimed that the success of MAML is attributed representation reuse.
Here, we investigate the necessity of representation change. We believe that the meta- meta-trained models should achieve a good performance in many other domains, which is referred as domain-agnostic adaptation in this paper. To this end, represenlation reuse is not appropriate since representation reuse uses the similarity between the source and target domains. The higher the similarity, the higher the efficiency. Therefore, when there are no strong similarities between the source and target domains, good representations for the source domain could be imperfect representations for the target domain. Table 2 which lists our experimental results on cross-domain tasks, shows that the MAMI, enjoying representation reuse worse than BOIL leveraging representation change, which will be discussed in detail in the next section.
3.2 BOIL ALGORITHM
Inspired by the necessity, we design an algorithm that updates only the body of the model and freezes the head of the model during the task learning to enforce representation change through inner updates. Because the gradients must be back-propagated to update the body, we set the learning rate of the head zero in the inner updates during both meta-training and meta-testing. Otherwise, the learning and evaluation procedures of BOIL are the same as those of MAML. Therefore, the computational overhead does not change.
Formally speaking, with the notations used in Section 2.1 the meta- meta-initiallzze parameters @ can be separated into body parameters 0, and head parameters 0n. that is, 0 = (0s, 0z). For sample image E IR', an output can be expressed as ÿ = fo( = fo, (Jo, (a:) E R", where fo, (I) E Rd. The task- specificc body parameters f,.r and head parameters 0p,r. through an inner loop given task T; are thus as follows:
Oo., = 0, a6 Vo, Ls,, (So) & On.,. = 02 or Vo, Ls. (Jo)
where dg, and aG are the inner loop learning rates corresponding to the body and head, respectively. MAML usually sets a as on 0). ANIL sets an. 0 and on. * O, and BOIL as j 0 and O% = O.
These simple differences force the change in the dominant factor of task-specific updates, from the head to the body. Figure body.Figure shows the main difference in the inner updates between MAML/ANIL and BOIL. To solve new tasks, the head mainly or only changes in MAML/ANIL (Raghu al. 2020). whereas in BOIL, the body changes.
3.2.1 PERFORMANCE IMPROVEMENT ON BENCHMARK DATA SETS AND
TASKS
Table 1:
Test accuracy (%) of 4conv network on benchmark data sets. The values in parenthesis in the algorithm name column of tables are the number of shots.
Domain
General (Coarse- grained)
Specific (Fine graine))
Dataset
minilmage Net
fieredImageNet
Cars
CUB
MAML(1)
47.44 1 0.23
47.44 1: 0.18
45.27
1 0.26
56.18
1 0.37
ANIL(1)
47,82 1 0.20
49.35
1 0.26
46.81
1 0.24
57.03
0.41
BOIL())
49.61 + 0.16
48.58
+:
56.82
+: 0.21
61.60
+ 0.57
MAML(5)
0.42
64.70
0.14
0.26
69.66
0.03
(NTL((5)
63.04
+ 0.42
65.82
+: 0.12
61.95
+ 0.38
70.93
0.28
BOIL(5)
66.45
1: 0.37
69.37
1: 0.12
75.18
1 0.21
75.96
0.17
Table I [nnd Table 2 display the superiority of BOIL on most benchmark data sets and on cross-domain adaptation lasks, where the source and target domains differ (i.e., the meta-training and meta-testing data sets are different). In Table I] the performance improvement is particularly noticeable on the specific domain data sets Cars and CUB. The results demonstrate that representation change is necessary even if there is a similarity between the source and target domains. [Table 2 shows that BOIL is closer the ultimate goal of meta-learning, which is domain-agnostic adaptation.
4

---

Published as a conference paper at ICLR 2021
Table 2: Test accuracy (%) of 4conv network on cross-domain adaptation.
adaptation
General to General
General to Specific
Specific
General
Specific
Specific
meta-train
ticredImageNet minilmageNet
minilmageNer
minilmageNer minilmageNNrr
Cars
Cars
CUB
Cars
meta-test
Ne
Cars
CUR
minilmageNNt
tieredInageNet
Cars
CUB
47.60 - 0.24
51.61 1 0.20
33.57 1 0.14
40.51 1: 0,08
26.95 1 0.15
28.46 = 0.18
32.22
1 0,30
29.64 = 0.19
ANIL.(1)
49.67 1: 0.31
52.82 1: 0.29
34.77 1 0.31
41.12
28.67
1 0.17
29.41 4: 0.19
33.07
0.43
28.32
0.32
BOIL(1)
49.74 0.26
53.23
0.41
36.12
0.29
44.20
33.71
+ 0.13
34,06 + 0.20
35.44
+ 0.46 34,79
0.27
(5ML(5)
65.22 1: 0.20
65.76
1 0.27
44,56
1 0.21
53.09 0.16
30.64
1 0.19
32.62 1 0.21
41.24
0.21 32.18
1 0.13
(NIL((5)
66.47
66.52
0.28
46.55
0.29
55.82
0.21
35.38
0.30
36.94
0.10
43.05
+
0.23
37,99
0.15
BOIL(5)
69,33 0.19
69.37
+ 0.23
50.64 = 0.22
60.92
+ 0.11
44,51
+ 0.25
46,09
= 0.23
47.30
0.22 45,91 + 0.28
Recently, Guo et al. 2019) noted that existing meta-learning algorithms have weaknesses in terms of cross-domain adaptation. We divide the cross-domain adaptation into four cases: general to general, general to specific, specific general, and specific to specific. Previous studies considered the cross-domain scenario from a general domain to a specific domain (Chen et al.) 2019 Guo et al. 2019). In this paper, we also evaluate the reverse case. BOIL outperforms MAML/ANIL not only on the typical cross-domain adaptation scenario but also on the reverse one. In particular, the performance improvement, when the domain changes from birds (CUB as meta-train set) to cars (Cars as meta-test set), implies that the representation change in BOIL enables the model to adapt to an unseen target domain that is entirely different from the source domain.
3.2.2 ABLATION STUDY ON THE LEARNING RATE OF THE HEAD
In this section, control the inner loop update learn- ing rate of the head to verify the effect of training the head to the performance. The results are depicted in Table 3) The best performance is achieved when the learning rate is 0 (BOIL). However, the accuracy rapidly decreases as the learning rate of the head grows. Even with 1/10x head learning rate com-
Head's Learning Rate (og)
minilmageNet
Cars
0.00 (BOIL)
66.45 0.37
75.18 + 002T
0.05
38.81
0.21
68.67 1 0.21
0.I0
0.16
68.86 :: 0.30
0.5 (MAML in ours)
61.75 : 0.42
53.23 = 0.26
Table 3: 5 5- 5-Shot test accuracy accord- ing to the learning rate of the head. pared to other layers, the test accuracies are significantly degraded. Therefore, it is thought that freezing head is crucial.
3.2.3 BOIL AND PRECONDITIONIN GRADIENTS
Some aspects of BOIL can be explained by preconditioning gradients (Lee & Choil 2018 Flennerhag jet al. 2020). Preconditioning gradients occur when a particular layer is shared over all tasks, warping the spaces (e.g.. rotating and scaling). For instance, one might consider the frozen head of BOIL to be a warp layer of the entire body (Flennerhag et al. 2020).
Preconditioning gradients can avoid overfitting in sigh-capacity model (Flennerhag et al. 2020), and such a benefit is still valid with BOIL. Indeed, many prior studies have suffered from an overfitting problem, and thus it is challenging to train the backbone network more extensively than the 4conv network with 32 filters (Finn et al.) 2017). By contrast, BOIL can increase the validation accuracy with more extensive networks. The accuracy of models with 32, 64, and 128 filters continues to increase to 64.02, 66.72, and 69.23, without overfitting. In Appendix El we report these results as well as the training and valid accuracy curves of BOIL for three different network sizes, in which the larger networks are trained well. We further hypothesize that the head is the most critical part of an overfitting problem, and BOIL can succeed in dealing with the problem by simply ignoring the head in the inner loops.
However, one essential difference between BOIL and the preconditioning gradients is whether the head is frozen. Prior studies did not freeze the last fully connected layer or used any additional fully connected layer to precondition the gradients, and hence representation reuse is still the major factor of their training. To the best of our knowledge, BOIL is the first approach that enforces representation change by freezing the head in the inner loops.
Model
Accuracy
To investigate the gain from representation change, we adapt BOIL to WarpGrad (Flennerhag et al. [2020] Four different models are tested, the architectures of which are fully described in Appendix F Table 4 shows the test accuracy of the four models, where the BarL-War
WarpGrad w/ last warp head WarpCirad who last warp head
83.19 0.79 83.16 0.69 83.68 0.82
BOIL- HaLL-WarrGrad w/ last warp conv
BOIL- WarpCirad w/o last warp cony
84,88 0.42
Table 4: Test accuracy(%) of WarpGrad and BOIL- Warp-Ga over 5 x 100 tasks.
WWe follow the setting in http:///iuu.ooo/ffeeeeaaaaa////a in Appendix F This task is related to long-adaptation task.
and the details about this implementation
5

---

Published as a conference paper at ICLR 2021
models freeze the fully connected layer from the corresponding WarpGrad model. It is observed that GOIL-WarpGrat improves WarpGrad and BOIL- -arpGrad without the last warp conv improves BOIL-Warpe Warp-War with the last warp conv. The latter result indicates that, to support BOIL. the last convolution layer must not be fixed but rather updated during the inner loops.
4 REPRESENTATION CHANGE IN BOIL
4.1
REPRESENTATION CHANGE BEFORE/AFTER ADAPTATION
Intra-ckoo
Inter-dioa
Beftee adaplation on
Afterta diainon
Dnfrre ecppha
Atteracapaaloon
Delzee staptal
Mfte ataplation
022
0.5
333
3.0
0091 0127 CEA
aad
oxsl
coael
0tet Coea caY
orst
caeel Caase coat
oxst
cntt caee caast cas
(a) MAMI..
(b) ANIL.
(c) BOIL.
Figure 2: Cosine similarity of 4conv network.
To analyze whether the learning scheme of BOIL is representation reuse or representation change, we explore the layer-wise alteration of the representations before and after adaptation. We compute the cosine similarities and CKA values of the convolution layers with the meta-trained 4conv network (as detailed in Appendix A). We first investigate the cosine similarity between the representations of a query set including classes and 15 samples per class from minilmageNee after every convolution module. In Figure 2] the orange line represents the average of the cosine similarity between the samples having the same class, and the blue line represents the average of the cosine similarity between the samples having different classes. In [Fiiurre 2 the left panel of each algorithm is before the inner loop adaptation, and the right panel is after inner loop adaptation.
The key observations from Figure which are also discussed in Section 4.2 with other experiments. are as follows:
The cosine similarities of MAML/ANIL (Figure 2a and Figure 2bb have the similar patterns, supporting representation reuse. Their patterns do not show any noticeable difference before and after adaptation. They make the average of the cosine similarities monotonically decrease and make the representations separable by classes when the representations reach the last convolution layer. These analyses indicate that the effectiveness of MAML/ANIL heavily leans on the meta-initialized body, not the task-specific adaptation.
The cosine similarities of BOIL (Figure 2c) have different pattern from those of MAMI/ANII, suuppoooiu representation change. The BOIL's pattern changes to distinguish classes after adaptation. Before adaptation, BOIL. reduces the average cosine similarities only up to conv3, and all representations are concentrated regardless of their classes after the last convolution layer. Hence, BOIL's meta-initialized body cannot distinguish the classes. However, after adaptation, the similarity of the different classes rapidly decrease on conv4, which means that the body can distinguish the classes through adaptation.
The reason why the change in BOIL before and after adaptation occurs only on conv4 is a peculiarity of the convolutional body, analyzed by Zeiler & Fergus 2014). Although the general and low-level features produced through the front convolution layers (e.g., colors, lines, and shapes) do not differ much from the task-specific adaptation, the discriminative representations produced through the last convolution layer (conv4) differ from class to class. The importance of the last convolutional layer on the performance in few-shot image classification tasks is also investigated by Arnold et al. y; Chen et al. (2020). These changes before and after the adaptation support the fact that BOIL enjoys representation layer reuse at the low- and mid-levels of the body and representation layer change in high-level of the body Nevertheless, the degree of representation layer reuse in a low- and mid-levels in BOIL is lower than that in MAML/ANIL, which is measured using gradient norms (Appendix G We also report the cosine similarity including head in Appendix H
'In the (Raghu al. 2020), representation reuse/change are defined at the model-level. Namely, represen- tation reuse indicates that none of the representations after the convolution layers signiticantly change during the inner loop updates. otherwise representation change. We extend this concept from the model-level to the layer-level. To clarify it, use representation layer reuse/change for the layer-level. Therefore, if model has even one layer with representation layer change, is said that the model follows representation change.

---

Published as a conference paper at ICLR 2021
Through these observations, we believe that MAML follows the representation reuse training scheme, whereas BOIL follows representation change training scheme through representation layer reuse before the last convolution layer and representation layer change at the last convolution layer.
Next, we demonstrate that BOIL enjoys representation layer reuse on the low- and mid-level and representation layer change the high- level of the body by computing the CKA (Kornblith et al. (2019) before and after adaptation. When the CKA between two representations is close to 1, the representations are almost identical. In Figure 3| as mentioned in Raghu al. (2020), CKA shows that the MAML/ANIL algorithms do not change the representation in the body. However, BOIL. changes the representation of the last convolution layer. This result indicates that the BOIL algorithm learns rapidly through repre- sentation change. addition, the representation change on the Cars data set is described in Appendix
CKA
1.0
0.4
0.2
VAVL
ANL
Figure 3: CKA of 4conv
4.2 EMPIRICAL ANALYSIS OF REPRESENTATION CHANGE IN BOIL
Table 5: Test accuracy (%) of 4conv network according to the head's existence before/after an adaptation.
mela-train
mim mimllmageNet Net
meta-test
migilmageNet
Cars
head
w/ head
w/o head (Niltnesst)g)
W/ head
W/o head (Nll--testin)
adaptation
before
after
before
after
before
after
before
after
MAML(1)
19.96 0.25 47.44 0.23 48.28 0.20 47.87 1: 0.14
20.05
0.16 33,57
0.14
1: 0.19 34.36
0.16
ANIL(I)
20.09 1 0.19 47.92
0.20 48.86 1 0.12 48.86 1 0.12
20.16 0,05 34.77 0.31
35.48
1: 0.24 35.48
0.24
B1IL(1)
19.94 + 0.13 49.61 = 0.16 24.07 + 0.19 46.73 + 0.17
19.94 = 0.06 36.12 0.29
23.30
+ 0.15
34.07
0.32
MAML(5)
20.04 = 0.17 61.75 = 0.42 64.61 0.39 64.47 = 0.39
19.97 = 0.18 44.50
= 0.21
47.66
+ 0.28 47.53
= 0.28
ANIL(5)
20.09 + 0.13 63.04
+ 0.42 66.11 +: 0.51 66.11
+ 0.51
20.08 + 0.07 46.55 0.29
49.62 + 0,20 49.62 0.20
BOIL.5)
20.04 + 0.21 66.45 0.37 32.03 + 0.16 64.61 + 0.27
20.06 0.16 50.64 0.22
30.33 + 0.18 50.40 0.30
Table 5 describes the test accuracy on minilmageNet and Cars of the model meta-trained on mini- ImageNet before and after an inner update according to the presence of the head. To evaluate the performance in a case without classifier, we first create template of each class by averaging the representations from the support set. Then, the class of the sample from the query set is predicted as the class whose template has the highest cosine similarity with the representation of the sample. This is the same as NIL-testing in (Raghu et al. 2020).
The results provide some intriguing interpretations:
With the head for all algorithms. Before adaptation, all algorithms on the same- and cross-domain are unable to distinguish all classes (20%). This status could be considered as an optimum of meta-initialization representations have to move quickly to their corresponding frozen head. Moreover, after adaptation, BOIL overwhelms the performance of the other algorithms. This means that representation change of BOIL is more effective than representation reuse of MAML/ANIL.
We also discuss it in AApeeddxx L. In BOIL, the
Without the head in MAML/ANIL. In this setting, representations from the body evaluated before and after adaptation. Before adaptation, MAML and ANIL already generate sufficient representations classify, and adaptation makes little or difference. The former observation matches the cosine similarity gap between an intra- and inter-class on each left panel in Figure 2a and Figure 2b) and the latter observation matches the CKA values of close to or exactly on conv4 of MAML/ANIL in Figure 3]
Without the head in BOIL. BOIL shows steep performance improvement through adap- tation on the same- and cross-domain. This result implies that the body of BOII. can be susk-specifically updated. It is matched with Figure 2c where the cosine similarity gap between the intra-class and inter-class on the space after conv4 is near zero before adaptation but increases after adaptation. This implies that the poor representations can be dramatically improved in BOIL. Therefore, the low CKA value on conv4 of BOIL in in/uueeee 3 natural.
To summarize, the meta- intttainitiaizztiionn by MAML and ANIL provides efficient representations through the body even before adaptation. By contrast, although BOIL's meta- initializationn provides less efficient representations compared to MAML and ANIL, the body can extract more efficient repre- sentations through task-specific adaptation based on representation change.
7

---

Published as a conference paper at ICLR 2021
Note that the penultimate layer (i.e.. conv4) of BOIL acts differently from the output layer (i.e., head) of MAML/ANIL. The penultimate layer of BOIL might seem like a pseudo-head head layer, but not at all. The output layer of MAML/ANIL is adapted based the well-represented features (i.e., features after conv4). In contrast, the penultimate layer of BOIL is adapted based on the woorly-represented features (i.e., features after conv3). It means that the head layer's ' role of MAMI/ANIL is to draw simple decision boundary for the high-level features represented by the output of the last convolutional layer. However, the penultimate layer of BOIL acts non-linear transformation so that the fixed output head layer can effectively conduct classifier role. We also report empirical analyses of penultimate layer of BOIL and an ablation study on learning layer in Apppediix J and Appendix K
BOIL TO A LARGER NETWORK
Many recent studies (Vuorio et al., 2019 Rusu et al. 2018 Sun et al. 2019) have used deeper networks such as ResNet (He et al.
2016). Wide-ResNet (Zagoruyko & Komodakis 2016), and DenseNet (Huang et al., (017) ) as a backbone network. The deeper networks, in general, use feature wiring structures facilitate the feature propagation. We explore BOIL's applicability to deeper network with the wiring structure, ResNet-12 12, and propose a simple trick to boost representation change by disconnecting the last skip connection. This trick described in Section 5.1
Table 6: 5- 5-y 5-Shot test accuracy (%) of ResNet- 12. LSC means Last Skip Connection.
Meta-train
minilmageNet
Cars
Meta-test
minilmageNet
tieredImageNet
Cars
Cars
minilmageNet
CUB
MAML w/ LSC
68.51 0.39
71.67 + 0.13
43.46
0.15 75.49
1 0.20
34.42 0.06
35.87:0..1
MAML w/o LSC
67.87
0.22
70.31 + 0.10
41.40 0.11 73.63 1
37.65
0.11
0.26
ANIL w/ LSC
68.54
0.34
71.93 :: 0.11
45.13
79.45
1
35.03
0.07
35.09
ANIL w/o LSC
67.20 0.13
69.79
- 0.24
43.46
= 0.18
75.32
38.15
0.15
36104
BOIL w/ LSC
70.50
0.28
71.86 0.21
49.69
0.17
80.98
+ 0.14
45.89
0.32
4.23
BOIL w/o LSC
71.30 + 0.28
74.12 + 0.30
49.71 0.28 83.99
0.20
48.41
+ 0.18 44.23 0.18
Table 6 shows the test accuracy results of ResNet-12, which is meta-trained and meta-tested with various data sets according to the fineness of the domains. This result indicates that BOIL can be applied to other general architectures by showing a better performance than MAML not only on standard benchmark data sets but also on cross-domain adaptation. Note that BOIL has achieved the best performance without the last skip connection in every experiment.
5.1 DISCONNECTION TRICK
Connecting the two learning schemes and ResNet's wiring structure, we propose a simple trick to eliminate the skip connection of the last residual block, which is referred to as a disconnection trick. In section 4.1 we confirmed that the model learned with BOIL applies the representation layer reuse at the low- and mid-levels of the body and representation layer change at the high-level of the body.
class
Inter-class
Beffore adiptation
Aftar adaptation
Before adoptation
After adaptation
1.00
1.00
0.75
0.75
0.2
0.5
0.00
0.00
B16041
slod2
brack3
block4
Mlocks
Medi2
block
blockd
bacc3
Mloc2
Modd3
block4
bloce3
Medid
brocc
block4
(a) BOIL w/ last skip connection.
(b) BOIL w/o last skip connection.
Figure 4: Cosine similarity of R2.
To investigate the effects of skip connections on a representation change learning scheme. we analyze the cosine similarity after every residual block in the same way as Figure 2] Figure 4a shows that ResNet with skip connections on all blocks rapidly changes not only the last block but also the other blocks. Because skip connections strengthen the gradient back proppaprpagation, the scope of representation layer change extends to the front. Therefore, to achieve both the effective representation layer reuse and the representation layer change of BOIL, we suggest a way to weaken the gradient back- proeppaapagt from the loss function by removing the skip connection of the last block. As shown in Figure 4b] with this simple disconnection trick, ResNet can improve the effectiveness of BOIL, as well as the representation layer reuse at the front blocks of the body and the representation layer change at the last block, and significantly improves the performance, described in Table 6
8

---

Published as a conference paper at ICLR 2021
We also report various analyses on ResNet-12 in the same way analyzed 4conv network and representation layer change in the last block in Appendix M and Appendixx N
6 RELATED WORK
MAML (Finn et al. 2017) is one of the most well-known algorithms in gradient- meta-learning, achieving competitive performance on few-shot learning benchmark data sets (Vinyals et al.) 2016 Ren et al. 2018: Bertinetto et al., 2018 Oreshkin et al. 2018). To tackle the task ambiguity caused by insufficient data in shot learning. numerous studies have sought to extend MAML in various ways. Some studies (Oreshkin et al. 2018 [Sun et al. 2019 Vuorio et al. 2019) have proposed feature modulators that make task specific adaptation more amenable by shifting and scaling the
representations extracted from the network body. In response to the lack of data for task-specific updates, there have also been attempts to incorporate additional parameters in a small number, rather than all model parameters (Zintgraf et al., 2018: Rusu et al., 2018; Lee & Choi 2018: Flennerhag ctt al. 2020). With a similar approach, some studies suggested a way to update only the heads in the inner loop. which has been further improved update the head using linear separable objectives.
(Raghu et al. 2020; Bertinetto et al. 2018 Lee et al.) 2019). Grant et al. (2018); Finn et al. (2018);
Yoon al. (2018); Na et al. (2019) have taken a probabilistic approach using Bayesian modeling and variational inference. In addition, Chen et al. (2020) showed that by allowing the discovered task-specific modules (i.e., a (small) subset of a network) adapt, better performance is achieved than when allowing the whole network to adapt. Notably, in few-shot image classification tasks, the authors showed that the last convolution layer (i.e., penultimate layer) is the most important. Such results were also observed in Arnold et al. (2019).
To tackle more realistic problems, few-shot learning has recently been expanding beyond the standard n-way k-shot classification. TTrintafillouu et al. (2019) constructed more scalable and realistic data set, called a meta-data set, which contains several data sets collected from different sources. In additions, Na et al. (2019) addressed n-way any-shot classification by considering the imbalanced data distribution in real-world. Furthermore, some studies (Cai & Shen 2020; CCen et al.) 2019) have recently explored few-shot learning on cross domain adaptation, which is one of the ultimate goals of meta-learning. In addition, Guo et al. (2019) suggested a new cross-domain benchmark data set for few-shot learning and showed that the current meta-learning algorithms (Finn et al., 2017 Vinyals et al.) 2016 Snell et al. 2017 Sung et al. 2018 Lee et al. 2019) underachieve compared to simple tuning on cross-domain adaptation. We demonstrated that task-specific update with representation change is efficient for a cross-domane adaptation.
7 CONCLUSION
In this study, we investigated the necessity of representation change for solving domain-agnoste tasks and proposed the BOIL algorithm, which is designed to enforce representation change by learning only the body of the model in the inner loop. We connected BOIL with preconditioning gradients and showed that the effectivenesses from a connection, such as an overfitting reduction and robustness hyperparameters change, still valid. Furthermore, we adapt BOIL to WarpGrad, demonstrating improved performance. This result decouples the benefits of representation change and preconditioning gradients. Next, we demonstrated that BOIL trains a model to follow the representation layer reuse scheme on the low- and mid-levels of the body but trains it to follow the representation layer change scheme on the high-level of the body using the cosine similarity and the CKA. We validated the BOIL algorithm on various data sets and a tross-domain adaptation using a standard 4conv network and ResNet-12. The experimental results showed a significant improvement over MAML/ANIL. particularly cross-domain adaptation, implying that representation change should be considered for adaptation to unseen tasks.
We hope that our study inspires representation change in gradient-based meta-learning approaches. Our approach is the first to study representation change and focuses on classification tasks. However, we believe that our approach is also efficient in other methods fields because our algorithm has no restrictions. Furthermore, connecting representation change to memorization overfitting addressed in (Yin et al. 2019 Rajendran et al.) 2020) will be an interesting topic.
ACKNOWLEDGMENNTS
This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No.2019-0-0007), Artificial Intelligence Graduate School Program(KAIST) and by Korea Electric Power Corporation (Grant number: R 188XA05).

---

Published as a conference paper at ICLR 2021
REFERENCES
Sébastien MR Amold, Shariq Iqbal, and Fei Sha. Decoupling adaptation from modeling with meta-optimizers for meta learning. arXiv preprint 1atvvI990016603 2019.
Luca Bertinetto, Joao Henriques, Philip HS Torr, and Andrea Vedaldi. Meta-learning with differen- tiable closed-form solvers. arXiv preprint
2018.
John Cai and Sheng Mei Shen. Cross-domain few-shot learning with meta fine-tuning. arXiv preprint arXiv:2005. 10544, 2020.
Wei-Y Yu Chen, Yen- Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Ma-Bin Huang. A closer look at few-shot classification. arXiv preprint arXiv:1904.0422: 04232, 2019.
Yutian Chen, Abram L. Friesen, Feryal Behbahani, Arnaud Doucet, David Budden, Matthew Hoffman, and Nando de Freitas. Modular meta-learning with shrinkage. Advances in Neural Information Processing Systems, 33, 2020.
Tristan Deleu, Tobias Würfl, Mandana Samiei, Joseph Paul Cohen, and Yoshua Bengio. Torchmeta: A Meta-Learning library for PyTorch, 2019. URL http::://aavvvvooae/ee http:///rraai.ooo/////99900000 Available at:
Rafael Rego Drumond, Lukas Brinkmeyer, Josif Grabocka, and Lars Schmidt-Thieme. Hidra: Head initialization across dynamic targets for robust architectures. In Proceedings of the 2020 SIAM International Conference Data Mining, pp. 397-405. SIAM, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning- Voolum 70, pp. 1 1112-111 1135. JMLR. org, 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-lcarning. In Advances in Neural Information Processing Systems, pp. 9516-9527, 2018.
Sebastian Flennerhag, Andrei Rusu, Razvan Pascanu. Francesco Visin, Hujun Yin, and Raia Hadsell. Meta-learning with warped gradient descent. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
URL wtt.thtpps////peeeeeeiieet
net bet//for forum? dieeiiiddieeeee Q1BFPB
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient- 2018.
based meta-learning as hicrarchical bayes. arXiv preprint
Yunhui Guo, Noel CF Codella, Leonid Karlinsky, John R Smith, Tajana Rosing, and Rogerio Feris. A new benchmark for evaluation cross- cross-domain few-shot learning, arXiv preprint arXiv:1912.07200. 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Nathan Hilliard, Lawrence Phillips, Scott Howland, Artëm Yankov, Courtney D Corley, and Nathan O Hodas. Few-shot learning with metric- conditional embeddings. arXiv preprint arXiv:1802.04376 2018
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4700-4708, 2017.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03166 2015.
Gregory Koch. Siamese neural networks for one-shot image recognition. 2015.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. arXiv preprint 1905:00905.0444 2019.
10

---

Published as a conference paper at ICLR 2021
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine ine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pp. 55, 2013.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Kwonjoon Lec, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10657-10665, 2019.
Yoonho Lee and Seungjin Choi. Gradient-basee meta-learning with learned layerwise metric and subspace. In International Conference on Machine Learning, pp. 2927-2936, 2018.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv: 2013.
Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großbergger Umap: Uniform manifold approximation and projection. Journal of Open Source Software, 3(29), 2018.
Donghyun Na, Hae Beom Lee, Saehoon Kim, Minseop Park, Eunho Yang, and Sung Ju Hwang. Learning balance: Bayesian veta-learning for imbalanced and out-of-distribution tasks. arXiv preprint www.iv:105..19111 2019.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv: 029999 2018.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pp. 722-729. IEEE, 2008
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In Advances in Neural Information Processing Systems, pp. 721-731, 2018.
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning feature reuse? towards understanding the effectiveness of maml. In International Conference on Learning Reprexentations, 2020. URL httphhttttssspeeeeeeeeeee
net/ wwt.m/orummmmoo 2or.grr
Janarthanan Rajendran, Alexander Irpan, and Eric Jang. Meta-learning requires meta-augmentation. Advances in Neural Information Processing Systems, 33, 2020.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. 2016.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua Tenenbaum, Hugo Larochelle, and Richard S Zemel. Meta-learning for semi-supervised few-shot classification. arXiv preprint
00676, 2018
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-255 2015.
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osin- dero, and Raia Hadsell. Meta-learning with latent embedding optimization. arXiv preprint 1r87:01807.05560. 2018
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems, pp. 4077-4087, 2017.
Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schicle. Meta-transfer learning for few-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 403-412 2019.
Flood Sung, Yongxin Yang. Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1199-1208, 2018.

---

Published as a conference paper at ICLR 2021
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, et al. Meta-dataset: dataset of datasets for learning to learn from few examples. arXiv preprint arXtv:2903.03096 2019.
Hung-Yu Yu Tseng, Hsin-Ying Ying Lee, Jia-Bin Huang, and Ming-Hsuan Yang. Cross-domain few-shot classification via learned feature-wise transformation. arXiv preprini arXiv:2001.087753 2020.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, pp. 3630-3638, 2016.
Risto Vuorio, Shao-Hua Sun, Hexiang Hu, and Joseph J Lim. Multimodal model-agnostic meta- learning via task-aware modulation. In Advances in Neural Information Processing Systems, pp. 1-12, 2019.
P. Welinder. S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongic, and P. Perona. Caltech-UCSD Birds 200. Technical Report CNS-TR-2010-001, California Institute of Technology, 2010.
Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in convolutional network. arXiv preprint 1505.00855 2015.
Mingzhang Yin, George Tucker, Mingyuan Zhou, Sergey Levine, and Chelsea Finn. Meta-learning without memorization. In International Conference on Learning Representations, 2019.
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. In Advances in Neural Information Processing Systems, pp. 7332-7342, 2018.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07/44 2016.
Matthew Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference computer vision, pp. 318-833. Springer, 2014.
Luisa M Zintgraf, Kyriacos Shiarlis, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning, arXiv preprint arXiv:1810.03642, 2018.
12

---

Published as a conference paper at ICLR 2021
A IMPLEMENTATION DETAIL
A.1 7-WAY S-OH SETTING
We experimented in the 5-way 1-shot and 5-way 5-shot, and the number of shots is marked in parentheses in the algorithm name column of all tables. During meta-training, models are inner loop updated only once, and the meta-batch size for the outer loop is set 4. During meta-testing, the number of task-specific (inner loop) updates is the same as meta-training. All the reported results are based on the model with the best validation accuracy.
A.2 MODEL IMPLEMENTATIONS
In our experiments, we employ the 4conv network and ResNet-12 for MAML/ANIL and BOIL algorithms. 4conv network has of convolution modules, and each module consists of a convolution layer with 64 filters, batch normalization (loffe & Szegedy (015). a ReLU non non-linearrity a 2 max-pool. ResNet- 12 (He et al., 2016) has the same structure with the feature extractor of TADAM (Oreshkin et al., 2018). It has four residual blocks, and each block consists of 3 modules of convolution, batch normalization, and leaky ReLU (Xu et al. 2015). At every end of each residual block, 2 X 2 max-pool is applied, and the number of convolution filters is doubled from 64 on each block. Each block also has wiring structure known as skip connection, which is a link made up of additions between the block's 's input and output feature for strengthening feature propagation.
Our proposed algorithms can be implemented by just divid- ing learning rates into for the body and the head. [Table shows the learning rates of each network and algorithm. ag and ar are the learning rates of the body and the head of the model during inner loops, and 3g and 3r, are the learning rates the body and the head of the model during outer loops.
Mocew network
Re-Ne-12
MAMI.
ANII.
BOL.
MAKIL
ANIL
BOIL
0.5
0.0
0.5
0.3
8.0
0.3
0.5
0.5
0.0
0.3
0.3
0.0
0.001
0001
00001
0.0006
0100
0.006
0.011
0.001
0,0006
Table 7: Learning rates according to the algorithms.
A.3 DATASET
We validate the BOIL and MAML/ANIL algorithms on several data sets, considering image size and fineness. Table is the summarization of the used data sets.
Table 8: Summary of data sets.
Data sets
minilmageNet
tieeedlmageNer
Cars
CUB
Source
Russakovsky et 2015
Rusakovaky 20055
et
Welnndr al. 2010
Image size
84
X4
843
Fineness
Coarse
Coarse 351
Fine
Fine 100
# meta training classes
64
98
meta-validation classes
50
meta-testing classes
20
160
49
50
Split setting
Vinyals et 2016
Ren et 2018
Tseng al.
2020
Hilliard et
2018
Data sets
FC100
CIFAR-FS
VGG-Flower
Aircrall
Source
Krizhovsky
2009
Krizhevsky al. 2009
Nilsback Zisserman
2008
Maji 2013
Imuge size Fincness
32X.22
32X322
32x32
32X32 Fine 70
Coarse
# meta-training classes
60
64
71
meta validation classes
15
meta- meta-testing classes
20
20
15
15
Split setting
Bertinetto
2018
Oreshkin al.
2018
ct
Na al. 2019
B VISUALIZATION USING UMAP
Through section 4.1 we show that conv4 in a 4conv network is the critical layer where representation layer change happens. We visualize these representations, the output of conv4, of samples from various data sets using UMAP (McInnes et al. 2018), which is an algorithm for general non-linear dimension reduction. Samples with the same line color belong to the same class. Many examples show the consistency with the intuition shown in Figuue I] When 1) similar instances with different classes sampled together and 2) representations on the meta-train data set cannot capture representations on the meta-test data set, MAMIJANII seems to be challenging to cluster samples on representation space since they are based representation reuse.
13

---

Published as a conference paper at ICLR 2021
B.1 BENCHMARK DATA SETS
betore
Alte ediptaliinn
befors ulapiaor
ccoplation
(a) MAML.
(b) ANIL.
(c) BOIL.
Figure 5: UMAP of samples from minilmageNet using the model meta-trained on minilmageNet.
Aetes sdaetation
sesptatis
(a) MAML. Figure 6: UMAP of samples from Cars using the model meta-trained on Cars.
(b) ANIL.
(c) BOIL.
B.2 CROSS-DOMAN ADAPTATION
Jefur addrraiio
actecation
(a) MAML.
(b) ANIL.
(c) BOIL.
Figure 7: UMAP of samples from mageed using the model meta-trained on minilmageNNt.
bethre atecaree
After actecation
befure odteration
petetationn
Aefure setecaatoon
zezetattion
(a) MAML.
(b) ANIL.
(c) BOIL.
Figure 8: UMAP of samples from Cars using the model meta-trained minilmageNet.
Lefor aebecation
beform adtetition
(a) MAMI..
(b) ANIL.
BOIT..
Figure 9: UMAP of samples from minilmageNet using the model meta-trained on Cars.
adopcarien
Afte adapitation
Zeftre adapoation
adaptation
beforu adapiation
acaptation
(a) MAML.
(b) ANIL.
(c) BOIL.
Figure 10: UMAP of samples from CUB using the model meta-trained on Cars.
14

---

Published as a conference paper at ICLR 2021
C RESULTS ON OTHER DATA SETS
We applied our algorithm to other data sets with image size of 32 x 32. Similar to the analyses on section 4 these data sets can be divided into two general data sets, CIFAR-FS (Bertinetto al. 2018) and FC100 (Oreshkin et al., 2018), and two specific data sets, VGG-Flower (Nilsback & Zisserman 2008 and Aireraft (Maji et al. 2013). Table 9 Table 10 and Table 11 generally show the superiority of BOIL even if image size is extremely tiny.
Table 9: Test accuracy (%) of 4conv network on benchmark dataset.
Domain
General (Connse-graine)
Specific (Fine-grained)
Dataset
CIFAR-FS
FC100
VGG-Flower
Aircraft
MAML(1)
56.55 1: 0.45
35,99
1. 0.48
60.94 1 0.35
52.27 1: 0.23
ANIL()
57.13
0.47
0.33
63.05
0.30
54.54
BOIL()
58,03
1: 0.43
38.93
0.45
65.64
1 0.26
53.37
1 0.29
MAML(5)
70.10
70.10
0.229
47.58
0.30
73.13 = 0.43
63.44
= 0.26
ANIL(S
72.07
BOIL(S)
73.61 1: 0.32
51.66
4: 0.32
79.81
4: 0.42
66.03 +: 0.14
Table 10: Test accuracy (%) of 4conv network on cross-doman adaptation.
udapiation
general to general
general to specific
specific to general
specific to specific
meta-train
FC100
CIFAR-FS
CIFAR-FS
CIFAR-FS
VGG-Flower VGG-Flower
Aircraft
VGG-Flower
mela-lest
CIFARFSS
FCTOO
VGG-Flower
Aireralt
CIFAR-ES
FC100
VGG-Flower
Airerali
M
62.58
0.35 52.81
0 19.69
27.03
54.8
32.45
37.05 0.19 25.70
ANIL(I)
63.05 + 0.39 55.36
0.47 50.61 = 0.29 27.39 + 0.09
35,90 0.20 33.84
0 0.30
31.59 + 0.22 24.55 + 0.14
BOIL(I)
60.71 + 0.43 54.18
+ 0.35
56,77 + 0.35 29.29 + 0.10
39.15 + 0.20 34.37
0.14
49.85 + 0.26
29.05 + 0.16
MAML(5
75.32
63.00
64.49 = 0.23 33.85
0.25
46.1 0.11 42,06
= 0.43
47.74 = 0.07 30.65 + 0.19
ANIL(S)
77.01
+ 0.51 63.89
+ 0.16 64.20 = 0.10 33.24
+ 0.21
44.52 + 0.25 40.51
0.26
50.28 + 0.12 28.74 + 0.23
ROIL(5)
76.33 0.30
68.55
74.93
0.11 39.96
0.11
55.48 0.21 47.17 0.38
64.68
0.23 39.81 0.25
Table 11: Way 5-Shot test accuracy (%) of ResNet- 12. The Isc means the last skip connection.
Meta
CIFAR FS
VGG-
Meta test
CIFAR-FS
FC100
VGG Flower
VGG Flower
CIFAR FS
Aireraft
MAML w/ Isc
75.30
1
0.30
0.29
42.91
0.20 28.50
0.12
MAML wlo lsc
+1.72
67.60
+ 0.34
59.20
0:200000 0.26
72.07
0.29
39.27 + 0.23
26.94
+ 0.18
ANIL w/ Ise
74.87 - 0.11
75.34
+ 0.45
63.72
+ 0.40
77.02
0.29
45.80
+ 0.32
27,24
- 0.13
ANIL woa Isc
71.39
1: 0.28
69.29
1 0.32
52.70
1 0.24
72.13
0.39
38.99
0.22
26.09
1 0.08
BOIL. w/ Isc
78.17
0.14
77.22
1: 0.45
73.90
0.38
82.00
0.17
50.91
0.35
35.54
+ 0.25
BOIL w/o
77.38
0.10
70.98
+ 0.34
73.9
+ 0.27
83,97
0.17
55.82
0.44
37.74
+ 0.21
D RESULTS UNDER THE ORIGINAL HYPERPARAMETERS
We also evaluate algorithm in the original setting (50 times smaller inner learning rate than ours) and confirm that BOIL is more robust to the change of hyperparameters than MAML. Such characteristic is investigated in Lee & Choi (2018). Table 13 shows the test accuracy of BOIL and MAML/ANIL with the same hyperparameters optimized for MAML, and Figure 11 and Table 12 describe it according to the number of adaptation(s). It is observed that BOIL the best or near-best, although the hyperparameters are not optimized for BOIL.. Moreover, BOIL. BOII.apiddy adapts and achieves considerable performance through just one adaptation.
Figure 11: Accuracy on minilmageNet according to the number of adaptation(s).
Table 12: Test accuracy (%) according to the number of adaptation(s). Training and testing are minilmageNet.
Adaguarion -
MAML(U)
32.01 0.24
36
15.51
0.18
16
46.2
0.19
16
17
1
0.18
16
0
46.21
0.66
46
0.18
ANIL())
BOIL(1)
45.79
0.45
47.15
0.30
46
0.34
47.61
0.34
47.67
0.31
47.70
0.32
47.70
0.33
47.71
0.3
47.74
0.32
47.76
0.31
MAML(5)
20.02
= 0.00
20.15
+
0.02
0.34
63.7
2
0.34
64.4
4
033
6955
0333
64.64
+
0.31
64.72
033
64.7
=
035
6433
0.40
ANIL(5)
20.00
2 0.00
24.52
19
51.59
0.17
58.84
1
0.46
62
06
0.37
62.34
0.36
62.45
0.38
62.51
0.37
62.55
0.8
62.59
0.39
BOIL(5)
58.15
0.33
63.56
0.30
64.11
64.27
64
0.30
64.
64.38
0.28
64.40
15

---

Published as a conference paper at ICLR 2021
Table 13: Test accuracy (%) under the same architecture, learning rate, and the number of inner updates with (Finn et al.
2017
Raghu et al. 2020).
munlmageNett
Mota-test
miniilaggoNtt
ueredImageNet
Cars
Cars
minilnageNet
CUB
(
46.25
ANIL())
0.22
50.04 + 0.13
32.87 0.39
45,31 = 0.27
29.12
30.39 = 0.21
BOIL(I)
47.76 0.31
51.35 0.18
34,89
50.54 0.41
32.40 0.19
32.99
0.29
5AML(5)
ANTL(S)
62.59 0.39
65.55 +: 0.16
45.44 0.11
62.67 + 0.25
36.89 0.16
40.38
0.19
BOIL(5)
64.40 0.28
65.81
0.26
48.39 + 0.25
68.56 = 0.34
43.34
0.11
E OVERFITTING ISSUE
We employ networks with various sizes of filters, 32, 64, and 128. The best validation scores of each model are 64.01, 66.72, and 69. 23, and these results mean that with BOIL, the more extensive network yields higher accuracy without overfitting.
0.9
0.7
WW
0.6
filter32 train
fiiter32 valid
0.5
filter64_ped train
filter64, _alid
filter128 train
filter128 28, valid
Figure 12: Training/Validation accuracy curve minilmageNNet according to filters in BOIL.
F WARPGRAD AND BOIL-WARPGRAD
F.1 IMPLEMENTATION DETAIL
We follow the default setting the public code except for meta train steps and the number filters, following Flennerhag et al. (2020). We change meta train steps to 100 and the number of filters to 128. The task is (Oway-Sshotte 100 samples used for task task-speeific updates, but the number of samples per class is not the same. Furthermore, this task supports the superiority of BOIL in long-term adaptations.
expectation) on Omniglot. Here, "in expectation" means that
F.2 ARCHITECTURE
Here are the architectures WarpGrad and BOIL-WarpGrad. The WarpGrad w/o last warp head model is the default one in the original code.
WarpGead
Wopp-WWapeea warp com.
suep
BOLL Carpp worp
Tallocome
HIII HHH
Warp-hoed
Figure 13: Architectures of WarpGrad and BOIL-WarpGrad.
16

---

Published as a conference paper at ICLR 2021
G
GRADIENT NORM
MAMI
DOIL
conv1 WW
conv2_
conv3_W W
conv4 WW
head W
Figure 14: Gradient norm.
We calculated the norm of gradients caused by an inner loop according to the algorithm. Although the norm gradients on the head BOIL is not really zero, we marked because the learning rate on the head is zero. The norm of gradients of biases is negligible (about 10-*), and thus it was omitted. MAML/ANIL has an extremely small norm or no norms on all convolutional layers. implies that representations little implies that representations change significantly. In addition, from the analysis of cosine similarity and CKA, mentioned that BOIL enjoys representation layer body. Nevertheless, Figure 14 shows that the amount of representation layer change a low- and mid-level in BOIL is larger than that in MAML/ANIL.
change. On the other hand, BOIL has large norm the conv4 layer. It
in a low- and mid-level of
H
COSINE SIMILARITIES INCLUDING HEAD
Figure 13 an extended version Figure
Before adaptation
Aftor adastation
Bufore adaptation
After adaptation
Befsre adbstatio
After adoptation
(107) C312 IINO canee hadd
(312 conid cand tohel herd
(097 SERG CERO canet Tead cand cand carnd conet held
(IV CER2 CRR/ caTV head
ca9 canid cand conet his
(a) MAML.
(b) ANIL.
(c) BOIL.
Figure 15: Cosine similarity of 4conv network including head.
I REPRESENTATION CHANGE IN BOIL ON CARS
This section describes representation change in BOIL on Cars. The structure this section the same that of (sectionn 4
Istra-cles
Deltre adaptaation
After septattoo
Delcre adagiation
Aftee adapkation
Selize adagtation
After adaptation
149/1 0997
cald
coeal
orel coee
0P9
cald
con4l
C9t Gred coeat
cand
add 0eet colel
coel coet
(a) MAML.
(b) ANIL
(c) BOIL.
Figure 16: Cosine similarity of 4conv network on Cars.
17

---

Published as a conference paper at ICLR 2021
CKA
1.0
0.4
0.2
MAML
convt
CnnV2
conv3
conve
hesd
Figure 17: CKA of 4conv Cars.
Table 14: Test accuracy (%) of 4conv network according to the head's existence before/after an adaptation.
Cars
meta-test
Cars
CUB
head
w/ head
w/o head (Nes-ie)s)ng)
W/ head
w/o head (Nll-lesting)
adaptation
before
after
before
alter
before
after
before
after
MAML(1)
20.03 + 0.25 45.27
= 0.26 47.87
# 0.18 47.23
= 0.24
20.01
= 0.08
29.64
31,01
+ 0.26 31.15
= 0.23
ANIL(1))
20.01 + 0.18
46.81
0.24
49.45
+ 0.18 49.45
+ 0.18
20.02
0.08
28.32 0.32
29.72
0.27 29.72
0.27
BOIL(1)
1: 0.19
56.82
0.21 25,46
1:
52.36
0.13
19.96
0.12 34.79
22,93
1
34.51
0,21
(AML(5)
20.11 1 0.16 53.23
0.26 59.67
0.22
59.38
0.23
20.00
0.22
32.18
0.13
36.
0.24 36.61
0.19
ANIL(5)
20.09 1 0.17 61.95
= 0.38 67.03
+ 0.36 67.03
- 0.36
19.99 = 0.18 37.99
= 0.15
43.27
+ 0.31 43.27
0.31
BOIL(5)
20.04 + 0.08
75.18
= 0.21
36.65
+ 0.11
71.52
= 0.27
20.02 = 0.05
45.91
= 0.28
29.04 + 0.18
47.02
0.25
J OUTPUT LAYER OF MAML/ANIL AND PENULTIMATE LAYER OF BOIL
To investigate the role of the penultimate layer (i.e., the last convolutional layer) of BOIL, we evaluated MAML/ANIL and BOIL through NIL-testing conv3 in the same way with NIL-testing on conv4 Table 5). except for the position of representations. Table 15 shows that the input of the penultimate layer (i.e., features after conv3) cannot be simply classified (i.e., the desired performance cannot be achieved), and these representation capacities are similar for all MAML, ANIL, and BOIL. Therefore, It is thought that MAMI/ANIL and BOIL. acts similarly until conv3 and BOIL is not simply the shifted version of MAMI/ANII.
Table 15: Test accuracy (%) of NIL-testing on conv3 and conv4 of 4conv network before/after an adaptation.
meta-train
minilmageNet
meta-test
minilmageNet
Cars
head
-el-testing on conv3
NlI-testing on conv4
NII-testing on conv.
-II-testing on conv4
adaptation
after
before
before
after
before
MAML(1)
32.56 0.14 1..73+0.11 0.14
48.28
0.20 47.87
0.14
0.09 31.03
34.47
0.19 34.36 0.16
ANIL(1)
34.34 + 0.10 34.34
0.10
48.86
+ 0.12 48.86
+ 0.12
31,09 + 0.08
31.09
0.08
35.48
+ 0.24 35.48
0.24
BOIL(1)
30.65 1 0.07 31.09
1 0.12
24.07
1 0.19 46.73
0.17
27.53 1 0.16 27,80
1::00-0100
0.15
23.30
0.15 34.07
0.32
MAML(5)
51.03 + 0.08 53.29
0.10
61.61
+ 0.39 64.47
+ 0.39
44.54 0.21 45.19
0,21
47.66
+ 0.28 47.53
= 0.28
ANIL(5)
55.55
- 0.14 55.55
0.14
66.11
+ 0.51
66.11
- 0.51
45.81 - 0.14
45.81
0.14
49.62
: 0.20 49.62 0.20
B5IL(5)
49.42 + 0.12 50.12
32.03
0.16
64.61
0.27
43.93
0.32
44.52
30.33
0.18 50.40
0.30
Furthermore, the input of the output layer before adaptation (i.e., features after the penultimate layer) is enough achieve the desired performance in MAML/ANIL in advance. From this result, believe the head layer's ' role of MAML/ANIL to draw a simple decision boundary for the high-level features represented by the output of the last convolutional layer. However, the penultimate layer of BOIL acts as non- non-linear transformation so that the fixed output head layer can effectively conduct a classifier role.
18

---

Published as a conference paper at ICLR 2021
K ABLATION STUDY ON THE LEARNING LAYER
In this section, we investigate whether training any representation layer during inner updates is better than training the output layer during inner updates by learning only one convolutional layer. Figure 18 shows the test accuracy according to a single learning layer in the body. For instance, convi plotted in red color indicates that the algorithm updates only the conv| layer during inner updates. In contrast, convl plotted in blue color is the case that the algorithm updates the convl layer and the head layer during inner updates.
Acouracy
Accuracy
75
65
70
35
50
ANIL
ANIL
45
BOIL
BOIL
Single Layer- Hese Adaptation
Single Head Adaptation
Single Layer Adaptation
Single Layer Adaptation
conu1
coru2
comv3
convil
conv2
conc's
convi
(a) minilmageNet.
(b) Cars.
Figure 18: Test accuracy according to the learning layer in the body.
On both minilmageNet and Cars, it is observed that training a higher-level representation layer (conv3, conv4) without updating the head (red line) performs better than training any single conv layer with the output layer (blue line). We also observe that learning only a lower-level representation layer (convl, 1, conv2) can significantly decrease accuracy. The results reassure that representation layer change in higher-level layers of the body boosts the performance discussed by the layer-wise analyses in section 4 However, training only lower-level layer behaves badly since lower-level layers retain general representations (a related discussion is in Section 4.1 e.g., the third key observation).
Table 16: Test accuracy (%) of 4conv network according to the learning layer(s). Standard deviation omitted.
Learning layer
layer
2 layers
layers
layers
all
convl
conv2
conv3
conv4
head
Algorithm
ANIL
BOIL
MAML
mini MetgeNet
54.74
41.07
67.07
66.19
63.04
60.79
67.44
86
61.24
67.18
67.
62.99
66.45
61.11
61.75
fieredImageNet
52.78
61.76 69.20
69.39
65.82
61.01
67.09
70.34
64.92
68.72
69.51
64.72
69.37
64.69
64.70
Cars
52.55 70.08 75.90
73.56
61.95
68.70
78.21
72.99
61.46
73.59
74.80
64.58
75.18
63.97
53.23
CUB
66.22 72.40 80.52 77.25 70.93
73.48
80.42
77.61
77.35
79.19
76.62
77.40
75.96
71.24
69.66
CIFAR-FS
71.58 70.47 71.01 70.88
69.87
72.15
71.85
74.43
71.43
72.93 75.56
72.07
73.61
71.73
70.10
FC100
48.03 48.04 48.97 47.93
45.65
47.71
48.23
53.86
48.78
52.95
53.11
48.25
51.66
47.59
47.58
VGG-Flower
77.96
76.84 76.02
77.10
72.07
74.69
76.10
81.63
74.73
83.61
82.46
77.74
79.81
77.72
75.13
Aircraft
65.01 61.63 64.91 65.91 63.21
62.93
63.37
66.62 62.71 66.33 67.12 63.39
66.03
62.88
63.44
We expand this ablation study to training multiple consecutive layers with and without the head. The results are reported in Table and Table 17 In Table 16 we consistently observe that learning with the head is far from the best accuracy. All the combinations having nice performances do not train the head in the inner loop update. We also find several settings skipping the lower-level layers in the inner loop that perform slightly better than BOIL. We believe each neural network architecture and data set pair has its own best layer combination. When it is allowed to search for the best combination using huge computing power, we further improve BOIL. However, the most important design policy is that the inner loop update should freeze the head and encourage to learn higher-level representation features but to reuse lower-level representation features. BOII. follows the design rule by simply
19

---

Published as a conference paper at ICLR 2021
Table 17: Test accuracy (%) of ResNet- 12 without last skip connection according the learning block(s). Standard deviation is omitted.
Learning block
block
blocks
blocks
blocks
all
block1
block2
block3
block4
head
Algorithm
ANIL
BOIL
MAML
minilmageNet
19.94 64.20
69.95
70.52
67.
69.19
67.75
70.12
69.
69
71.30
66.44
67.87
RictedImageNet
20.10
47.64 68.57
70.41
72.22
55.22
69.69
70.11
72.38
67.6
69.61
71.67
73.44
71.02
71.25
Cars
19.98
55.86 74.71
74.45
75.32
65.30
70.41
74.06
69.51
68.
16
58.43
69.78
83.99
71.75
73.63
CUB
20.02 74.84 79.26
81.58
74.66
75.07
80.12
82.09
74.24
80.26
82.75
74.94
83.22
75.66
76.23
CIFAR-FS
19.98 69.90 77.65
78.39
72.47
69.65
78.06
79.83
72.38
77.31
79.15 71.22
78.63
71.83
71.79
FC100
20.15 48.32 50.86
49.60
45.61
47,44
51.75 51.82 46.93 50.72 50.55
45.21
49.87
46.29
44,90
VGG-Flower
19.98 80.32 84.68
82.22
73.77
79,80
85.13
80.75
72.14
83.53
82.69
71.93
82.17
72.00
72.43
Aircraft
20.06 71.48 76.97
77.22
78.62
72.17
78.47
77.61
79.51
76.75
76.89
78.16
78.85
78.79
77.15
freezing the head in the inner loop that is already almost the best approach in most cases. In Table 17 there are the cases where learning a classifier leads to performance improvement. We thought that this is because the ablation study on ResNet-12 is done at the block-level. More precisely, one block includes many layers and this issue discussed in Appendix
L ADDITIONAL CONSIDERATIONS OF THE HEAD OF BOIL
We additionally discuss what the ideal meta- intta-inttialization is. Because the few-shot classification tasks are constructed with sampled classes each time, every task consists of different classes. Since the class indices are randomly assigned at the beginning of each task learning, the meta-initialized parameters cannot contain any prior information the class indices, For instance, is not allowed that the meta-initialized parameters encode class similarities between class and class j. Any biased initial guess could hinder the task learning. The meta-initialized parameters should be in in-between (local) optimal points of tasks as depicted in Figure so that the network can adapt to each task with few task-specti specitic updates
loss
07,
0
Figure 19: initialization.
Ideal meta-
0.8
0.8
0.7
0.7
0.6
0.6
0.5
0.5
10000
20000
30000
10000
20000
30000
10000
20000
30000
10000
20000
30000
(a) Comparison with centering algorithm.
(b) Comparison with fix algorithm.
Figure 20: Valid accuracy curves of (a) centering algorithm and (b) fix algorithm on Cars.
When the head parameters 0n =
nn E Rund have orthonormal rows (i.e..I(On.sll = for all i and On, = 0 for all i * j), the meta-initialized model can have the unbiased classifier. Here, T denotes the transpose of and || ||a denotes the Euclidean norm. With the orthonormal rows, therefore, each logit value On.,, fe, (xx) can be controlled independently of other logit values. Recall that the softmax probability pj for class j of sample æ is computed as follows:
On (z)
1
P3
(4)
Di= On. (o)
E (Pa,, - 0b.)) " f0, (x)
'The similar consideration is discussed
Drumond al. 2020
20

---

Published as a conference paper at ICLR 2021
In Equation 4 indeed, the softmax probability only depends on the differences of the rows of the head parameters 0g, -h.j. Adding a vector to all the rows (i.e., 8,, + 0/,5 + c for all i) does expect the same nice meta-initialized model, when a parallel shift of the rows of the head parameters can make orthonormal rows. To support this experimentally, we design the centering algorithm that operates a parallel shift of 0 by subtracting the average of the row vectors of 02 after every outer update on both MAML and BOIL, i.c., [0x,1 - .h., On.n 0n] where On 1
not change the softmax vector. So,
n 2 On Figure 20a shows that this parallel shift operations does not affect the performance of two algorithms on Cars.
Next, we investigate the cosine similarity between On.5 0 and On.j 0n. for all different i, j. and fixed k. From the training procedures of MAML and BOIL, it is observed that the average of cosine similarities between the two gaps keeps near 0.5 during meta-training (Figure 21). Note that 0.5 is the cosine similarity between 0z.i On and 0h. 0b. when 0n.i fr. and 0r.k T are orthonormal. From the results, we evidence that the orthonormality of 0g important for the mnta-initialization and meta learning algorithms naturally keep the orthonormality.
1.0
MAKL
BOIL
0.6
0.2
10000
20000
Figure 21: Average of cosine similarities between gaps.
From the above observation, we design the fix algorithm that fixes O, to be orthonormal for the meta-initialized model. Namely, MAML-fix updates O, in inner loops only, and BOIL-fix does not update 0/. The fix algorithm can be easily implemented by initializing 9, to be orthonormal through the Bram-Schmide method from random matrix and setting the learning rate for the head of the model during the outer loop to zero.
Figure 20bcepiccss the valid accuracy curves of the fix algorithm on Cars. The experiments substantiate that orthonormal rows of f, are important and that BOIL improves the performance. (1) Comparing MAML MAML-fix (the left panel of Figure 20b), MAML-fix outperforms MAML. It means that the outer loop calculated through the task-specific head following MAML is detrimental because the outer loop adds unnecessary task-specific information the model. (2) Comparing vanilla models to fix models (both panels of Figure 20b), fixed meta- mntaa-nitializze head with orthonormality is less over-fitted. (3) Comparing BOIL BOIL-fix (the right panel Figure 206), although BOIL-fix can achieve almost the same performance with BOIL with sufficient iterations, BOIL. converges faster to a better local optimum. This because O, is trained so that the inner loop can easily adapt fo, (1)) to each class.
M REPRESENTATION CHANGE IN RESNET-12 12
CKA
1.0
Figure 22 shows the CKA of ResNet according to the algorithm. Like the 4conv network, MAML/ANIL algorithms change the values only in the logit space, i.e., the space after head, regardless of the last skip connection. However, the BOIL algorithm changes the values in the representation spaces. By disconnecting the last skip connection, repre- sentation layer change is concentrated on the high-level representation space, i.e., the CKA of BOIL w/o LSC is smaller than that of BOIL w/ LSC after block4, Table 18 shows empirical results of ResNet-12.
0.0
MANL wis LSC
ANL with LSC
0.0
POL, wib LSC
boccl
Beeck
Bock3
101d
Figure 22: CKA of ResNet- 12 on minilmageNet.
Table 18: 5-Way 5-Shot test accuracy (%) of ResNet-12 meta-trained on minilmageNet according to the head's existence before/after an adaptation.
metaa train
mini Imageet
meta-lest test
minilmageNet
Cars
head
w/ head
w/o head (NIL-testing)
w/ head
w/o head (NIL-testing)
adaplation
before
hefore
after
before
before
after
MAML w/ LSC
20.02
0.21 68.51
1 0.39
70.44
1 0.30 70.37 0.32
20.06
1 0.07
43,46
=
0.15
46.08 = 0.22 46.05 1 0.19
MAML w/o LSC
20.04
0.36 67.87
0.22
69.35
0.15 69.28
0.14
19.98
4 0.16
41.40
0.11
43.55
0.17
43.56
0.16
ANIL LSC
19.85 0.19 68.54
0.34
70.31
70.31
19.99
0.15
45.13
47.16
47.16
0.20
ANIL w/o LSC
19.97 0.21 67.20
+ 0.13
68.47
+ 0.21
68.27
20.03
+ 0.23
43.46
0.18
45.16
0.11
45.16
BOIL w/ LSC
20.01
0.18 70.50
+ 0.28
44.65
+ 0.49
70.34
0.31
20.02
+ 0.08
49.69
+ 0.17
38.73
+ 0.16
49.54
0.23
BOIL w/o LSC
20.00
0.00 71.30
0.28
40.04
+: 0.33
71.18
0.29
20.00
0.00
49.71
0.28
32
0.29
51.41
0.32
21

---

Published as a conference paper at ICLR 2021
Furthermore, we identified representation change of ResNet- meta- eeta-trained on Cars in BOIL.
Latra class
Inter dlass
adeptalion
adaptation
1.00
1.00
1.00
0.75
0.75
0.75
0.50
0.50
0.50
0.23
0.25
0.2
0.00
(a) MAML w/ LSC.
(b) ANIL w/ LSC.
(c) BOIL w/ LSC.
Betore
After adaphation
Before
After adaptaticn
Betore adsptation
adappatiion
1.00
1.00
1.00
0.75
0.75
0.5
0.50
badd2 Medk Mocel blold brek2 neek
beecd ock2 Medk bicet cledd biddd beokd tbas
Moodt bied2 Moda Mecet tbed bredi eek2 Madk
(d) MAML w/o LSC.
(e) ANIL w/o LSC.
(l) BOIL w/o LSC.
Figure 23: Cosine similarity of ResNet-12 on Cars.
CKA
1.0
MAML w/ LSC
ANIL
DOIL
0.2
MAML win LSC
ANIL w/o
0.0
BOIL w/o LSC
black1
block2
block3
block4
head
Figure 24: CKA of ResNet- on Cars.
Table 19: 5-Way 5-Shot test accuracy (%) of ResNet- meta-trained on Cars according to the head's existence before/after an adaptation.
ineta-train
Cars
meta Test
Cars
CUB
head
w/ head
w/o head (NIL-testing)
W/ head
w/o head NIL--testng)
adaptation
before
before
after
before
before
after
MAML w/ LSC
20.11
0.22
75.49
0.20
77.01 1 0.14 76.76
0.17
20.13
0.10
35.87 =
0.19
36.57
0.20
36.62 0.17
MAMI. w/o LSC.
20.04
0.08 73.63
0.26
75.81
0.20
75.61
+ 0.25
20.03
+: 0.14
34.77
0.26
36.26
0.21 36.04
0.27
ANIL LSC
20.06
0.35
79,45
80,88
80.88
0.19
0.14
35.09
0.19
35,80
0.16
ANIL. w/o SCC
20.20
0.33 75.32
0.15
76.90
+ 0.16
76.90
0.16
20.01
+ 0.18
36.06
0.14
37.34
-
0.13
37.34
0.13
BOIL w/ LSC
20.00
0.00 80.98
+ 0.14
40.54
1 0.11
81.11
0.24
20.00
0.00
43.84
0.21
37.27
0.12
45.44
0.23
BOIL w/o LSC
20.00
0.00 83.99
0.20
50.42
0.23
83.60
0.18
20.00
0.00
44.23
0.18
31.69
0.14
44.19
+ 0.17
N REPRESENTATION LAYER CHANGE IN THE LAST BLOCK OF RESNET- 12
In this section, we explore the representation layer reuse and representation layer change in the last block of deeper architecture network. Figure 25 and Figure 26 show the cosine similarities between representations after layers in the last block (i.e., block4) on minilmageNet and Cars. In the case of MAML/ANIL. there is little last block. contrast, in the case BOIL, the gap between intra-class similarity and inter-class similarity is enlarged through adaptation in some all layers of the last block, which indicates that representation layer reuse in low-level layers of the last block and representation layer change in high-level layers of the block mixed even in the last block.
representation layer change in all layers of the
Furthermore, is observed that representation at high-level layers the last block changes more when the last skip connection does not exist (e.g., Figure 25f and Figure 26f than when the last
22

---

Published as a conference paper at ICLR 2021
skip connection exists (e.g.. Figure 25e and Figure 26e). This result confirms that the disconnection trick strengthens representation layer change at high-level layers by not directly propagating general representations from prior blocks.
Intra-clen
Inter-diea
Blfore adaplation
After adaptation
1.0
08
0.2
rd.
In2
resitall
(a) MAMI w/ LSC.
Before adaptation
After adaptation
0.0
0.2
mM07
1WL3
1601
14.33
(b) MAML w/o LSC.
adaptation
adaptation
0.8
0.2
02162
bn2
18h.2
cond
reccual
bn2
1012
mid
(c) ANIL w/ LSC.
Before adaptation
After adaptation
1.0
0.8
0.2
0.0
1002
0-2
16..
1661
C0012
10..2
18..3
(d) ANIL w/o LSC.
adaptation
adaptation
0.8
0.0
63192
1d.1
bnz
1d.22
convd
bis rasical rold mexcoo
Canit
202 rok2
c0162
1012
Ind
rosiaal rdud
(e) BOIL w/ LSC.
adaptation
adaptation
0.0
02
m702
1e0.3
n60.
conid
160.2
con72
140.3
( BOIL w/o LSC.
Figure 25: Cosine similarity block4 (the last block) ResNet- 12 on minilmagenet.
23

---

Published as a conference paper at ICLR 2021
Intra-clae
Inter-ciass
Before adaptation
After adaptation
10
0.4
cond
10k2
maapcol
(a) MAML w/ LSC.
Before adaptation
After adaptation
1.0
0.4
maapued
(b) MAML w/o LSC.
Before adaptation
After adaptation
1.0
0.4
cond
reld
r932 03162
maapcol
(c) ANIL w/ LSC.
Before adaptation
After adaptation
1.0
0.4
1012
10LL
maxpcol
(d) ANIL w/o LSC.
Before adaptation
After adaptation
1.0
0.4
0.2
maxpoel
BOIL LSC.
Before adaptation
After adaptation
1.0
0.4
0.2
BOII w/o LSC.
Figure 26: Cosine similarity in block4 (the last block) of ResNet- 12 on Cars.
24


Reviews:
[{"review_id": "tyR0Ka-kRrb", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary:\nThis paper proposes a variant of MAML, called body only in the inner loop (BOIL), in which the output layer is not trained (learning rate is set to 0, always) but gradients are still backpropagated to the remaining layers, which are trained as in MAML. In essence, this is the inverse of the ANIL method of Raghu et al., which showed that MAML mainly adapts only the output layer for few-shot image classification. This paper shows that never training the output layer of the network (simply leaving it at its randomly-initialized values) can provide much better performance across a number of few-shot image classification tasks, including those requiring domain transfer.\n\nPros:\n- Consistently strong performance across a number of tasks\n- Simple method\n- Many experiments showing similar benefits\n- Side studies and ablations to try to better understand the reason for the improved performance provided by BOIL\n\nCons:\n- I am not convinced about the explanation of representation change versus representation reuse. Clearly, a lot of representation reuse is still occurring; i.e., all but one layer. The only difference is that instead of the final layer adapting and the remaining layers staying fixed, as in MAML, the final layer is fixed and the penultimate layer (now the final adapting layer) adapts. The preceding layers remain essentially fixed however (i.e., are reused). \n- The experiments are somewhat unfair. Instead of performing a hyperparameter search for each method to choose the best settings for each, a single setting is used for all methods. While there is a lot of overlap between the methods, it’s clear from Appendix E that MAML can perform better using those hyperparameters in some domains than is reported in the main text. While it’s likely that BOIL will still do well (as it still does better than the improved MAML scores), it would be more fair to report the higher score for MAML. In particular, it’s likely that the much lower learning rate (as used in Appendix E) prevents the severe overfitting commonly observed in MAML.\n \n#### Decision:\nOverall, I am on the fence about this paper. The results are strong and will be interesting to the community; however, the explanation does not seem valid to me and the evaluation could be more fair. I thus recommend a weak accept as readers will be interested in the results and can draw their own conclusions about the reasons for them.\n\n#### Questions:\n\n1. Why does adapting only the penultimate layer count as representation change but adapting only the output layer is representation reuse? This distinction seems arbitrary. Clearly, there is a lot of benefit to adapting the penultimate layer as shown by the experiments but this seems like an identical amount of representation reuse as MAML with a network that is missing a layer and instead has a random transform appended to the end. How is this different from that?\n\n2. What does Figure 2 look like if the cosine similarity of the output layer is plotted in each subplot along with that of the conv layers? Does BOIL then become a shifted version of MAML, as Figure 3 indicates it might?\n\n#### Comments:\n- A recent work [1] also examines fixing some layers, including the output layer, to obtain better performance than allowing all layers to adapt. It should be cited and discussed in the related work as well.\n- Please define $f$ in equation (1).\n- The final sentence of Section 3 is unintelligible to me. Please clarify it.\n- In Figure 2, the subfigures are far too small to read. Further, the subtitles say “adatation” instead of “adaptation”.\n- In general, the writing could use more editing.\n- The whitespace around the titles is overly compressed. Please move some text to the Appendix instead of cramming as much as possible into the main body.\n\n[1] Modular Meta-Learning with Shrinkage. Chen, Friesen, Behbani, Doucet, Budden, Hoffman, and de Freitas. \n\n\n***********\n\nAfter the discussion period, I have increased my score to 7 as the authors have provided a strong set of ablations and changes to address my concerns and those of the other reviewers.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "kjC6jiLe54V", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper performs an investigation (with an associated new algorithm, BOIL), into the relative importance of representation change vs representation reuse in few-shot learning with MAML. It also compares to related prior work, 'Rapid Learning or Feature Reuse, (Raghu et al 2020)', that explored this question, finding representation reuse to be the key component of MAML, and their associated algorithm, ANIL, which only updates the network head (not the body) in the inner loop of training.\n\nThe paper studies the important role played by representation change when performing few-shot learning across domains, and to explore this, proposes an algorithm BOIL (Body Only in the Inner Loop), which, in contrast, only updates the network *body* in the inner loop, and keeps the network head frozen. The results show that BOIL matches performance of MAML/ANIL in single domain few-shot learning (mini ImageNet, tiered MiniImagenet), and provides performance improvements in fine-grained and cross-domain few shot learning (Cars, CUB, cross-domain between these datasets). \n\nWhile the results are interesting, I have a key concern on how BOIL is working and some of the analysis results:\n\nBy freezing the head of the network during inner loop training, the head effectively acts as a fixed random projection layer, and  it is very possible that the penultimate layer now acts like a pseudo-head. This is what seems to be indicated by Figure 3, where the three first layers comprising of the body have extremely high representation reuse before/after adaptation, and the fourth layer, becomes much less similar, possibly acting as a head. It is important to determine whether this is what is happening, because it affects how much BOIL is actually reliant on representation change.\n\nBefore recommending accept, I would like to see more CKA comparisons (and any other representation analysis) of the BOIL representations across the different architectures/datasets, to confirm that BOIL is indeed working through representation change. \n\nMinor Comments:\n--- It would be helpful to know how the few shot learning tasks are setup on Cars and CUB\n--- The key detail of which representations are being compared in Figure 2 (cosine similarity plot) is very unclear. Also don't understand why representations before/after adaption are being plotted separately, instead of cosine similarity applied to before/after pair.\n\n--------------------------------------\n--------------------------------------\n--------------------------------------\n**After Rebuttal and Discussion**\nI appreciate the changes and additional experiments added by the authors, and am recommending accept. \n\nFor the final version, I would strongly encourage the authors to make the representation change/representation reuse argument clearer. Specifically, the authors might want to first present the BOIL algorithm, highlight key aspects of the algorithm (frozen head), present performance results, and then the ablation results in Appendix N to highlight the importance of freezing the head. After this, the paper could switch to a discussion on representation reuse/change and present the analysis results, making clear what is happening at a layer level vs at the algorithm level.\n\nRight now, I think it's still a little confusing that representation change often refers to the algorithm not the layers, and this actually reduces the impact of what is a very striking result!\n\nI hope the authors can make these changes, and with the clearer messaging, this should be a very interesting paper for the community, and provide many interesting directions for future work!\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "SNgmc2xTO2C", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nPrevious work studied MAML and showed that representation reuse is the main contributing factor in performance and not representation change. This paper first asks the question of whether representation reuse is enough for meta-learning? The paper hypothesizes and empirically evaluates the need/benefit for representation change in meta-learning tasks especially for cross-domain transfer. For this, they propose BOIL, a variant of MAML where the head of the network is not updated during inner loop updates.\n\nStrengths:\n1. The paper is very well written.\n2. They do a simple, clearly described empirical study to understand what changes in representation happen in MAML during its training/testing and how to modify the MAML to encourage representation change.\n3. The proposed change to MAML, BOIL shows improved performance in several of the benchmark datasets especially in cross-domain tasks (compared to MAML)\n4. Their overall study is beneficial for the gradient based meta-learning models similar to MAML to better understand what is happening and directions to modification to training that could lead to potential improvement in generalization performance. \n\nWeakness/Comments:\n1. While the paper focuses on MAML, it is not clear what and how their findings and design modifications apply for other meta-learning methods such as CNP and also to gradient based meta-learning in Reinforcement Learning. Some discussion on the scope of the finidings/methods would be very helpful.\n2. I am concerned if what is happening in BOIL is that the problem of not have representation change is just being passed on to the body. i.e the top layer of body is like the head which is reusing and the everything below is changing. i.e I am wondering for deep networks the problem is just passed on to the lower part and not really handled. At some point we probably want the layers to change less and be reused. Authors comments on this would be appreciated.\n3. Even in a perfect representation change setting, there will be limitation of generalization to test time tasks depending on the distribution of the training tasks. One would hope that this covers a much larger set of domain transfers than with the representation reuse case. Some brief discussion on this would be good. i.e BOIL may not be able to transfer to any cross domain, if not, what class of domains can it successfully transfer to?\n4. My guess is that the speed of adaptation might decrease in BOIL compared to MAML. This might not be a problem in SL, but might be of interest in RL. What do you think?\n5. Do we lose anything by not allowing the head to change during inner loop training? Would it be possible to alternate between lr of inner loop being zero and non-zero for the head?\n6. One can imagine if we had a lot of training tasks with a wide distribution and a model that cannot memorize all that the training of , and good optimizer, MAML would be forced to perform representation change as the training loss cannot be successfully reduced using just training reuse as the the model's capacity is limited. Would this be an alternative at the cost of more data? What are your thoughts?\n\nOther Comments:\n1. I think if we can achieve representation change, then it can also help make MAML less prone to memorization overfitting (\"Meta-Learning without memorization, ICLR 2020, Meta-Learning requires meta-augmentation, NeurIPS 2020\"). That might also a be something to explore. \n\nQuestions to authors:\nPlease reply to the above comments/weakness mentioned. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a new algorithm, BOIL, on the importance of representation change vs reuse in MAML. All reviewers found the paper insightful, with some proposing a few changes to make the paper even stronger. Like them, I recommend accepting the paper.
Decision: Accept (Poster)