Towards Efficient Gradient-Based Meta-Learning in Heterogenous Environments
Authors: Thomas Goerttler
Luis Müller
Klaus Obermayer
Venue: Submitted to ICLR 2023 (2023)
Keywords: few-shot learning
heterogeneous datasets
cross-domain adaptation

Abstract: A challenging problem for machine learning is few-shot learning, as its models usually require many training samples. Since meta-learning models have strong fine-tuning capabilities for the distribution of tasks, many of them have been applied to few-shot learning. Model-agnostic meta-learning (MAML) is one of the most popular ones. Recent studies showed that MAML-trained models tend to reuse learned features and do not perform strong adaption, especially in the earlier layers. This paper presents an in-detail analysis of this phenomenon by analyzing MAML's components of different variants. Our results show an interesting relationship between the importance of fine-tuning earlier layers and the difference in the distribution between training and testing. As a result, we determine a fundamental weakness of existing MAML variants when the task distribution is heterogeneous, e.g., the numbers of classes do not match during testing and training. We propose a novel nonparametric version of MAML that overcomes these issues while still being able to perform cross-domain adaption.

Under review as a conference paper at ICLR 2023
TOWARDS EFFICIENT GRADIENT-BASED IN HETEROGENOUS ENVIRONMENTS
Anonymous authors Paper under double-bling review
ABSTRACT
A challenging problem for machine learning is few-shot learning, as traditionally, models trained with SGD require many training samples to converge. Since meta- learning models have strong fine-tuning capabilities for the distribution of tasks, many of them have been applied to few-shot learning- Model-agnostic meta- learning (MAML) is one of the most popular ones. Recent studies showed that MAML-trained models tend to reuse learned features and do not perform strong adaption, especially in the carlier layers. This paper presents in-detail analy- sis of this phenomenon by analyzing MAML's components for different variants. Our results show interesting relationship between the importance of fine-tuning earlier layers and the difference in the distribution between training and testing. As result, we determine a fundamental weakness of existing MAML variants when the task distribution is heterogeneous, e.g., the numbers of classes or the do- main do not match during testing and training. We propose a novel nonparametric version of MAML that overcomes these issues while still being able to perform cross-domain adaption.
INTRODUCTION
Learning tasks from only few observations is known as few-shot learning and of major interest in the machine learning community (Finn et al. 2017: Vinyals et al. 2016 Snell et al.) 2017 Cai & Shen 22200 Tseng et al., 2020). Usually, problem is solved by minimizing the empirical risk with many training samples in many iterations. However, humans learn new tasks very quickly by using knowledge obtained in their lives a priori (Salakhutdinov et al.] 2012). Meta-learning is motivated by how humans learn, where the goal is learning how to learn and common approach solve few-shot learning problems due to its ability to efficiently leverage information from many tasks.
Model-Agnostic Meta-Learnin (MAML) has been one of the most successful meta-learning algo- rithms for few-shot learning in recent years (Finn et al.] 2017). In MAML, the network is meta- an
task. Its formulation of the meta-lcarning objective inspired a plethora of research (Yoon et al., 2018 Li et al.) 2017: Vuorio 2019 Finn al. 2018), the extent that MAML exists both concrete meta-learning algorithm but also as a paradigm that influences meta-learning methods to this day.
optimized for fast gradient-descent based fine-tuning
Previous work has discussed whether MAML actually allows rapid fine-tuning or simply leverages its retreetpprssnnttioonn effectively (called featur reuse). Raghu et al. (2020) found out that freezing the earlier layers of network during fine-tuning tinettnnig improves the performance, meaning that fine-tuning of the network body is not the major factor contributing to its few-shot capabilities, indicating feature reuse. [Oh et al. (2021) discovered that in the case of cross-domain adaptation, a change the earlier layers is beneficial and proposed to fix the network head instead to enforce earlier weight change, a method they call body only inner loop (BOIL). However, as we will argue in Section 3 its fixed final layer is impractical when the numbers of classes differ across tasks, which is considerable limitation in real-world scenarios.
In this paper, we develop novel technique called NP-MAML, which has nonparametric head but still trainable via gradients. Similar to BOIL, NP-MAML enforces changes in earlier layers to solve cross-domain tasks. In addition, is flexible the heterogeneous task distribution. We

---

Under review as a conference paper at ICLR 2023
compare the performance and representation change of these approaches under different challenges: cross-domain adaption ponents of the network and want to specify their respective role with regard fine-tuning and task adaptation.
and task dimensionality. We further deliver an analysis of the different com-
META- FOR FEW-SHE SHOT LEARNING
In few-shot learning, a dataset consists tasks coming from a task distribution. Each task contains of support set S with labels samples and query set Q where predictions have be made on the samples (Vinyals et al. 2016). A typical support set consists of examples for each of classes, wherefore a problem is usually described N N-way-E Kshoo classification problem.
Next to non-episodic approaches (Gidaris & Komodakis 2018 Qi et al. 2018 Chen et al.
2019
many meta-learning methods have been applied to few-shot learning problems Vinyals al. 2016
Li et al,, 2017| Yoon et al.) 2018) Snell et al. 2017). They are particularly useful as they can learn configuration from which easy solve various tasks. The configuration has similar features or recurring patterns shared across the tasks and allowed to be transferred to novel unseen tasks.
2.1 OPTIMIZATION-BASER META-LEARNING
Optimization- based meta-learning Ravi & Larochelle 2017) follows the idea of meta-learning task-specific optimizer U. that transforms some initial set of parameters into aask-paraetters 6. Although U can be chosen arbitrarily, it is typically modelled as m-step, gradient-based up- date scheme, denoted (mm) (0). While methods like Meta-LSTMs model ((m) explicitly via an LSTM (Ravi & Larochelle 2017). which iteratively transforms 0, given both loss and loss gradient, the popularity and success of MAML is due to its efficient and model-agnosti design, allowing (mm) to take any differentiable form and rather meta-optimizing 0, the initial parameters, leading to superior performance and more flexibility.
2.2 MODEL-AGNOSTIC META-LEARNING
MAML (Finn et al. 2017) introduces couple of new interpretations of the meta-learning problem that allow its efficient and flexible training. Firstly, task data is assumed come from a lask distri- bution p(r), which in turn will allow us to form an expectation over task performance. Secondly, let L,, (x: 8*) denote the loss of a model fe on task-data x {(21, 31), .... (ZT. yr)). parameterized by 8* To stay consistent with the notation introduced hitherto. we will denote with S- the support set of task 7 and with Q, the query set of task r. Then, we can express the optimization objective of MAML as
min Errp(r) Lm((Q::
where we write (S,; to denote a m-step optimizer, transforming meta-parameters given the support S,. Intuitively. MAML improves on-task performance on average by optimizing the initial parameters of model fe and subsequently fine-tuning those parameters on the support set S,, where on-task performance measured by evaluating the fine-tuned model on the query set Q..
As p(r) is high- digh-dimensional and typically unknown, computing the actual expectation integral is feasible, which is why we define the meta-loss MAML
((0)
Σ (()) TET
[T
where T is batch of tasks, sampled from p(r). and where we replace the expectation with an empirical mean. This meta-loss is then optimized with standard gradient descent with step-size $ i.e.,
((0) (11-1) BV ou ((0(1-1)).
(3)
2

---

Under review as a conference paper at ICLR 2023
A crucial detail of MAML is that the gradient descent update in Equation 3 involves second-order derivatives if task optimizer (im) is gradient-based. The authors thus propose first-order ap- proximation to the update that performs just as well as the full-gradient version of MAML, Other approximations are proposed by (Nichol et al. 2018 Rajeswaran et al. 2019)). Other extensions of MAML stabilized and improved the method by proposing various modifications (Antoniou et al, 2019) decoupling the gradient-based adaption procedure (Rusu et al. 2019) and integrating target samples into the fine-tuning step (Antoniou & Storkey, 2019).
2.2.1 RAPID LEARNING OR FEATURE REUSE
After investigating the adaptation behavior MAML, Raghu et al. (2020) claimed that MAML tends learn nearly fixed representations in the network body rather than rapidly fine-tuning them, a phenomenon they call feature reuse. They analyzed the representation similarity of the layers before and after fine-tuning and observed hardly any changes the early layers. Their proposed technique ANIL (almost no inner loop). which freezes the network body during fine-tuning, slightly outperforms MAML on several tasks like MinilmageNet and Omniglot and is additionally faster. Oh et al. (2021) proposed BOIL (body only inner loop), complementary approach freezing only the network head during fine-tuning, showing that in the case of (cross-)domain Jdommin adaption, a fast adaptation of weights in the earlier layers, especially the penultimate layer, is not only possible but also highly beneficial to performance.
2.2.2 MODULARIZATION OF MAML
To accumulate the perspective of previous research and ours, we formulate modular interpretation of MAML. Raghu et al. (2020) and Oh al. (202I) have found out that fine-tuning different com- ponents of the architecture affect the runtime and the overall performance. They analyze the effects of MAML. by performing experiments architecture consisting of four convolutional blocks and a linear layer, which we further call conv4. We will later discuss each component of the conv4 architecture in the context of the algorithmic and architectural challenges.
few-shot image datasets such MinilmageNet with an
Let fo be model parameterized by 0. We modularize fo by partitioning the fo into sub-components. e.g., single layer, a group of layers that form one modular unit, or an entire sub-network. We call a modularization of size n of fo if and only if
g (.)),,))..... (()) 0n))
fo (. (..
(4)
and
0 ... 01
(5)
Furthermore, we denote with g*
g the components that receive task-specific fine-tuning and call g* a meta-learning configuration. Then, the meta-update according to MAML for parameter 0 becomes
0, = 0, - 3 ((Q
(6)
with
Φ (U(m) 0,
if gf E eg* otherwise,
where S is the support set and (m) is optimizer with steps. We can recover the original MAML by setting g* =g.
We study the advantages and challenges of different configurations of g* We partition the conv4 network into three components: Jearly denotes the first three convolutional blocks, Ipenuit denotes the last convolutional block (penultimate layer), and Ihead denotes the last linear layer. Previously proposed methods can be obtained by only optimizing the weights of selected sub-components (see Table in Appendix A).
3

---

Under review as a conference paper at ICLR 2023
3 CHALLENGES OF FEW-SHOT LEARNING
Standard few-shot learning benchmark datasets like Omniglot or MinilmageNet are single-modal homogenous, and balanced, whereas real-world datasets are multi-modal, heterogeneous, and im- balanced (Vuorio et al. 2019; Triantafillou al., (020). The data can come from different domains (multi-modal) and have a flexible number of shots (imbalanced) and ways (heterogeneous). We divide the challenges into two groups, distinguishing between algorithmic and architectural chal- lenges and discuss how the methods from Section 2.2.1 may deal with these challenges.
3.1 ALGORITHMIC CHALLENGE: DOMAIN SHIFT
Domain shift is straightforward yet challenging problem of cross-domain adaptation (Oh et al., 2021 Triantafillou et al. 2020 Cai & Shen 2020 Tseng et al. 2020). Therein, a model is meta- trained with task from domain A and is meta-tested with task from domain B, which has not been available during training. Oh et al. (202I) showed that BOIL outperforms MAML and ANIL in this setup, as they learn the general features for domain A and are not able to adapt these features for the different domain B, even when domain shift necessitates adaptation. Thus, our proposed technique in Section 4 aims at maintaining the cross-domain adaptation capabilities of BOIL. A more detailed discussion on domain adaption can be found in Appendix B
3.2 ARCHITECTURAL CHALLENGE: TASK DIMENSIONALITY
Under task dimensionality, we summarize the dimensionality of both the samples and the labels within a task. Whereas it is always possible to scale images to common n x m pixel grid, for the output dimensionality of task (ways), there is no trivial modification besides re-initialization. This limitation is caused by the fully-connected linear layer whose weights need fixed output size to be transferred The head of the network turning a hidden representation Pbody (z; 0) into logits and probabilities is typically expressed as
p( x:) a WT. Phody (x; 0) + b):
(8)
where x is an input sample, parameters W e HN and E R-V form the linear layer, H is the output dimension of the network body and a is the softmax operation. Since the dimension of W has to be fixed, a linear architecture is limited because it cannot adapt distributions with varying output dimensions. A parametric solution of this is proposed by Triantafillou et al. (2020) where the prototypes of the support set form the initialization for the final linear layer. Another proposed solu- tion is UnicornMAML, where the same weight is meta-learned for every neuron in the output layer and replicated in the inner loop to match the task dimensionality Ye & Chao (202)). However, both approaches explicitly construct head to the parametric paradigm. The nonparametric version we propose in the following section recovers predictions implicitly from support set representations and alleviates the need for physical linear layer altogether.
4 NONPARAMETRIC MODEL- AGNOSTIC META-LEARNING
In this section, we propose novel technique called nonparametric model-agnostic meta-learning (NP-MAML), which can deal with both algorithmic and architectural challenges of Section 3 In addition, it also is implicitly permutation invariant, quality standard MAML approaches lack, as discovered by Ye & Chao (2022).
We derive method follows. Raghu et al. 2020) proposed nonparametric head for MAML and suggested no updates in the inner loop (NII. == No Inner Loop during fine-tuning. Utilizing metric-based approaches of Vinyals et al. (2016) and Snell al. (2017) in NII., similarity metric
'architcctural challenges take the form of simple, physical limitations like mismatches of in- and output dimensions and algorithmic refers to problems with performance, lack of generalization instability of the results (such as very high variance on the classification accuracy)
lthoughh we will not study the effect of rescaling in this work, we encourage a further investigation into whether rescaling has impact on the predictive performance of meta-learning models for image processing.
4

---

Under review as a conference paper at ICLR 2023
d is applied to the representation Gbody (z) and the prototype c₁, defined as
Jbody (1),
(9)
K
ZES.
for every class label of the support set S
N
p x, S) = Σ
xkldd( (x), y/.
(10)
with k(-) as a kernel normalizing the distance measures into a probability and where again gbodt (x) denotes the representation of samples x by the network body. Coincidentally, this approach is very adaptive to the task dimensionality during testing, as predictions are formed on the dimensional- ity of the body representations. The NIL had is explicitly designed to leverage the strong meta- representations acquired during MAML training, demonstrating that the network still performs well on in-domain tasks, even when no test-time fine-tuning performed. However, it is well known in the meta-learning community that models generally benefit from an agreement train- and test- time loss Vinyals et al. (2016). To enable such conditions for a NIL-type predictor, we propose new method, nonparametric model-agnostic meta-learning (NP-MAML), which replaces the para- metric head of MAML with a nonparametric head also for training. allows updating the earlier layers during fine-tuning and can be incorporated during meta-opuimizaiinn Since the nonparamet- ric head does not require fixed output size, it can deal with architectural challenges. the same time, head predictions are formed entirely based on the body representations, enforcing a BOIL-type fine-tuning of the network body that is beneficial to the cross-domain setting.
We note that applying a NIL head at test time to model trained with BOIL is synthetically possible but leads to non-meaningful results, as the model still relies on the representations learned in the head. We verify this claim experimentally in Section 5.2 where we use NIL-testing BOIL to architecturally cope with variations in the number of ways at test time.
In NP-MAML, we use the predictive distribution of Equation I0 and extend it such that it allows gradients to flow through the fine-tuning stage:
p x.
Σ ke- ((dd(ynady (1; (r;UU();)) ca) Vi:
(11)
where d is again similarity metric, measuring the similarity between the representation Ibody (x; US: f)) of sample x by the network body and prototype c, Note that contrast to NIL, the parameters of the body representation optimizer that is similar to a contrastive loss (Andonian et al. 202T), defined as
the result of fine-tuning with U, special one-step
a U(::0) = K
V
Σ 2
--d dl d((bndy (S;; (S):),,)))
(12)
Intuitively, U fine-tunes the network such as to maximize the extra-class distances between repre- sentations of support samples, meaning that features from different classes should be less similar than features from the same class. Here, we use the same distance metric d from which we also form predictions on the query set in Equation TT We further visualize the fine-tuning effect on the net- work's features in Appendix C where we see that the fine-tuning step acts pulling force on the metric space, disentangling intra-class from extra-class features. We hypothesize that NP-MAML fine-tuning learns to condition the metric space, such as to ease classification.
Note that although the prototypes G, also depend on 8, we stop the gradient flow for the prototype when calculating the meta- gradient, achieving similar runtime performance as the other methods. Although this is not first-order approximation of the meta-gradient, this simplification is based on previous evidence that for successful meta-learning, not the entire meta-gradient is required (Finn al. 2017).
EXPERIMENTS
In this section, show the results of several experiments, which empirically support that NP- MAML can deal with heterogeneous challenges while ensuring cross-domain adaption and out- performing previous adaptations of MAML. In addition, we analyze the different components of
5

---

Under review as a conference paper at ICLR 2023
In- and Cruss- Domain Clasification
MinilmageNett
FC100
1-shot
1-shot
S-shox
ANTI
47.69 0.62
62.58
0.53
0.53
BOIL
50.16 + 0.64
65.31 = 0.53
36.03 + 0.57
47.83 + 0.52
MAML
45,42 : 0.61
61.84 = 0.55
34.23 : 0.55
44.50
0.54
NP-MAML
49.82 + 0.64
67.39 = 0.53
38.57
+ 0.57
50.86
+ 0.54
Proto-Net
48.12 - 0.81
66.17 = 0.67
31.41
: 0.67
42.78 1: 0.70
UnicoMMAM
48.74 + 0.87
63.44 = 0.76
35,95
0.80
45.15 + 0.76
Table I: In this table, the classification accuracy [ [%] of all models on set of MinilmageNet and FC100 are depicted. All models have been trained on MinilmageNet.
MAML and show how each part of the network contributes to performance and representation change. All models are trained homogeneous environment and are subsequently tested in homo- geneous and heterogeneous environments. As first step, this allows for more fine-grained control of cause and effect relationships with regard to predictive performance than we would have if we applied it Meta-Datases Triantafillou et al. (2020). Our setting is also most realistic for real- world applications, where we meet environmental conditions unexpected at training time. Note, for example, that in our setting, models are not explicitly trained to cope with domain shift, yet showing strong performance in unexpected environment changes a crucial property in real-world problems. Despite all this, we regard validation of our findings on Meta-Dataset
important further step.
We design the experiments in a way that the effects of individual parts of the environment (domain, number of ways, and shots) are investigated and partially isolated. We train the conv4 architecture, the standard architecture in previous literature on few-shot learning with MAML (Finn et al. 2017]]
Raghu et al.) 2020 Oh et al., 202T Vuorio et al.] 2019 Yoon et al., 2018 Finn et al., 2018) on MinilmageNeet to solve a N N-way-K K-soot classification task. We further model domain shift by testing on variety of few-shot learning datasets, such as FC100 (Oreshkin et al., 2018), CIFAR-FS (Devos et al., 2019) and CUB (Wah et al.) (011), and measure the test accuracy. We further provide test-time accuracy for MinilmageNet to study the impact of NP-MAMI when going from the in- to the cross-domain setting. We choose the hyper-parameters of the original MAML-approaches (Finn) cet al. 2017: Raghu al., 2020; Oh et al. (02T) summarized
Appendix D For all performance hold-out set, where the - indicates the 95% confidence interval over tasks, following Finn et al. (2017). We further average these results across multiple seeds. We provide results for Minilmage Net and FC100 in the main body of the text and refer to Appendix [F] for results on CIFAR-FS and CUB, as well as to Appendix FF for studying an increase in the test-time environment change also with respect the number shots and Appendix G for detailed analysis of the importance of very early network layers on in- and cross-domain problems.
results, show classification accuracy
5.1 DOMAIN SHIFT
In our first experiment, train each model on 5-way-1-shot and 5-way-5-shot on MinilmageNet and test it on MinilmageNet (in-domain) and FC100 (cross-domain), where the numbers of shots K matches its training conditions.
In Table I we observe that our approach NP-MAML is competitive with BOIL on 1-shot Minilm- ageNet and better in all other cases. As expected, ANIL performs especially poorly in the cross- domain setup, as an adaptation of the body representations is crucial to cope with domain shift.
5.2 META-FEATURE QUALITY WHEN VARYING NUMBER OF WAYS
In this experiment, we look at the quality of the features of the penultimate layer, which we call meta-features. The model is trained as Section 5.1 on MinilmageNet on the homogeneous 5- way-K-shot problems. We test the model on both MinilmageNet and FC100 but vary the number of ways of each task (N = 4, 7 and 10). Since MAML, ANIL and BOIL can architecturally not fine-tune the last layer (see Section 4 we use the NIL-testing proposed by Raghu et al. (2020.. NP-MAML can be fine-tuned, and we depict both results without fine-tuning (0 inner loop steps) and with fine-tuning NP-MAML. (1 inner loop step).

---

Under review as a conference paper at ICLR 2023
(a) I-shol maxed-waps
MinillageNeet
FC100
4-way
5-way
7-way
10-way
4. 4.way
5-way
7-way
10-way
ANIL
55,29
0.77
49.64
0.63
40.78
+ 0.49
32.96
:: 0.36
39,03
1 0.67
33.61
0.53
26.39
0.42
20.62
1 0.31
BOIL
30.98 1 0.45
25.90
0.38
19:17
0.28
14:47
= 0.21
29.68
: 0.45
24.50
0.36
18.31
0.28
13.48
= 0.21
MAML
0.76
47.11
1 0.63
38.61
1 0.47
30.79
- 0.34
40.84
1 0.69
35,09
0.57
27.22
0.44
21.23
1 0.30
NP PP-MAMI.00 (0)
49.18
0.70
43.44
0.60
35.20
1 0.44
28.38
0.34
40.24
1 0.64
34.76
1 0.52
27.38
0.40
21.24
1: 0.31
NP 1AMMAML
55,50
1.500::0.75
49.82
0.64
41.48
0.47
33,59
0.36
44,09
1 0.67
38,57
0.57
30,37
0.4
2383
0,30
NettoNet
53.91 0.95
48.12
0.81
40.02
0.61
32.36
0.45
36.56
1 0.78
31.41
1 0.67
24.54
1 0.51
18.99
1: 0.37
UnicomMAML
53.18
1 0.99
48.74
0.87
39.87
0.68
31.88
0.49
41.78
0.94
35.95
0.80
27.78
0.61
21.87
0.42
5-shot ways
MinilmageNet
FC100
4-way
S-way
7-way
10-way
4-way
S-way
7-way
10-way
ANIL
70.89 + 0.60
66. 17
0.54
58.24
+ 0.43
50.16 + 0.34
52.43
+ 0.63
46.14
+ 0.54
31.84
+ 0.44
30.75 + 0.29
BOIL
39,52 - 0.50
33.47
0.42
26.23
0.31
20.02
= 0.25
35.81
+ 0.49
30.30
0.41
23.61
0.32
17.96 + 0.24
MAML
68.48 + 0.62
63.15
+ 0.55
55.03
+ 0.43
46.62
- 0.33
51.04
+ 0.64
44.75
0.54
36.41
+ 0.43
29.60
+ 0.30
NP-MAML (0)
70,70
63.49
0.54
57.95
0.42
50.47
= 0.36
57.89
+ 0.63
51.89
0.56
43.66
0.45
36.21
0.32
NP.MAMI.U (1)
72.27 + 0.60
67.39
0.53
59.59
= 0.43
$1.69
= 0.35
57.37
+ 0.60
50.86
+ 0.54
42.35
4 0.44
34,58
+ 0.31
Proto-Net
71,66 0.73
66.17
0.67
58.49
0.55
50.92
= 0.41
48.55
+ 0.82
42,78
0.70
35.23
+ 0.54
28.57
0.39
UnIcomMAMI.
68.66 0.89
63.44
0,76
55.48
0.62
48.00
0.46
51.63
4 0.92
45.15
0.76
37.65
0.60
30.59
+ 0.43
Table 2: The classification accuracy [%] of the models 1-shot and 5-shot in-domain for in- domain (MinilmageNet) and cross-domain (FC100) tasks, where the number of ways varies during test time, All models have been meta-trained on MinilmageNet.
In Table 2p. the results for 1-shot in- and cross-domain can be found. In the case of an in-domain adaptation, NP-MAML (fine-tuned) performs the best; however, it is only slightly better than ANIL (the difference is smaller than 1%). This holds true across all varying numbers of ways, which does not seem to have an impact on the performance difference. NP-MAML is, without fine-tuning, significantly worse than ANIL or MAML, but with one fine-tuning step. its performance improved by 5%. BOII. performs comparatively poorly in the mixed way setting, which confirms our claim from Section thhat BOIL is reliant on the train-time head, reinforcing one of the motivations for our nonparametrin approach. UnicornMAML performs similarly ANIL and MAML in the in-domain tasks. Here, its trainable linear layer does not yield any improvement over NIL testing. However, in the case of cross-domain tasks, UnicornMAML is significantly better than ANIL, BOIL, and MAML. Nevertheless, it still falls short in comparison to our novel technique, NP-MAML.
In the case of a cross-domain adaptation, NP-MAML again performs best, followed by MAML with a difference of around 3.5% across ways. NP-MAMI. without fine-tuning is only slightly worse than MAMI.. We observe similar results in the case of 5-shots and more few-shot learning datasets (Appendix E
For 5-shot learning (Table p)), we similar results, with the difference that for NP-MAML the fine-tuning step cannot improve performance on FC100. However, in the results reported in Ap- pendix E see an increased necessity for fine-tuning NP-MAML in the case of fine-grained prob- lems such as CUB. In addition, it seems that for the 5-shot problems in general, Unicorn-MAML less suited than MAML or ANIL, presumably because the NIL-predictor improves more strongly with the number of available shots.
The above results show that in the cross-domain setting, the ability to fine-tune the nonparametric predictor, as is the case for NP-MAML, gains importance with decrease in available support set samples.
5.3 REPRESENTATION SIMILARITY ANALYSIS
In our last experiment, we address the question rapid learning feature reuse?, as posed by Raghu et al. (2020), for NP-MAML. Similar to Raghu et al. (000 Oh et al. (202I): Goerttler & Obermayer (202)) we measure the centered kernel alignment (CKA) similarity of representations of the query set before and after adaptation for the three components Jearly Ipenult and Ihead of conv4 of all models we compared.
7

---

Under review as a conference paper at ICLR 2023
1-Shot 1-Shot
5-Wey-5-Bhe Shol
5-Wey-1-He 1-lo--Shha Shot
5-Way 5-to-1-Shot
(X (10iingggggee
1.0
BOIL
0.6
MAML
0.8
NP-MAMI
0.4
Eary
Ponultmmat
Head Early
Penutimate
Head
Early
Ponu
Head
Eary
Penultimato
Hoad
5-Way-1-Sh 1-Shot
5-Way
5-Way-1-t-555ae 1-ot--Shot
S-Way-S-to-1-e
1.0
CKA (FC100)
0.8
0.6
Earl
Ponultmmae
Early
Ponutimato
Earl
Ponu
Eary
Penultimato
(a) NP-MAML VS. NILA(ANIL, BOIL, MAML)
4-way
5-way
7-way
10-way
NP-MAL
CXA (FC/F0)) S-atot
UnicomMAMM
0.8
Eary Poruttmtone
Early
Eafy
Eary
Penulimans
(b) NP-MAML vs. UnicornMAML
Figure 1: CKA similarity of network representations before and after adaptation for the models of different scenarios from previous experiments. Representation similarity was obtained for the three components gearty, 9penult and Ihead of the conv4 architecture.
In Figure Ia we see the CKA similarities for different scenarios of our experiments. We can gen- erally observe that the representation of the head changes much less for NP-MAML than for the others, meaning a good performance does not require a huge change there. In the case of cross- domain tasks, the adaption for MAML and NP-MAML is larger than in the case of in-domain tasks, underlying the overall need for representation change in cross-domain environments. For the penul- timate layer, the similarity of BOIL. is much less than for NP-MAML. Interestingly, all observations do not change significantly when the number of shots changes during training and testing.
We additionally analyzed how the representation change differs between UnicorMAAML and NP- MAML the setup of the mixed ways because they are the only ones who can update the weights during fine-tuning (see Figure [bb We see that NP-MAML changes more in the early and the penultimate layers, which, as we know from the analysis done in Oh et al. (202T), can improve cross-domain adaptation capability. We hypothesize that the increased representation change of NP- MAML compared to UnicornMAML. contributes to cross-domain performance in our experimental setting.
DISCUSSION
In this section, we discuss all major components of MAML during training and testing under domain adaptation: training and testing (data processing, importance of low-level features and high-level features, type of predictor).
Task Distribution The results of Section 5 show that ANIL and MAML perform very well on in-domain tasks, even if the number of shots and ways is different in testing However, in the case of cross-domain adaption, we see that these algorithms perform poorly, as their features are not changing accommodate for the domain shift. BOIL, which is good in cross- noms-nomain adaptation, fails in the case of more realistic conditions, such as a flexible number of ways, as it seems to rely
In particular, see 5.2 and Appendix F
8

---

Under review as a conference paper at ICLR 2023
strongly the head applied during training, we showed in Sub-section 5.2 NP-MAML solves this issue by maintaining the strengths of BOIL (enforeing representation change) but still shows strong performance with flexible numbers of ways and shots.
Fine-Tuning As discussed in Section 3 fine-tuning on heterogeneous test environments is chal- lenging for existing architectures, gap filled by our approach, NP-MAML. From the results in Appendix F (Table (b), we see that fine-tuning is not improving the results when increasing the number of shots, which believe partially attributed the class prototypes that average across five samples. Still, we see benefit to fine-tuning again in the 5-shot setting of coarse- to fine-grain domain shift, demonstrated on the CUB dataset in Appendix E
In addition, we observed that NP-MAML reaches their cross-domain performance with just one fine-tuning step (similar to BOIL), whereas ANIL and MAML use and 10 for training and testing, respectively. We think that NP-MAML has many similarities to BOIL, e.g., strong cross-domain adaption and body-only inner learning.
We note that due to the absence of parametric head in NP-MAML, any fine-tuning thereof is naturally omitted. However, another perspective is to view NP-MAMI. as employing a pseudo- head, a classic linear layer whose parameters are, however, implicitly formed by the output of the network body (Snell et al. 2017). We want to present this perspective to highlight that NP-MAML is not necessarily variant of BOIL but follows distinct paradigm of abandoning the parametric head as a standard component of meta-learning architectures.
Early Layers The analysis of the importance of early layers shows that penultimate-only models perform at least as well as full models in in-domain tasks but are significantly worse for cross- domain adaption (see Appendix G). The effect of only updating the penultimate layer harms BOIL. more than NP-MAML, which indicates a higher resilience of our approach to this effect. Since for NP-MAML, the penultimate version is always better for in-domain tasks, it can be summarized that fine-tuning them is beneficial for cross-domain tasks, but for in-domain tasks, we better leave them not-updated. Especially in the case of larger models (e.g., ResNet), this is important note, as speeds up the fine-tuning quite significantly (as also argued in (Oh et al. 2021)) )
Penultimate Layer Oh et (202I) have already discovered the importance of fine-tuning the penultimate layer for few-shot learning, especially for cross-domain adaptation. Our results from Section [5.]] and Appendix [F] confirm that this also extends to our heterogeneous setting, further strengthening the case for methods such as ours that enable strong representation change in the network body rather than the head.
Predictor (Head) The choice of the predictor (head) influences the generalization performance the most. our experiments, we could show the superiority of nonparametric heads. Especially for heterogeneous tasks, it allows for good adaptation. In addition, NP-MAML is the only model where training and test conditions match precisely. That this is desirable has been discussed by Vinyals et al. (2016).
7 CONCLUSION
In this paper, we proposed NP-MAML, a nonparametric version of MAML, which can deal with architectural and algorithmic challenges in few-shot learning. In addition, we explored the key components of MAML-type models and proposed a common framework to describe them. Our ex- perimental results show that NP-MAML outperforms established methods with parametric heads. Under domain shift, our method beats ANIL and MAML due to its ability to perform representa- tion change also the earlier layers. In the case of heterogeneous task distributions, NP-MAMI. very flexible in fine-tuning the support set, something which BOIL is limited at architecturally. In our in-depth analysis of MAML key components, we revealed the hitherto unknown importance of fine-tuning very early layers for improved cross-domain capabilities. Our representation similarity analysis showed that NP-MAML adapts less in the network body during the inner loop than BOIL; however, we demonstrate that although NP-MAML adapts its parameters less in total, the distri- bution of adaptation over the network components is much more uniform, which seems to be the crucial factor for performance in the presented heterogeneous tasks.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Milad Abdollahzadeh, Touba Malckzadeh, and Ngai-Man Cheung. Revisit multimodal meta- learning through the lens of multi-task learning, CoRR, abs/2110. 2021.
:
attaaaaa/...../ 4202
Alex Andonian, Taesung Park, Bryan Russell, Phillip Isola, Jun- Yan Zhu, and Richard Zhang. Con- trastive feature loss for image prediction. In IEEE/CVE International Conference on Computer Vision Workshops, ICCVW 2021, Montreal, BC, Canada, October 11-17, 2021, pp. 1934-1943. IEEE, 2021. doi:
100220. URL httph: / doi orgg 10. /t.
ICCVR54120 2021.00220
Antreas Antoniou and Amos J. Storkey. Learning learn by self-critique. In Hanna M. Wal- lach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 9936 9946, 2019. URL /R//ttpps///soceeeingg. neur httpppssccpppppppp paper/ 601 Bdf aataaa/001.....27711111556611113
201
Antreas Antoniou, Harrison Edwards, and Amos J. Storkey. How to train your MAML. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https:://penreview..rr wwwp://wwwww.eeeeotet.....
HJGven05Y7
Sébastien M R Amold, Praateek Mahajan, Dehajyoti Datta, Ian Bunner, and Konstantinos Saitas Zarkias. learn2learn: A library for Meta-Learning research. August 2020. URL http://
arxiv.org/ abs///b///// /20bs/22008.. 2008 1 12284
John Cai and Sheng Mei Shen. Cross-domain few-shot learning with meta fine-tuning. CoRR, 10544
abs/2005. 10552005.10544, 2020. URL
Wei-Yu Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, OpenReview.net, 2019. URL https: seopenreview.ne/ffoooe http://oorrrmmm.-BBBaAAAAAAA
Tristan Deleu, Tobias Würff, Mandana Samiei, Joseph Paul Cohen, and Yoshua Bengio. Torch- meta: A Meta-Learning library for PyTorch, 2019.
06576 Available at: https://github.c https://gihub..oo//rrtaaaedeeee wwt..///ssttadddddeyyooooe-eeeee
Arnout Devos, Sylvain Chatel, and Matthias Grossglauser. Reproducing meta-learning with dif- ferentiable closed-form solvers. In Reproducibility in Machine Leurning, ICLR 2019 Work- shop, New Orleans, Louisiana, United States, May 6, 2019. OpenReview.net, 2019. URL
http::////peeeeviee net Bttp://Eooummmi-dX002EEI
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leamnin for fast adaptation of deep networks. In Doina Precup and Yee Whye Teh (eds.). Proceedings of the 34th Interna- tional Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-1 I August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 1126-1135. PMLR, 2017. URL www.pperssss/SE//. prrasrceeerrngss..rprrsess//s/e
httt::/pprcced: ngs www..m.........m..............em
html
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi and Roman Gareet (eds.), Advances in Neural Information Processing Systems 31: Annual Con- ference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 9537-9548, 2018. URL https: http:::://proceeee http://pooocdccooocoocoooooocecc ngs nwwrips....... cc/ paper http/////c.........00000000331.0 Abstraattbbbtttt.t.ooo html
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forget- ting. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR
2018, Salt Lake City, UT, USA. June 18-22. 2018, pp. 4367-4375. Computer Vi-
sion Foundation / IEEE Computer Society, 2018. doi: 10.119/CVVP..2088.0055
URL bttp:://penaccess.c www.cccesss..eec.e...... thecvE com/ www//oootnnttcceeeecoooooccoceee content _vprr____p_____ 2018 (w.18/htmm/// htthh/mmmmmmarr..m
Dynami F..Few-Shot.. Sicthhtt__iuueuae (ttpssaa___P______ @VVPR_2018 paperpppereem html
10

---

Under review as a conference paper at ICLR 2023
Thomas Goerttler and Klaus Obermayer. Exploring the similarity of representations in model- agmostic meta-learning. CoRR, abs/2105.05757, 2021. URL httpps//aaxiv..ortt abs/
21055 05757
Hal Daumé III and Daniel Marcu, Domain adaptation for statistical classifiers. J. Artif. Intell. Res., 26:101-126, 2006. doi: 10. 16.faar 1/.2. 1872. URL http:://doi.org/10...000
1872
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few shot learning. CoRR, abs/ (0s/1777709835, 2017. URL http::/arx....rg//
09835
Vivek Myers and Nikhil Sardana. Bayesian meta-learning through variational gaussian processes. CoRR, abs/21 10.12110.1110 1044, 2021.
10.4444,,
abs/
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta meta-ig algorithms. CoRR, abs/1803. 9803,02999, 2018. URL httt/hhpp//artttee /ttp://rxx.....r////////2/3 02999
Jaehoon Oh, Hyungjun Yoo, ChangHwan Kim, and Se- Young Yun. BOIL: towards represen- tation change for few-shot learning. In 9th International Conference on Learning Represen- tations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL
https://openzeview net /eet/eoumm""
Boris N. Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. TADAM: task depen- dent adaptive metric for improved few-shot learning. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa- Bisa-Bianchi, and Roman Garnett (eds.), Ad- vances in Neural Information Processing Systems 31: Annual Conference Neural Infor- mation Processing Systems 2018, NeuriPS 2018, December 3-8, 2018, Montréal, Canada, pp. 719-729, 2018. URL http::ts//poccceciieeeeee neurips. cctpppppeer/21//aaa... Abst htt.
Hang Qi, Matthew Brown, and David G. Lowe. Low-shot learning with imprinted weights. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 5822-5830. Com- puter Vision Foundation / IEEE Computer Society, 2018. doi: 10.1109/CVPR.201 00610. URL
thecvf. com/ comm///oteeetttvcccee catpnnt_vvrrrr111111ee 2018 2018//tm1/11 Low Lhoot_eezznigg CVV 2 2ww___aaeeree__eeeee pappp.httm
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning feature reuse?
towards understanding the effectiveness of MAML. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL httpppps///pppeeevieeeee net wwt.:/rrttffettte///ee/
forum? rd=
Aravind Rajeswaran, Chelsea Finn, Sham M. Kakade, and Sergey Levine. Meta-learning with implicit gradients. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Flo- rence d' d'Alché-Bur Emily B. Fox, and Roman Garnett (eds.), Advances in Neural In- formation Processing Systems 32: Annual Conference on Neurul Information Process- ing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp.
113-124, 2019. URL 072b030ba 26626 444.040006000.444444444444444--. Abs
Carl Edward Rasmussen and Christopher K. Williams. Gaussian processes for machine learn- ing. Adaptive computation and machine learning. MIT Press, 2006. ISBN 026218253X. URL
https:/ www. worl dww.l.cc.ccoooo www.dat.o./ooo...o.ooooo.oo.oe wttg//ll////// 2/5773
Sachin Ravi and Hugo Larochelle, Optimization as a model for few-shot learning. In 5th Interna- tional Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL htttps://ppeeeeeeeee net/
Forum??d==rJY0-Kcll
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhi- heng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and
11

---

Under review as a conference paper at ICLR 2023
Li Fei-Fei. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis., 115(3): 211-252, 2015. doi: 10. 10.10077112223-055-01166. URL https://aoi. wttp:/iioor....00.....0......
611263-015-08165
Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osin- dero, and Raia Hadsell. Meta-leamning with latent embedding optimization. In 7th International Conference on Learning Representations. ICLR 2019, New Orleans, LA, USA, May 6-9. 2019. OpenReview.net, 2019.
Ruslan Salakhutdinov, Joshua B. Tenenbaum, and Antonio Torralba. One-shot learning with a hi- erarchical nonparametric bayesian model. In Isabelle Guyon, Gideon Dror, Vincent Lemaire, Graham W. Taylor, and Daniel L. Silver (eds.), Unsupervised and Transfer Learning Work- shop held at ICML 2011, Bellevue, Washington, USA, July 2, 2011, volume 27 of JMLR Pro- ceedings. pp. 196-206. JMLR org. 2012. htt:: httttp:://ppocceeingg..o. www.:/....mm..poo...m ptess//2222 [salak Saalakhutdinov www.ioooollllaaaaaea /aa hatpa//dd...ooieaaaaaaaaaaaaatan
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In Is- abelle Guyon, Ulrike von Luxburg. Samy Bengio. Hanna M. Wallach, Rob Fergus, S. V. N. Vish- wanathan, and Roman Garnett (cds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 4077 4087, 2017. URL httpp:::/prroceedddttctee/// httml 2ttp://8866811111122222222222220
neur .ts.cc/
paper/2017
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta- dataset: A dataset of datasets for learning to learn from few examples. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. httptttss://ppeseeeeepe
OpenReview.net, 2020.
net
Hung-Yo Yu Tseng, Hsin-Yine Ying Lee, Jia-Bin Huang, and Ming- Hisgg Yang. Cross-domain few-shot classification via learned feature-wise transformation. In 8th International Conference on Learn- ing Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL /ttpttttts///ppeeeee
n fow.mmpd=s11111pP11.1
Oriol Vinyals, Charles Blundell, Tim Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 29: Anmual Conference Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 3630-3638, 2016. URL https: /ttp://////ooceeeenn//ee neurips. /c/ Abcc--bsttrccc .tmml
Risto Vuorio, Shao-Hua Sun, Hexiang Hu, and Joseph J. Lim. Multimodal model-agnostic meta-learning via task-aware modulation. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' A'ché-Buuc Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 1-12, 2019. URL https: http://:s///pcceeeeeeeeeee neurips....p....... cc/papprr2219//as Awst .tml
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-ucsd birds-200-2011 dataset. Technical Report CNS-TR-2011 1-001, California Institute of Technology, 2011.
Han-Jia Ye and Wei-Lun Chao. How train your MAML to excel in few-shot classification. In The Tenth International Conference on Learning Representationss ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL httptttss///oeeeeee/ee net.iiewwwweeeeeeeeeeeeeeeeeeeee htt.://wwww../tooomimeeeeo.
TkpJtaR
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learming. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Gamett (eds.), Ad- vances in Neural Information Processing Systems 31: Annual Conference on Neural Informa- fion Processing Systems 2018, NeurlPS 2018 December 3-8, 2018 Montréal, Canada, pp-
12

---

Under review as a conference paper at ICLR 2023
7343-7353 2018. URL http://s/.pooeee........////e//p 2at.:0/00.222222000000000---a-a- .tml
Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In
David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part volume 8689 of Lecture Notes Computer Science, pp. 818 833. Springer, 2014. doi: 10.1007/ 978- 978-3-319- 10590-11 1) 53. URL /ttp:://eoel...a///....077777133
53
13

---

Under review as a conference paper at ICLR 2023
Model
Configuration
(oarty Spenett Sheat
ANIL
Raghu
di 2020
19heat
Oh
(gurrt Spennt
BOIL
Oh di
272
{9pont)
Table 3: Meta-learning configurations corresponding to methods from (Finn et al. 2017 Raghu al. 2020 Oh al. (02T). BOIL-4 indicates a variant BOIL. where only the penultimate layer of conv4 meta-learned.
A INSTANTIATION OF PREVIOUS METHODS OF OUR MODULARIZATION
Our modularization decomposes the network into different parts Previously proposed methods Finn et al. 2017 Raghu et al. 2020: Oh et al. 2021) can be obtained by only optimizing the weights of selected sub-components (see Table
B (CROss-)DOMAN) ADAPTATION AND GENERALIZATION
The problem of domain adaptation refers the situation where (labeled) data in one domain A exists plentiful, while (labeled) data in different domain B (typically called the target domain) is scarce, requiring models to transfer knowledge gained in the domain A to make predictions in domain B (Tseng al. 2020). As stated by III & Marcu (006),, the idea is not to ignore data from domain B entirely but rather reduce the acquisition effort by transferring from domain A. This opposed to domain generalization, where models have no access to any data from B during training. The latter sometimes also referred to as cross-domain transfer (Triantafillou et al. 2020) or cross-domain adaptation (Oh et al. 2021).
In this work, we adopt the use of cross-domain adaptation as referring to both having little or no access data from domain B, to stay consistent with the current literature in meta-learning (Tiantafillouu et al. 2020 Cai & Shen 2020 Tseng et al. 2020).
B. MULTI-MODAL LEARNILG
To adapt the above distinction for another knowledge transfer problem setting, in multi-modal learn- ing, one presents a model with data from both A and B, hoping that predictive performance improves via knowledge transfer between the domains, that for example, reuse of general image features that can be found in both domains (Vuorio et al. 2019 Abdollahzadeh et al. 2021). Yet, knowledge transfer is not limited to such rather obvious commonalities. Vuorio al. (2019) explicitly model
such multi-mod modal settings with MAML and show performance improvement over a MAML model
that treats each task coming from the same domain. Research in this direction has direct implications on mixtures-of -omoma datasets, such as Meta- -ata-Dttaset (Triianafiilouu et al.) 2020
B.2 COARSE- VS. FINE-GRAINED DOMAINS
Another important notion in cross-domain adaptation the notion of coarse- and fine-grained do- mains (Triantafillou et al. 2020 Oh et al. 2021), referring to the level of abstractions of the "topics" contained within the data. For example, in the context of image classification, a domain containing classes like dogs, cats, elephants, and giraffes is considered coarser than domain with classes like German Sheppard, Corgi, Husky, and Bulldog. As "level of abstraction" " is not clearly defined, this notion is typically used only in distinguishing adaptation from coarse- to a fine- fine-grained domain.
domains, for example, when performing domain
C INVESTIGATING THE EFFECT OF THE NONPARAMETRIC PREDICTOR
In Figure 2 we investigate the effect of the nonparametric predictor on the features of the network body visually.
14

---

Under review as a conference paper at ICLR 2023
(a) Before Fine-Tuning
(b) After Fine-Tuning
3
2
-1
-1
-2
-2
2
2
Figure 2: T-SNE plot of the features of a 3-way-1-shot task, produced by network body trained with an NP- -P-predictor. Features are computed from test samples of MinilmageNet task, (a) before and (b) after fine-tuning and colored according to their ground-truth label. We observe that the NP-based fine-tuning, as introduced in Equation 12] resolves the cluttered features in the middle of the left plot, pulling features that do not belong to the same tasks apart. The model that produced the above example was trained for 30.000 iterations 5-way- I-shot MinilmageNet, with the same hyperparameters as for our experiments Section
D IMPLEMENTATION DETAILS
In this section, outline the exact implementation details of each of the models used throughout the experiments in Section 5
All models were trained and tested on the conv4 architecture (Finn et al.! 2017 Raghu et al. 2020 Oh et al. (021). comprised of four convolutional blocks, each containing a x 3 convolutional layer with one stride, a 2D Batch-Normalizztion layer, a ReLU non-linearity and max- max-pooling layer with two strides. For models with parametric heads (ANIL, MAML, and BOIL), we formed predictions from the convolutional features with linear, fully-connected layer. UnicornMAML. was trained and tested always with its linear head. In its original paper, they always used pre-trained weights wherefore our results do not match their high performance.
MinilmageNet, FC100, CIFAR-FS and CUB data has been preprocessed with the data-loadens of the torchmeta library (Deleu et al. 2019). Constructing the conv4 architecture, as well as fine-tuning the models, has been aided by the learn2learn framework (Arnold et al. 2020).
D.1 MODEL CONFIGURATIONS AND HYPERPARAMETERS
We carefully selected model configurations and their corresponding hyperparameters based on best practices in previous work (Finn et al. 2017 Raghu et al. 2020 Oh al. 202T). In addition, to run MAML training at a similar time to the other models. we used first-order MAML throughout all of our experiments. In addition, as outlined Section 2.2.2, we used an approximation the meta-gradien of NP-MAMI. and NP-MAMI-4 models. We trained and evaluated each model on three different seeds and reported averages across seeds.
Gradient-basee meta-learning typically requires the following hyperparameters: A number of train- ing iterations Tirain a number of test iterations Tist a number of fine-tuning steps during training Strain. number of fine-tuning steps during test time Stest. a number of test shots (in the query set) Kiest: an outer learning rate 3, an inner learning rate a, a batch-size the number of tasks sampled each iteration and over which the meta- getaiint is formed, a flag indicating whether to approximate the meta- gradient and a method for forming predictions if nonparametric testing is required (c.g.,
NIL. testing (Raghu al. 2020))
15

---

Under review as a conference paper at ICLR 2023
Method
Approximate MetaGrad
Nonparanetric Prodictor
ANIL
5
10
0.001
0.01
No
NIL
MAML
5
10
0.001
0.01
Yes
NIL
BOI..
0.001
No
NII.
NP -AAM
1
0.001
Yes
Pairw. distunce
UnicoroMAMI.
0.001
Yes
Table 4: Hyperparameter settings for ANIL, MAMI, BOIL, NP-MAML and UnicorMMAML, used for the experiments in Section
(a) 1-shot mived-ways on CIFAR-FS FS and CUB
CIFAR-FS
CUB
4-way
5-way
7-way
10-way
4-way
5-way
7-way
10-way
ANIL
43.75
0.96
0.22
31.04
24.48
0.78
31.13
24.18
BOIL
31,13
0.63
26.11
0.54
19.89
0.41
15.06
0.31
28.48
1 0.53
23.31
1 0.45
17.12
0.36
12.45
+ 0.25
44.85
0.98
31.09
24.64
43.35
0.90
37.63
0.76
30.19
055
23.48
NP MP-MA (0)
46.64
0.95
40.69
0.81
31.99
0.66
24.64
4 0.47
41.59
1 0.84
36.36
6
0.71
28.81
0.55
22.47
: 0.41
NP-MAML
52.08
1.05
0.89
38.15
31,05
48.60
43,02
0.00
34.80
0.62
27.63
0.46
UnicorMAML
45.46
1 0.99
39.78
0.82
32.09
0.66
25.73
0.50
45.07
0.89
40.18
0.73
32.20
0.60
25.05
0.44
5-shot mixed-ways CIEAR-FS and CUB
CIEAR-FS IS
CUB
4-way
S-way
7-way
10-way
4-way
5-way
7-way
10-way
ANIL
60.37 + 0.93
55.01
0.8
47.05 = 0.65
40.10
= 0,51
61.8 + 0.86
53.55 + 0.80
47.80
0.65
40.15 = 0,52
BOIL
23.70
MAML
58.63
+ 0.94
52.96
083
44.88
0.65
30.79
= 0.34
57,30
+ 0.87
$1.01 + 0.78
42.76
+ 0.60
35.26
1 0.47
NP-MAML
NP-MAML (1)
68.47 + 0.92
63.18
082
55 50
0.69
48.61
= 0.54
65.69
+ 0.86
59.78
0.78
51.93 + 0.64
44.19 = 0.52
39.53
Table 5: Further results of classification accuracy [%] of ANIL, BOIL, NP-MAML, MAML, and UnicornMAML 1-shot and 5-shot cross-domain tasks (CIFAR-FS and CUB) with a varying number ways during test time. The models were pre-trained on MinilmageNet.
Following previous work (Raghu et al. 2020 Oh et al. 2021 we train each model on 30.000 iterations on MinilmageNet, and following Finn et al. (2017). J. we evaluate test performance on total of 500 randomly sampled tasks from the test set of the target domain (either MinilmageNet or FC100). Additionally, across all experiments and models, we set K = 15. Further, for 5-shot training testing, set the batch-size to 2, and for 1-shot training testing, set the batch-size to 4. All other hyperparameters are method method-specific and are presented for each of ANIL, BOIL, NP-MAML, and MAML in Table 4 Genultimate-on/e methods BOIL-4 and NP-MAML-4 follow the hyperparameter settings of BOIL and NP-MAML, respectively.
All experiments have been conducted single NVIDIA Titan Xp GPU.
E ADDITIONAL RESULTS FOR VARYING THE WAYS
We performed additional experiments CIFAR-FS and CUB, which can be found in Table The results confirm our observation Section 5.2
F ADDITIONAL RESULTS ON THE PERFORMANCE WHEN VARYING NUMBER OF SHOTS
We repeat the experiments from Section 5.1 and 5.2 and additionally interchange the numbers of shots while testing the models (1-shot to 5-shot and vice versa).
In Table DA, NP-MAML good and outperforms when we are in a cross-domain setup. Interestingly, ANIL is very bad when we have five shots in training but only in training.
the results when K is constant. Especially in the of 1-to-5-shot problem,
16

---

Under review as a conference paper at ICLR 2023
(a) Switching number of shots with fixed ways
MinilmageNet
FC100
3.to-1shhe
1-to-S-shot
5-to-1 S-hot--shot
140-5-shol
ANIL
27.23 0.42
61.56 + 0,56
25.05 1 0.35
41.71
0.56
BOIL
49.20
0.63
63.92
0,54
35,51
+ 0.53
46.83
0.52
MAML
43.23
0.60
60.42
0.57
32.37
1 0.52
45.44
0.55
NP-MAMI
45.68
0.58
63.41 1: 0.51
38.57
1 0.52
50.28
0.53
(b) I-to-5-shot with mixed ways
MinilmageNeet
FC
4-way
5-way
7-way
10-way
4-way
5-way
7-way
10-way
ANIL
63.20
54.96
0.34
0.63
0.54
36.15
29.24
ROTL
38,33 + 0.500
32.68
+ 0.42
25.34
+ 0.32
19:45
0.23
35.31
0.53
29.84 + 0.44
23.02
+ 0.33
17.59
+ 0.24
MAML
60.81
52.26
0.66
0.55
36.69
29.53
NP NP-MAMI (0)
69.23
+ 0.61
64.16
+ 0.54
56.10
+ 0.45
48.43
0.38
56.42
0.64
50.91
+ 0.54
42.26
4 0.44
34,68
+ 0.32
NP-MAML (1)
69.18
0.59
63.41
+ 0.51
54.31
+ 0.42
45.82
0.33
56.18
0.65
50.28
+ 0.53
41.25
+ 0.42
33.40
+ 0.32
(c) 5-to-I-shot with mixed ways
MinilmageNet
FC100
4-way
5-way
7-way
10-way
4.way
S-way
7-way
10-way
ANIL
55.84
0.73
50.47
0.64
42.25
+ 0,48
34.22
0.36
40.83
0.63
35.34
0.53
27.62
+ 0.40
21.53
+ 0.29
BOIL
31.15
+ 0.44
25.96
+ 0.37
19.44
+ 0.29
14.37
0.21
29.94
0:43
24.45
+ 0.34
18:40
+ 0.27
13.52
+ 0.19
MAML
53.86
0.73
48.34
0.63
39.93
31.99
0.35
40.49
0.63
34.26
0.53
27.05
0,41
20.95
0.29
NP-MAML (0)
46.52
0.67
41.18
+ 0.59
33.15
+ 0.44
26.35
0.34
39.75
0.59
34.11
+ 0.53
26.98
+ 0.40
21.20
+ 0.31
NP-MAM
50.91
0.68
45.68
0.58
37.31
30.28
0.35
41.56
0.62
38.57
0.52
27.96
0.40
21.88
0.29
Table 6: these tables, the number of shots is switched between training and testing. We see the classification accuracy [%] of the models MinilmageNet. In (a), the number of ways is fixed, whereas, in (b) and (c), we have a flexible number of ways. In (b), the number of shots increases during testing, and in (c) decreases.
test of MinilmageNeet and FC100, all trained on
When also the number of shots varies, BOIL is again not able to achieve good results. When in- creasing the number of shots during testing (see Tb), NP-MAML and ANIL are overall the best performers in the case of in-domain, and NP-MAMI is the best in the case of cross-domain. The only difference is that fine-tuning in this heterogeneous setup is no longer beneficial for NP-MAMI.
When decreasing the number of shots, NP-MAML suffers from the small numbers of samples in the case of in-domain (Table 6p)). Interestingly. ANIL is very good in contrast to the fixed setup of Table Gh. We fine-tune the last layer in the fixed case, whereas use NIL-testing in the mixed setup, which seems to influence the performance in case of varying the number of shots. In the case of dromin,,main, NP-MAML outperforms the other approaches.
G ADDITIONAL RESULTS OF EXPERIMENTS OF THE PENULTIMATE LAYER
We want to investigate the role the earlier layer Jearly. which are in the standard setup of MAML. and BOIL fine-tuned. We observe the significance of updating the early layers by comparing the predictive performance of BOIL and NP-MAML with only updating the penultimate layer (suffix -4) in the inner loop versus updating all layersIn Table we present the results for w-ooy-few-shot and mixee-way-l-shhee
In the case of fixed-way (Table 71), the penultimate only variant performs similarly for in-domain tasks but is significantly worse for cross-domain tasks ( 12% for NP-MAML and 4% for BOIL). For the mixed-way setup (Table ()). is similar. in-domain problems, the penultimate-only variant is slightly better, but updating all weights in the inner loop is beneficial for the cross-domain setup. For the mixed-way 1-shot in-domain case (b), NP-MAML-4 consistently performs best across all ways after fine-tuning, Hrwuningg.wwver..... However, for the in-domain case, the fine-tuned NP-MAML-4 improves only slightly over the finettundd NP-MAML with a performance difference of about 0.5%. As for all mixed-way way experiments, BOIL, and by extension BOIL-4, cannot compete with the other methods, and we also observe no notable performance difference between BOIL-4 and BOIL in neither the in- nor the cross-domain case. For the mixed-way way cross-domain case (c), the fine-tuned NP-MAMI MAML-4
17

---

Under review as a conference paper at ICLR 2023
(a) S-way-ffe-siie
MinilnageNet
FC100
I-shot
5-shot
1-shot
5-shot
BOIL
50.16 0.64
65,31 1 0.53
36.03 1: 0.57
47.83
0.52
BOIL-4
49.61 0.62
64.15 + 0.53
32.97
+ 0.52
43.48
0.51
NP MAML
49.82 0.64
67.39
1 0.53
38.57
0.57
50.86
0.54
NP 4
50.73 0.62
67.13 1: 0.53
36.12 1 0.55
48.40
0.54
(b) mixed-way-
MinilmageNett
FC100
4-way
5-way
7-way
10-way
4-way
5-way
7-way
10-way
BOIL
25.90
19.17
0.21
24.50
BOIT -4
29.67 + 0.40
24.44
+ 0.33
17.98
+ 0.25
13.14
0.18
29.08
0.4
24.10
0.35
17.90
+ 0.27
13.18
0.20
NP-MAML
43.44
27.38
NP-MAMI
55.50
+ 0.75
49.82
+ 0.64
41.48
+ 0.47
33.59
+ 0.36
44.09
0.67
38.57
0.57
30.37
+ 0.43
23.83
+ 0.30
NP-MAML-4
49.26
- 0.70
43.11
+
34.74
0.45
27.69
0.34
31.80
24.63
19.22
0.29
NP-MAML-4()
56.02
0.75
50.73
+ 0.62
41.98
+ 0.48
34,06
0.37
41.74
0.65
36.12
0.58
28.44
4 0.43
22.45
+ 0.30
miixed-way-5-shot
MinilnageNett
FC100
4-way
5-way
7-way
10-way
4-way
5-way
7-way
10-way
BOIL
39.52 + 0.50
33.47
+ 0.42
26.23
+
0.31
20.02
+ 0.25
35.81
0.49
30.30
= 0.41
23.61
+
0.32
17.96
+ 0.24
BOIL-4
35.97
0.44
30.20
+ 0:37
23.08
0.29
17.54
0.22
35.
0.49
29.73
17.49
0:23
NP-MAML (0)
70.70 - 0.60
65.49
+ 0.54
57.95
+ 0.42
50.47
+ 0.36
57.89
- 0.63
51.9
0.56
43.66
+ 0.45
36.21
+ 0.32
NP-MAML
72.27
67.39
0.4
51.69
0.35
$7.37
0.60
50.86
0.54
42.35
0.44
34.58
0:31
NP-MAML-4 (0)
70.07
0.60
65.09
+ 0.54
57.51
+ 0.43
49.78
+ 0.35
$3.23
0.60
0.60
47.08
0.53
38.80
+ 0.42
32.15
+ 0.30
MP
1
Table 7: Classification accuracy [%] of BOIL, BOIL-4, NP-MAML-4 and NP-MAML, each trained on 5-way-1-shot MinilmageNet and evaluated both fixed-way way (a) and mixed-way (b, c) problems. Since only NP-MAML and NP-MAML-4 can be fine-tuned in the mixed-way setting, NP-MAML and NP-MAML-4 results suffixed with the number of fine-tuning steps in parenthe- ses.
falls behind the fine-tuned NP-MAML by about 3%. In all mixed-way experiments, fine-tuning improves the performance of NP-MAML. The results for 5-shot similar.
We also present comparisons of BOIL and NP-MAML with their penultimate-only counterparts for varying the number of shots between training and test time in Section T Tables and 10
MinilnageNeet
FC100
5-to-1-shot
1-to-5-sbot
5-to-1-shot
1-to-S-shot
BOIL
49.20
63,92 0.54
35,51
0.53
46.83
BOIL-4
48.66
0.63
62.65
0.55
33.08 4 0.50
41.87 4 0.49
(P-MAML(()
45.68
0.51
38.57
0.52
50.28
NP-MAMI-+
44.38 0.60
64.10 0.51
32.60
0.48
47.39
1 0.53
Table 8: Classification aceuracy [%] of BOIL and NP-MAML, as well as their penultimate-only counterparts BOIL-4 and NP-MAML-4, Evaluation results for 1-shot are obtained from correspond- ing models trained 5-way-5-w --ot- MinilmageNeet and evaluation results for 5-shot are obtained from corresponding models trained on 5-way-1-shot MinilmageNet. The (1) after the NP-MAML and NP-MAML-4 results indicate that one fine-tuning step was performed before test performance was measured. Improvements of NP-MAML over its penultimate-onle counterpart under domain shift become more evident when either in- or decreasing shots during test time. When switching from 5 to 1 shots in-domain (on MinilmageNet), BOIL and BOIL-4 much stronger than the -NP variants. This effect mirrored in favor of the -NP variants when switching from to 5 shots cross-domain (on FC100).
18

---

Under review as a conference paper at ICLR 2023
MinilmageNeet
FC100
4-way
S-way
7-way
10-way
4-way
5.way
7-way
10-way
BOIL
38.33
0.50
32.68
+ 0.42
25.34
+ 0.32
19.45
+ 0.23
35.31
0.53
29.84
0.44
23.02
1 0.33
17.59
1 0.24
BOIL-4
35.70
0.44
29.87
1 0.38
22.53
1: 0.28
16.91
1 0.20
34.15
0.49
28.86
0.42
21.92
0.31
16.66
1 0.23
NP-MAML (0)
69.23 - 0.61
64.16
1000:00 0.54
56.10
0.45
48.43
0.35
56,42
0.64
90.91
0,54
42.26
1 0.44
34.65
1 0.32
NP-MALL (1)
69.18 1: 0.59
6341
0.51
54,31
1 0.42
45.82
0.33
56.18
0.65
50.28
0.53
41,25
0.42
33.40
0.32
NP-MAML-4
69.53 = 0.60
64.64
+ 0.53
56.58
1 0.43
48.39
+ 0.34
53.29
0.64
47.56
0.53
39.14
1 0.44
31.92
0.31
MAML-M- (1)
69.37
1 0.59
64.10
1 0.51
35.60
1 0.4
47.05
0.32
53.05
0.63
47.39
0.53
38.89
1
0.43
31.70
1 0.31
Table 9: Classification accuracy [%] of BOIL and NP-MAML, as well as their penultimate- counterparts BOIL-4 and NP-MAML-4, each trained on 5-way-1-shod MinilmageNet tasks and eval- uated on a varying number of ways and five shots on in-domain (MinilmageNet) and cross-domain (FC100) tasks. Since only NP-MAMI. and NP-MAML--4 can fine-tuned the mixed-war setting, NP-MAML and NP-MAML-4 results suffixed with the number of fine-tuning steps in parenthe- ses. In-domain results for -NP variants differ only slightly from each other. Cross-domain results show improvements in favor of NP-MAML, highlighting the importance of early layer fine-tuning in cross-domain settings, as discussed in Section 6 BOIL-4 performance is generally worse compared to BOIL, independent of in- cross-domain.
MiniImugeNeet
4-way
5-wny
7-way
10-way
4-way
5-way
7-way
10-way
BOIL
31.15 - 0.44
25.96
19.44
0.29
14,57
+ 0.21
29.94
0.43
24.45
0.34
18:40
+ 0.27
13,52
+ 0.19
BOIL-4
29.64 = 0.39
24.23
2 0.33
17.93
0.25
13.110.11
29.52
0.42
24.19
0.35
18.07
1 0.27
13.37
+ 0.20
NP MAML (0)
46.52
0.67
41.18
1 0.59
33.15
1 0.44
26.35
1 0.34
39.75
0.59
34.11
0.53
26.98
0.40
21.20
0.31
NP-MAML (3)
50.91
0.68
45.68
1 0.58
37.31
1 0.44
30,28
0.35
41.56
0.62
35.57
0,52
27.96
1 0.40
21.88
0.29
MAMLAML- (0)
43.95
0.69
38.04
1 0.61
30.07
1 0.46
23.17
0.37
35.02
0.52
29.44
0.45
22.95
1
0.35
17.72
0.27
NP-MAML-1
49.91
0.69
44.38
4 0.60
36.30
+ 0.44
29.04
0.35
38.28
0.56
32.60
0.48
25.81
1 0.38
20.19
0.28
Table 10: Classification accuracy [%] of BOIL and NP-MAML, well as their penultimate- only counterparts BOIL-4 and NP-MAML-4, each trained on 5-way-5-shot MinilmageNet tasks and evaluated on varying number of ways and 1 shot on in-domain (MinilmageNeet and cross- domain (FC100) tasks. Since only NP-MAML and NP-MAML-4 can be fine-tuned in the mixed- way setting, NP-MAML and NP-MAML-4 results are suffixed with the number of fine-tuning steps in parentheses. When switching from 5 shots during training to shot during test time, NP-MAML (1) generally performs best with higher performance differences under domain shift.
19

---

Under review as a conference paper at ICLR 2023
H EXTENDED DISCUSSION OF MAML'S MODULES
In the following two paragraphs, we give further insight into some observations from Section
H.1 FINE- TUNING EARLY LAYERS FOR
ADAPTATION
In this section, we discuss the interesting finding that fin-ttuning earlier layers the conv4 substan- tially improves classification accuracy under domain shift. In particular, the domain shift we studied shifts from MinilmageNett to FC100, both containing fairly general image features (Russakovsky et al. 2015). Previous work understanding convolutional networks in the context of image pro- cessing has established the view that early convolutional layers typically learn low-level features like edges and corners, whereas high-level convolutional layers learn high-level features like faces or outlines of various objects (Zeiler & Fergus 2014). We hypothesize that, thus, fine-tuning earlier expect low-level features vary. In the case of
layers becomes more important the more MinilmageNet and FC100, we can observe such a variation due to the fact that FC100 images are substantially smaller (32 32 pixels) than MinilmageNet images (84 84 pixels). As has become customary in cross-domain few-shot-learning images from all domains are typically resized to common shape to fit the model architectures (Vuorio al. 2019 Triantafillou et al., 2020 Oh al., 2021]. The effects of such a resizing on low-level image features can be observed in Figure
We further highlight that conv4 with nonparametric predictor flexible enough deal with dif- ferent image sizes, albeit not with a different number of channels. A study on the performance differences with and without resizing on NP-MAMI. and especially NP-MAMI-44 is suggested as future work.
H.2 POTENTIAL OF NONPARAMETRIE PREDICTORS IN META-LEARNING RESEARCH
Our nonparametric addition to the BOIL algorithm performs favorably compared to parametric so- lutions in cross-domain few-shot classification. Further, established in Section 2.2.2 our method is flexible w.r.t the in- and output dimensionality of given task, making it straightforward to apply in heterogeneous environments. As result, we encourage the use of nonparametric components, especially in meta leeaa-lerrning as it enables architectures to keep up with increasing task diversity (Tn- antafillouu et al., 2020). However, we believe that the potential of rapid learning in MAML goes far beyond the scope of image classification. Consequently, future meta-learning architectures are required to become increasingly flexible without losing ability to rapidly learn new tasks from only few samples. We leave future work to challenge the meta-learning system even further, pos- ing task distributions as mixtures of, e.g., predictions on time series, audio, and speech. We believe that favorable network initialization can be learned that bridges all of those domains.
Further, we recognize the use of Gaussian Processes (Rasmussen & Williams 2006) as another way to form nonparametric predictive distributions over few-shot tasks, which can be employed for both regression and classification. Applying Gaussian Processes to meta-learning has already received attention (Myers & Sardana 2021).
20

---

Under review as a conference paper at ICLR 2023
Figure 3: Image samples from FC100 (top) vs. image samples from MinilmageNet (bottom). Im- ages have been resized to a common 512 x 512 pixels. We observe that FC100 samples are much blurrier when resized, requiring different features for edges, corners, etc.
21


Reviews:
[{"review_id": "e2PHsohHoVG", "reviewer": "Reviewer_nVJV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors investigate MAML's inner loop adaptation dynamics in a variety of different task distributions and find that in some cases, earlier layer inner loop adaptation is key for improvements, i.e. heterogeneous tasks. They also propose a method that builds on MAML that is able to improve performance in heterogeneous tasks, while remaining competitive in the remainder of the tasks.", "summary_of_the_review": "I gave this paper a weak accept due to its interesting empirical insights and novel direction of exploring heterogeneous task distributions, and would be willing to increase my rating if:\n\n- More SOTA models are included in the ablation studies.\n- The proposed method is incorporated on top of existing and suitable SOTA methods to check if there are any benefits once a model is sufficiently strong.\n- Some experiments are ran with the baseline stated in the strengths and weaknesses section, as it would dispel any doubts about how a more 'default' MAML model would perform in heterogeneous settings when it is enabled to do so by learning a quickly adaptable 'single class' weight matrix, that is cloned N times for a given N-way task and fine-tuned on the inner loop.\n- More care is taken to include relevant related work, as it feels like the currently included work includes some historical work, and some more recent papers but misses much of the in-between work that is directly related with what the authors are doing in this paper.", "strength_and_weaknesses": "Strengths:\n\n- Excellent and thorough empirical evaluations in terms of task formulations\n- Interesting empirical insights about MAML\n- Novel proposed method that is both well-motivated and effective\n\nWeaknesses:\n\n- The paper seems to be missing key related work, such as [1, 2, 3] and many others, but those three in particular studied MAML and came up with lots of useful insights that could have been integrated into this work.\n- Possibly a side-effect of the above, the empirical experiments don't seem to incorporate some of the more powerful MAML-related models or ablation with such SOTA models along with the proposed method on top. As a result, it's hard to judge whether the proposed model is unique in its improvement in heterogeneous cases or whether its benefits disappear when used in conjunction with other powerful and existing SOTA models.\n\n*Request: You state that there are 'no straightforward' ways of making MAML heterogeneous, however, what I've seen done, is learning a single 'class' weight matrix, that is cloned N times for an N-class problem, and then fine-tuned in the inner layer -- therefore learning a generic weight matrix that is quickly adaptable for any new N-way task coming in, no matter what N is. I'd love to see that baseline.\n\n\n1. Rusu AA, Rao D, Sygnowski J, Vinyals O, Pascanu R, Osindero S, Hadsell R. Meta-learning with latent embedding optimization. arXiv preprint arXiv:1807.05960. 2018 Jul 16.\n2. Antoniou A, Edwards H, Storkey A. How to train your MAML. arXiv preprint arXiv:1810.09502. 2018 Oct 22.\n3. Antoniou A, Storkey AJ. Learning to learn by self-critique. Advances in Neural Information Processing Systems. 2019;32.\n\nFor additional few-shot papers that could be of interest, to ensure completeness, might be worth looking at\n\n4. Hospedales T, Antoniou A, Micaelli P, Storkey A. Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence. 2021 May 11;44(9):5149-69.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "cgVcsXrliyX", "reviewer": "Reviewer_SRwM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "* This paper proposes an approach to few-shot classification developing on related works such as MAML and Body Only Inner Loop (BOIL).\n* In few-shot learning settings which involve domain shift between the training and evaluation settings, \nmodels necessarily require some adaptation in their parameters in order to be able to solve the testing time tasks. This is because the representations learned by the earlier layers need to change in order to be effective on the new task.\n* The BOIL method approaches this problem by forcing the inner layers of the network to adapt in the inner loop of meta-learning.\n* However, the BOIL method is constrained in that one must pre-determine the number of output classes at training (and therefore at testing time, which must be equal to the number at train time) because this architectural property of the model cannot change once it is trained. This is a limiting factor when seeking to apply few shot models in practice, when testing scenarios may necessarily be different from training scenarios.\n* This work proposes a method to combine a non-parametric classification head with an adaptation procedure for a network body. This allows both the lower level representations of the model adapt and also the number of classes at test time to vary.\n* In evaluation, the paper standardises a setting where a 4-conv architecture (used in the original MAML paper) is trained on MiniImageNet, and then considers various evaluation paradigms. These include: testing on MiniImageNet (in-domain), testing on FC100 (out of domain), varying the number of ways/shots at evaluation time vs training time.\n* The paper also conducts representational similarity analysis on the methods.\n* The key empirical finding is that the proposed strategy can improve performance in settings over BOIL, MAML, and ANIL (uses no inner loop during training for the network body) where domain adaptation and changes in ways/shots between train and test time occur. \n* In settings where one or both of these do not change, the method mostly performs competitively with the other strategies.\n\n", "summary_of_the_review": "Overall, I think this idea has promise. However, there are too many errors in the paper's presentation and a general lack of motivation about the method used for it to be accepted at this stage. I think the paper needs substantial revisions and cleaning up in the methodological presentation, as well as some more comprehensive experiments in the domain-shift setting (with other datasets), before I can recommend it for acceptance. ", "strength_and_weaknesses": "## Major comments\n\nThe paper's key idea to try to make an algorithm like BOIL more flexible by allowing a non-parametric head makes sense and is generally well-motivated. The experimental results are also overall encouraging, though I am not sure why the proposed method performs poorly compared to baselines in Table 3c, in-domain. \n\nHowever, the paper was hard to read and I feel that the main methodological idea was not that well explained or motivated. Here are a concrete set of improvement areas:\n\n* There are a number of typos/lack of careful definitions in the presentation of the mathematical part of the paper. It is possible to decipher some of these, but it makes understanding the method a lot harder. For example: in eqn 9, what is $d$? What is $l$? I think $l$ is a typo for $k$, but this makes following it harder. In Equation 10 and 11, why are we subtracting $c_i$ and $c_j$? Are these meant to be commas (like in equation 9)? What are $d_{\\text{support}}$ and $d_{\\text{query}}$? What is $g_{\\text{body}}(S;\\theta)$?I don't think these are clearly defined enough, and it makes the contribution very hard to follow.\n* The intuition behind the objective in Eqns 10 and 11 was not explained clearly enough. The figure in the appendix helps, but given this is a very important part of the paper, I think it needs more explanation. Why is this particular form of the objective better than something else?\n* Relatedly, the paper says that the BOIL formulation cannot work well with the NIL head. I am not fully sure why this is the case, and the explanation in the paper was not clear enough -- this seems like the simplest baseline to try. Also, even if it doesn't work, it should be evaluated in the experiments as the most obvious baseline method.\n* Given a major part of the study is domain shift, studying only one setting of domain shift from MiniImageNet to FC100 is in my opinion, not sufficient. I think the paper should at least present analysis on Cars, CUB, or something like that from the BOIL paper. In addition, it would be good to evaluate methods at least overall on Meta-Dataset, given it's a more realistic testbed for FSL methods.\n* Unless I missed it, Table 3 doesn't get much attention in the text, and I would be curious to know more about why the method does worse in the in-domain setting here. The performance deltas in Table 3c, MiniImageNet are quite substantial.\n\n\n## Minor\nSome other typos/suggestions:\n* BOIL could use a citation in the intro.\n* In eq 8, the transpose operator and leading dimension are both using $T$. It's better to use a different letter for the dimension, or change the way the transpose is typeset.\n* Details on proto-MAML are not very clear -- what is the subscript $k$? \n* End of section 2.2.2: typo when defining the last convolutional block?\n* Table 1 is missing a heading -- I think the FC100 heading is in the wrong place, and MiniImageNet heading is missing. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "EtSN82VwIV", "reviewer": "Reviewer_gDT4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Traditional MAML is well used in few-shot learning domains. However, recent works show that MAML does not perform well in a fast adaptation. This paper gives a detail about feature reuse phenomenon and found the relationship between lower layer and the distribution differences in training and testing.", "summary_of_the_review": "I think this method needs to be compared to SOTA-embedding few-shot learning methods for verifying the effectiveness of MAML + prototypical network not the BOIL and ANIL which are using a fully-connected layer.", "strength_and_weaknesses": "Pros\n\nIn this work, the problem of MAML is divided into two main challenges including algorithmic challenge and architectural challenge . algorithmic challenge represents the performance of cross domain under the domain shift and architectural challenge is related to the model structure such as pixel input and the dimension of the output. These two challenges are reasonable and the results seem good in the experiment section.\n\nCons\n\nNP-MAML updates the body layer with a distance using prototypes. In this respect, for the fair comparison, the architectural challenge needs to be compared with embedding methods and it is similar to embedding few-shot learning methods such as a prototypical network. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9qLB5HFrrxS", "reviewer": "Reviewer_wHQT", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new variant of the MAML algorithm, called \"NP-MAML\" for non-parametric MAML. The method uses a non-parametric head (final layer) using distances to a prototype to assign labels to new data, while still updating the body of the network in the meta-learning inner loop. The paper compares this method to related methods (BOIL/ANIL/MAML) on few-shot image classification tasks.", "summary_of_the_review": "Overall, I thought this was a nice and interesting modification to MAML. I think the paper could be improved by demonstrating the performance of NP-MAML on additional tasks, as well as providing more intuition for what types of changes are happening in the representation (extending the CKA analysis—although perhaps this latter point is better left for future work).", "strength_and_weaknesses": "Strengths:\n- Simple idea, well motivated\n- Clear experiments that support the main conclusions of the paper\n\nWeaknesses:\n- Would be nice to see experiments on other domains (besides images)\n- The representation similarity experiments are intriguing, but could be further developed\n\nMinor comments:\n- In the introduction, the BOIL acronym is introduced in the fourth paragraph without explaining what it is.\n- Missing header in Table 1", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject