The Curse of Low Task Diversity: On the Failure of Transfer Learning to Outperform MAML and their Empirical Equivalence
Authors: Brando Miranda
Patrick Yu
Yu-Xiong Wang
Oluwasanmi O Koyejo
Venue: Submitted to ICLR 2023 (2023)
Keywords: meta-learning
machine learning
transfer learning
deep learning

Abstract: Recently, it has been observed that a transfer learning solution might be all we need to solve many few-shot learning benchmarks -- thus raising important questions about when and how meta-learning algorithms should be deployed. 
In this paper, we seek to clarify these questions by 
1. proposing a novel metric -- the {\it diversity coefficient} -- to measure the diversity of tasks in a few-shot learning benchmark and 
2. by comparing MAML and transfer learning under fair conditions (same architecture, same optimizer and all models trained to convergence).
Using the diversity coefficient, we show that the popular MiniImagenet and Cifar-fs few-shot learning benchmarks have low diversity. 
This novel insight contextualizes claims that transfer learning solutions are better than meta-learned solutions in the regime of low diversity under a fair comparison. 
Specifically, we empirically find that a low diversity coefficient correlates with a high similarity between transfer learning and Model-Agnostic Meta-Learning (MAML) learned solutions in terms of accuracy at meta-test time and classification layer similarity (using feature based distance metrics like SVCCA, PWCCA, CKA, and OPD). 
To further support our claim, we find this meta-test accuracy holds even as the model size changes. 
Therefore, we conclude that in the low diversity regime, MAML and transfer learning have equivalent meta-test performance when both are compared fairly.
We also hope our work inspires more thoughtful constructions and quantitative evaluations of meta-learning benchmarks in the future.

Under review as a conference paper at ICLR 2023
THE CURSE OF Low TASK DIVERSITY: ON THE FAILURE OF TRANSFER LEARNING TO OUTPERFORM MAML AND THEIR EMPIRICAL EQUIVALENCE
Anonymous authors Paper under doublc-blind review
ABSTRACT
Recently, has been observed that a transfer learning solution might be all we need to solve many few-shot learning benchmarks - thus raising important questions about when and how meta-learning algorithms should be deployed. this paper, we seek clarify these questions by 1. proposing novel metric the diversity coefficient - to measure the diversity of tasks a few-shot learning benchmark and 2. by comparing Model-Agnostic Meta-Learning (MAML) and transfer learning under fair conditions (same architecture, same optimizer, and all models trained convergence). Using the diversity coefficient, we show that the popular Mini- ImageNet and CIFAR-FS few-shot learning benchmarks have low diversity. This novel insight contextualizes claims that transfer learning solutions are better than meta-learndd solutions in the regime of low diversity under a fair comparison. Specifically, we empirically find that low diversity coefficient correlates with a high similarity between transfer learning and MAMI. learned solutions in terms of accuracy meta-test time and classification layer similarity (using feature based distance metrics like SVCCA, PWCCA, CKA, and OPD). To further support our claim, we find this meta-test accuracy holds even as the model size changes. There- fore, conclude that in the low diversity regime, MAML and transfer learning have equivalent meta-test performance when both are compared fairly. We also hope our work inspires more thoughtful constructions and quantitative evaluations of meta-learning benchmarks in the future.
INTRODUCTION
The success of deep learning in computer vision (Krizhevsky et al.) 2012 He et al. 2015), natural language processing (Devlin al. 2018 [Bronn et al.) (020)), game playing (Silver et al., 2016 Mnih et al. 2013 Ye et al. 202T). and more keeps motivating a growing body of applications increasingly wide variety of domains. In particular, deep learning is now routinely applied to shw-shot learning a research challenge that assesses a model's ability to learn tasks, new distributions,
of deep learning
environments. This has been the main research area where meta-learning algorithms have been applied - since such a strategy seems promising in a small data regime due its potential learn to learn or learn adapt. However, was recently shown (Tian et al. 2020) that a transfer learning model with a fixed embedding can match and outperform many modern sophisticated meta-learning algorithms on numerous few-shot learning benchmarks (Chen et al. 2019 2020 Dhillon et al., 2019, Huang and Tao 2019). This growing body of evidence coupled with these surprising results meta- leeaa-lernnig-- raises the question researchers are applying meta-learning with the right inductive biases (Mitchell] 1980; Shai Shalev-Shward
to adapt
2014) and designing appropriate benchmarks for meta-learning. Our evidence suggests this is not the
Our work is motivated by the inductive bias that when the diversity of tasks in a benchmark is low then meta-learning solution should provide no advantage to minimally meta-learned algorithm- e.g. like only fine-tuning the final layer. Therefore in this work, we quantitatively show that when the task diversity novel measure of variability across tasks is low, then MAML (Model-Agnostic Meta-Learning) (Finn et al.) 2017) learned solutions have the same accuracy as transfer learning (i.e., supervised learned model with fine-tuned final linear layer). We want emphasize the

---

Under review as a conference paper at ICLR 2023
importance of doing such an analysis fairly: with the same architecture, same optimizer, and all models trained to convergence. We hypothesize this was lacking in previous work (Chen et al. 2019 2020; Dhillon et al. 2019 Huang and Tao 2019). This empirical equivalence remained true even as the model size changed thus further suggesting this equivalence is more a property of the data than suggest taking a problem-centric approach to meta-learning and suggest
of the model. Therefore, applying Marr's 's level of analysis (Hamrick and Mohamed] 2020, Marr 1982) to few-shot learning to identify the family of problems suitable for meta-learning. Marr emphasized the importance of understanding the computational problem being solved and not only analyzing the algorithms or hardware that attempts to solve them. An example given by Marr is marveling the rich structure of bird feathers without also understanding the problem they solve is flight. Similarly, there has been analysis of MAML solutions and transfer learning without putting the problem such solutions should solve into perspective (Raghu et al. 2020 Tian et al.) (020). Therefore, in this work, we hope to clarify some of these results by partially placing the current state of affairs in mnca-learning from moblem--cnntric view. In addition, important novelty of our analysis is that we put analysis of intrinsic properties of the data as the driving force.
Our contributions are summarized as follows:
1. We propose a novel metric that quantifies the intrinsic diversity of the data of a few-shot learning benchmark. We call it the diversity coefficient. It enables analysis of meta-learning algorithms through a problem-centric framework. also goes beyond counting the number of classes number of data points counting the number of concatenated data sets and instead quantifies the expected diversity/variabbiity of tasks in a few-shot learning benchmark. We also show it's strong correlation with the ground truth diversity.
2. We analyze the two most prominent few-shot learning benchmarks Minilmagenet and Cifar-fs and show that their diversity is low. These results are robust across different ways to measure the diversity coefficient, suggesting that our approach is robust. In addition, we quantitatively also show that the tasks sampled from them are highly homogeneous.
3. With this context, we partially clarify the surprising results from (Tian et al., 2020) by comparing their transfer learning method against models trained with MAML (Finn et al.) 2017). In particular, when making a fair comparison, the transfer learning method with a fixed feature extractor fails to ousperform MAML. We define fair comparison when the two methods are compared using the same architecture (backbone), same optimizer, and all models trained to convergence. We also show that their final layer makes similar predictions according to neural network distance techniques like distance based Singular Value Canonical Correlation Analysis (SVCCA), Projection Weighted (PWCCA), Linear Centered Kernel Analysis (LINCKA), and Orthogonal Procrustes Distance (OPD). This equivalence holds even as the model size increases.
4. Interestingly, we also find that even in the regime where task diversity is low (in Minilma- genet and Cifar-fs), the features extracted by supervised learning and MAML are different - implying that the mechanism by which they function is different despite the similarity of their final predictions.
5. As an actionable conclusion, we provide a metric that can be used to analyze the intrinsic diversity of the data in shot learning benchmarks and therefore build more thoughtful environments to drive research in meta-learning. following test to predict the empirical equivalence of MAML and transfer learning: if the task diversity is low, then transfer learned solutions might fail to outperform meta-learned solutions. This test is easy to run because our diversity coefficient can be done using the Task2Vec method (Achille UCLA et al., 2019) using pre-trained neural network. In addition, according our synthetic experiments that also test the high diversity regime, this test provides preliminary evidence that the diversity coefficient might be predictive of the difference in performance between transfer learning and MAML.
addition, our evidence suggests the
We hope that this line of work inspires a problem- centric first approach to meta-learning - which appears to be especially sensitive to the properties of the problem in question. Therefore, hope future work takes a more thoughtful and quantitative approach to benchmark creation - instead of focusing only on making huge data sets.
2

---

Under review as a conference paper at ICLR 2023
2 BACKGROUND
In this section, we provide a summary of the background needed to understand our main results.
Model-Agnostic Meta-Learning (MAML): The MAML algorithm (Finn et al., (017)) attempts to meta-learn an initialization of parameters for neural network so that it is primed for fast gradient descent adaptation. It consists of two main optimization loops: 1) an outer loop used to prime the parameters for fast adaptation, and 2) an inner loop that does the fast adaptation. During meta-testing, only the inner loop is used to adapt the representation learned by the outer loop.
Transfer Learning with Union Supervised Learning (USL): Previous work (Tian et al. 2020) shows that initialization trained with supervised learning, on union of all tasks, can outperform many sophisticated methods in meta-learning. In particular, their method consists of two stages: 1) first they use union of all the labels in the few-shot learning benchmark during meta-training and train with standard supervised learning (SL), then 2) during the meta-testing, they use an inference method common in transfer learning: extract a fixed feature from the neural network and fully fine-tune the final classification layer (i.e., the head). Note that our experiments only consider when the final layer is regularized Logistic Regression trained with LBGFS (Limited- -Limited-memory memory algorithm).
Hellinger Distance for distance between distributions: When the ground truth distribution for a class e is known p" (zz y/c) one can compute the true distance between classes. Therefore one can compute the ground truth statistics on the distribution of distances classes. The distribution diversity coefficient we propose is the expectation of the distance between Hellinger distances between distributions (explained in more detail in section 3.1. We review the standard Hellinger distance in the supplementary section[]
Task2Vee Embeddings for Distance computation between Tasks:
The diversity coefficient we propose is the expectation of the distance between tasks (explained in more detail in section B) Therefore, it is essential to define the distance between different pairs of tasks. We focus on the cosine distance between Task2 Vec (vectorial) embeddings in (Achille UCLA et al. 2019). Therefore, provide summary of the Task2 method to compute task embeddings. The vectorial representation of tasks provided by Task2Vec (Achille UCLA et al. 2019) is the vector of diagonal entries of the Fisher Information Matrix (FIM) given a fixed neural network as a feature extractor also called a probe network- after fine-tuning the final classification layer the task. The authors explain this is good vectorial representation of tasks because 1. It approximately indicates the most informative weights for solving the current task (up a second order approximation) For rich probe networks like CNNs, the diagonal is more computationally tractable. The Task2Vec embedding of task is the diagonal of the following matrix:
FD..fo = F(D,, fw) = Bww.sr-aaalrllllll...... V log x. fw) V P x. fw)
(1)
where fur is the neural networks used a feature extractor with architecture and weights w, p(x r) is the empirical distribution defined by the training data D, = {( v:)) for task r, and ply x. fw) is a deep neural network trained to approximate the (empirical) posterior p(y xx T). We'd d like to emphasize that the dependence on target label since Task2 Vec fixes the feature extractor (using fw) and then fits the final layer (or "head") approximate the task posterior distribution ply x, r). In addition, it's important to have a fixed probe network to make different embeddings comparable (Achille UCLA al. 2019
THE DIVERSITY COEFFICIENT
The diversity coefficient aims to measure the intrinsic diversity (or variability) of tasks in few-shot learning benchmark. In this section we introduce two possible data centric definitions and explain their value.
3.1 DISTRIBUTIONAL DIVERSITY COFFFICIENT (WITH HELLINGER DISTANCES)
At a high level, the distributional diversity coefficient is the expected distance between a pair of distributions for different classes. Using the Hellinger distance dæ the definition is:
divn (B) EhteseCee (dn (p" (ar,y) y c).p' (s. y c())
(2)
3

---

Under review as a conference paper at ICLR 2023
where pa (zz, c) is the distribution of data for class and Class is the set of allowed classes. This formulation useful if you know the ground truth distribution or have estimates of the distribution.
3.2 TASK BASED DIVERSITY COEFFICIENT (WITH TASK2VEC)
At high level, the diversity coefficient is the expected distance between pair of different tasks given a fixed probe network. In this work, we choose the distance to be the cosine distance between vectorial representations (i.e. embeddings) of tasks according Vec as described in section 2 We define the diversity coefficient of a few-shot learning benchmark B as follows:
div( B) = Ber ( -Di-p(rnsuln) (n).- Du-pelra)) [d(Foysur PDa.fa)
where fw is the neural networks used feature extractor with architecture and weights w, p(x | T) is the empirical distribution defined by the training data D, {(T,, B:)] for task т, Th, 72 tasks sampled from the empirical distribution of tasks p(r B) for the current benchmark B (i.e. batch of tasks with their data sets D = (Ti, Dr.))=))) ) a task T; the probability distribution p(a, r) of the data, d is a distance metric (for cosine), fw is the neural networks used as a feature extractor with architecture f and weights w. and p(x 7) is the empirical distribution defined by the training data D, {z 3/ |i= for task 7. We'd d also like to recall the reader that the definition of a task in this setting is of a n-way, k-shot seo- learning task. Therefore, each task has classes sampled with it examples used for the adaptation. Note that in this setting combine the support and query set as the split is not relevant for the computation of the task embedding using Task2
3.3 JUSTIFICATION FOR TASK BASED DIVERSITY CORFFICIENT WITH TASK2VEC
We justify the use of Task2 because of it's well-founded justification as an approximation for task and its impressive properties from previous work (Achille UCLA et al. 2019).
Previous work showed Vec is a strong approximation to task because: 1. it's able of predicting task similarities that match human intuitions about taxonomic and semantic relations between different visual tasks classes for similar plants exhibit non-trivial clustering (Achille UCLAA et al. 2019) 2. when taxonomical distance are available they find that the embedding distance correlates positively 3. since Vec is based on the FIM which is a Riemannian metric on the space of probability distributions (Amari and Nagaoka 2007) - implies distances based on Task2Vec are good approximations to the ground truth distance of task distribution. We confirm this ground truth strong correlation (with Pearson r = 0.990) in the rightmost plot of figure 3
In addition, Tesk2Vec exhibits impressive properties that strongly suggestive that it also captures amount of information about a task. Therefore, more diverse benchmark contains more information which increases the value of the task based diversity coefficient. Some of these properties are: 1. the ability to use Tesk2Vec to select the close to best pre-trained features extractor close to the ground truth optimal expert while costing substantially less (Achille UCLA et al., 2019). 2. the correlation of the (nuclear) norm of the Task2Vec embedding with the difficulty of the task and test performance.
In addition, Task2Vec (through FIM) is related the (Kolmogorov) complexity of task (Achille et al. 2018).
We want to emphasize one of the main novelties of the task based diversity coefficient is its use of an extensively tested Tas embedding in non-trivial ways. In particular the correction of the common belief that USL better meta-learner than MAML. in the low diversity regime explained in detail in the experiments section 4
EXPERIMENTS
This section explains the experiments backing up main results outlined in our list of contributions. Experimental details are provided the supplementary section and the learning curves displaying the convergence for fair comparison are in supplementary section
4

---

Under review as a conference paper at ICLR 2023
4.1 THE DIVERSITY COEFFICIENT OF MINIIMAGENET AND CIFAR-FS
To put our analysis into a problem-centric framework, we first analyze the problem to be solved through the lenses of the diversity coefficient. Recall that the diversity coefficient aims quantify the intrinsic variation (or distance) of tasks in a few-shot learning benchmark. We show that the diversity coefficient of the popular Minilmagenet and Cifar-fs benchmarks are low with good confidence intervals using four different probe networks in table [ We argue it's low because the diversity values are approximately in the interval [0.06, 0. 1.117] given that the minimum and maximum values would be 0.0 and 1.0 for the cosine distance. In addition, the individual distances between pairs of tasks are low and homogenous, as shown in the heat maps in the supplementary section K. figures 13 and
Probe Network
Diversity on MI
Diversity on Cifar-fs
Resnet18 8 (pt)
0.117 = 2
0.100 + 2.18e
Resnet18 (rand)
0.0955 + 1.29e-5
0.103 + 1.05e-5
Resnet34 (pt)
0.0999 + 1.95e-5
0.0847 + 3.06e-5
Resnet34 (rand)
0.0620 8.12e-6
0.0643 + 9.64e-6
Table I:
The diversity coefficient of Minilmagenet (MI) and Cifar-fs is low. The diversity coefficient was computed using the cosine distance between different standard 5-way, 20-shot classification tasks from the few-shot learning benchmark using the Task2Vec method described in section 3 We used 20 shots (number of examples per class) since we can use the whole task data to compute the diversity coefficient (no splitting of support and query set required for the diversity coefficient). We used Resnet18 and Resnet34 networks as probe networks both pre-trained on ImageNet (indicated as "pt" on table) and randomly initialized (indicated as "rand" on table). We observe that both type of networks and weights give similar diversity results. All confidence intervals were at 95%. To compute results, we used 500 few-shot learning tasks and only compare pairs of different tasks. This results in (500² /200//2- 124, 750 pair-wise distances used to compute the diversity coefficient.
4.2 Low DIVERSITY CORRELATES WITH EQUIVALENCE OF MAML AND TRANSFER LEARNING
Now that we have placed ourselves in problem-centric framework and shown the diversity coefficient of the popular Minilmagenet and Cifar-fs benchmarks are low we proceed to show the failure of transfer learning (with USL) to outperform MAML. Crucially, the analysis was done using fair comparison: using the same model architecture, optimizer, and training all models to convergence details in section D We used the five-layer CNN used in (Finn et al.) 2017 Ravi and Larochelle] 2017) and Resnet 2 as (Tian et al. 2020). We provide evidence that in the setting of low diversity:
1. The accuracy of adapted MAML meta-leamer vs. adapted USL pre-trained model similar and statistically significant, except for one result where transfer learning with USL is worse. This is shown in 2able|]and]]
2. The distance for the classification layer decreases sharply according to four distance-based metrics- - SVCCA, PWCCA, LINCKA, and OPD as shown in figure 10 This implies the predictions of the two are similar.
For the first point, we emphasize that tables I and Itken taken together support our central hypothesis: that models trained with meta-learning are not inferior transfer learning models (using USL) when the diversity coefficient is low. Careful inspection reveals that the methods have the same meta-test accuracy with intersecting confidence intervals making the results statistically significant across few-shot benchmarks and architectures. The one exception is the third set of bar plots, where transfer learning with USL is in fact worse.
For the second point, refer to figure 10 and observe that the depth of the network increases, the distance between the activation layers of a model trained with MAML vs USL increases until reaches the final classification layer- where all four metrics display noticeable dip. In particular, PWCCA considers the two prediction layers identical (approximately zero distance). This final point is particularly interesting because PWCCA is weighted according to the CCA weights that stabilize
5

---

Under review as a conference paper at ICLR 2023
with the final predictions of the network. This means that the PWCCA distance value is reflective of what the networked actually learned and gives a more reliable distance metric (for details, refer to the appendix (o.)ion[..3). This is important because this supports main hypothesis: that at prediction time there is an equivalence between transfer learning and MAML. when the diversity coefficient is low.
Meta-train Initialization
Adaptation at Inference
Meta-test Accuracy
Random
adaptation
19.3 = 0.80
MAMLO
no adaptation
20.0 + 0.00
USL
adaptation
15.0 + 0.26
Random
MAML.5 adaptation
34.2 1.16
MAML5
MAML5 adaptation
62.4 + 1.64
USL
MAML5 adaptation
25.1 + 0.98
Random
MAML10 adaptation
34.1 + 1.23
MAMLS
MAML10 adaptation
62.3 + 1.50
USL
MAML10 adaptation
25.1 + 0.97
Random
Adapt Head only (with LR)
40.2 + 1.30
MAML5
Adapt Head only (with LR)
59.7 = 1.37
USL
Adapt Head only (with LR)
60.1 + 1.37
Table 2: MAML trained representations and supervised trained representation have statisti-
cally equivalent meta-fest accuracy on Minilmagenet which has low diversity. The transfer model's adaptation is labeled as "Adapted Head only (with LR)" which stands for "Logistic Regres- sion (LR)" used in
Tian et al. 2020 For experimental details see section D33
Performance Comparison MAML vs Transter Learring (USL)
MARL
USL
80-
00-
SCNN
Raweet2
Chieni, SCHN Difants TasM6l12
Delesel, Arthoiduure
Figure 1: MAML trained models and union supervised trained (USL) models have statistically equivalent meta-test accuracy for Minilmagenet and Cifar-fs with Resnet12 and five layer CNNs. This holds for both the Resnet12 architecture used in (Tian al. 2020) and the 5 layer CNN (indicated as "5CNN") in (Ravi and Larochelle] 2017). Results used a (meta) batch-size of 100 tasks and 95% confidence intervals. All MAML models were trained with inner steps during meta-training. "MAML5" and "MAML10" in the bar plot indicates the adaptation method used at test time iee we used 5 inner steps and 10 inner steps at test time. Minilmagenet is abbreviated as "MI" in the figure.
4.3
IS THE EQUIVALENCE OF MAML AND TRANSFER LEARNING RELATED TO MODEL SIZE OR Low DIVERSITY?
An alternative hypothesis to explain the equivalence of transfer learning (with USL) and MAML could be due to the capabilities of large neural networks to be better meta-learners in general. Inspired by the impressive ability of large language models to be few-shot (or even zero-shot) learners (Brown) jet al.) 2020; Bommasani et al.) 2021 Radford et al. 202T Devlin et al. 2018) we hypothesized that perhaps the meta-learning capabilities of deep learning models is function of the model size. If this

---

Under review as a conference paper at ICLR 2023
true, then expected larger for smaller models and the difference to decrease as the model size increased. Once the two models were, of the same size but large enough, hypothesized that the meta-test accuracy would be the same. We tested this to rule out that our observations were a consequence of the model size. negative and surprisingly the equivalence between MAMI. and USL seems hold
the difference in meta-test accuracy of MAML and USL to be
The results even as the model increased - strengthening our hypothesis that the low task diversity might be bigger factor explaining our observations. We show this in figure 2 and we to the fact this statistical equivalence holds even when using only four filters the case where we expected the biggest difference.
draw attention
Modal Size Effect Performanc Difference of vs MAML
Model Sie Effect Performance Offference USI MAM
MAPES Arc
NAMLDO USL Adi
0:35
0250
0323
0.325
4450 450
0.00
0.300
0.43
0.475
0.00
0.450
0445
0.40
20
20
Figure 2:
The meta-test accuracy of MAML and transfer learning using USL is similar in statistically significant way regardless of the model size. In this experiment, used the Minilmagenet benchmark, the five layer CNN used in (Finn et al.] 2017) Ravi and Larochelle (017). and only increased the filter size using sizes 4, 8, 16, and We made using the same architecture, optimizer, and trained all models to convergence. During meta-training, the MAML model was trained using inner steps. The legends indicating MAM15 and MAMLI0 refer to the number of inner steps used at test time. We used a (meta) batch size of 100 tasks.
the comparison was fair by
4.4 MAML LEARNS DIFFERENT BASE MODEL COMPARED TO UNION SUPERVISED LEARNED MODELS EVEN IN THE PRESENCE OF LOW TASK DIVERSITY
The first four layers of figure 10 shows how large the distance is of MAML representation compared to SL representation. In particular, is much larger than distance value in the range [0. 0.1] from previous work that compared MAML vs. adapted MAML (Raghu al. 2020). We reproduced that and indeed MAML vs. adapted MAML has a small difference (smaller for us) supporting our observations that MAML layer even when the diversity is low. Results are statistically significant.
USL learned representations are different at the feature extractor
4.5 SYNTHETIC EXPERIMENTS SHOWING CLOSENESS OF MAMI AND TRANSFER LEARNING AS DIVERSITY CHANGES
In this section, we show the closeness of MAML and transfer learning (with USL) for synthetic experiments for low and high diversity regimes in Figure 3 In the low regime, the two methods are equivalent in a statistically sigmificant way - which supports the main claims of our paper. As the diversity increases, however, the difference between USL and MAMI. increases (in favor of USL.). This will be explored further in future work.
The tasks are the usual n-way, k-shot tasks, but the data comes from Gaussian and the meta-learners are tasked with classifying from which Gaussian the data points came from in a few-shot learning manner. Benchmarks are created by sampling Gaussian distribution with means moving away from the origin as the benchmark changes. Therefore, the Gaussian benchmark with the highest diversity coefficient has Gaussians that the furthest from the origin. We computed the task diversity coefficients using the Vec method as outlined in Section using a random 3-layer fully connected probe network described in Section
7

---

Under review as a conference paper at ICLR 2023
Dfference MAM.
Stversty Efed Performance Dffenncce NAML
Tisk2ve; Diversity Synthelic Gouss
0.59
0.43
04
0.30
69
333
3.3
0.0
0.00
2.13
0.00
825
6.50
8.16
0.43
8.58
0.3
0.2
0.4
08
0.5
1.9
Diancity
Figure 3: Left and center plots show the meta-test accuracy of MAML and transfer learning using USL is similar in a statistically equivalent way in the low diversity regime in the 5-way, 10-shot Gaussian Benchmarks. The right most figure shows the strong correlation between Task2Vec diversity and the ground truth Hellinger Diversity. Results used a (meta) batch-size of 500 tasks, 95% confidence intervals and the Pearson value was r r= 0.990. MAML5 and MAMLIO indicate adaptation at test time of the MAML5 trained model.
4.6 CAN WE TRUST THE COMPUTED DIVERSITY COFFFICIENT INDEED REFLECTS THE DIVERSITY OF BENCHMARK?
In this section, we argue that indeed the diversity is low and valid because: 1. the task based diversity using Tesk2 reflects the true diversity as argued in section3.3.2. our results are robust to the choice of the probe network because Tesk2Vec depends solely on the task, and ignores interactions with the model (Achille UCLA et al. 2019)- implying our results are valid regardless of the probe network 3. despite the evidence provided by the authors of Task2Vec (Achille UCLA et al. (019)) nevertheless tested this dependence and used 4 probe networks to compute the task based diversity coefficient the choice of using the expectation was empirically based on the bell shape of histogram of
distances of tasks (via Task2Vec) in figures 15 16 They were Gaussian shaped so the expectation was a sufficient statistic. 5. Although the apparently large number of tasks based on combinatorial arguments (e.g. 45 = 7624512) 500 samples are enough to make strong statistical inferences about the population. If we assume the distribution of the data is Gaussian, then we expect to see a single mode with an approximate bell curve. If we plot the histogram of task pair distances of the 500 tasks and see this then we can infer our Gaussian assumption is approximately correct. Given that we do see that in figure 15
16 17 then can infer our assumption is approximately correct. This implies we can make strong statistical assumptions about the population - in particular, that we have good estimate of the diversity coefficient using 500 samples. 6. The heat maps for Minilmagenet and cifar-fs were homogeneous (uniform color) as shown in figures 14 12
5 RELATED WORK
Our work proposes a problem-centiie framework for the analysis of meta-learning algorithms inspired from previous puzzling results (Tian et al. 2020). We propose the use of a pair-wise distance between tasks and analyze how this metric might correlate with meta-leamning The closest line of work for this is the long line of work by (Achille UCLA et al. 2019) where they suggest methods to analyze the complexity of a task, propose unsymmetrical distance metries for data sets, reachability of tasks with SGD, ways to embed entire data sets and more (Achille UCLA et al. | Achille et al.) 2018 2019 2020). We hypothesize this line of work to be very fruitful and hope that more people adopt tools like the ones they suggest and we propose in this paper before researching or deploying meta-learning algorithms. We hope this helps meta-learning methods succeed in practice since cognitive science suggests meta-learning is powerful method humans use to learn (Lake et al. 2016)). In the future, we hope to compare (Achille UCLA et al. 2011))* 's distance metrics between tasks with ours to provide a further unified understanding of meta-learmin and ours is that we focus our analysis from a meta- meta-learning perspective applied to few-shot learning - while their focus is understanding transfer learning methods between data sets.
and transfer learning. A contrast between their work
8

---

Under review as a conference paper at ICLR 2023
Our analysis of the feature extractor layer is identical to the analysis by (Raghu et al. 2020). They showed that MAML functions mainly via feature re-use than by rapid learning i.e., that model trained with MAML changes very little after the MAML adaptation. The main difference of their work with our is: 1) that we compare MAML trained models against union supervised learned models (USL.) instead of only comparing MAMI against adapted MAMI., and 2) that we explicitly analyzed properties of the data sets. In addition, we use a large set of distance metrics for our analysis
including: SVCCA, PWCCA, LINCKA and OPD as proposed by (Raghu al. 2017 Morcos et al. 2018 Korbblith et al., 2019 Ding et al. 2021].
Our work most influenced by previous work suggesting modern meta-learning requires rethinking (Tian et al. (020). The main difference of our work with theirs is that we analyzed the internal representation of the meta-learning algorithms and contextualize these with quantifiable metries of the problem being solved. Unlike their work, we focused on fair comparison between meta-learning methods by ensuring the same neural network backbone was used, Another difference is that they gained further accuracy gains by using distillation method we did not analyze and leave for future work.
Another related line of work is the predictability of adversarial transferability and transfer learning. They show this both theoretically and with extensive experiments (Liang et al.) 202I). The main difference between their work and ours is that they focus their analysis mainly transfer learning, while we concentrated on meta-learning for few-shot learning. In addition, we did not consider adversarial transferability- while that was a central piece of their analysis. Further, related work is outlined in the supplementary section
DISCUSSION
In this work, we presented a problem-centric framework when comparing transfer learning methods with meta-learning algorithms- using USL and MAML as the representatives of transfer and meta- learning methods respectively. We showed the diversity coefficient of the popular Minilmagenet and Cifar-fs benehmark is low and that under fair comparison MAML is very similar transfer learning (with USL.) at test time. This was also true even when changing the model size removing the alternative hypothesis that the equivalence of MAMI, and transfer learning with USI. held due to large models. Instead, this strengthens our hypothesis that the diversity of the data might be the driving factor. The equivalence of MAML and USL was also replicated in synthetic experiments. Therefore, we challenge the suggestions from previous work Tian et al. (2020) that only good embedding can beat more effective than sophisticated meta-learning especially in the low diversity regime, and instead suggest this observation might be due to lack of good principles to design meta- learning benchmarks. In addition, our synthetic experiments show promising scenario where we can systematically differentiate meta-learning algorithms from transfer learning algorithms- which supports our actionable suggestion to use the diversity coefficient to effectively study meta-learning and transfer learning algorithms. In addition, this problematizes the observations that fo-MAML in meta-data set (Triantafillou et al., 2019) is better than transfer learning solutions since our synthetic experiments show MAML. is not better than USL in the high diversity regime. To further problematize, we want to point out that meta- meta-learning methods are not better than transfer learning as observed by (Guo et al. 2019) as observed in our synthetic experiments. Meaning that further research is needed in both data sets especially from problem-centric perspective with quantitative methods like the ones we suggest.
We also have theoretical results from statistical decision perspective in the supplementary section M that inspired this work and suggest that when the distance between tasks is zero - then the predictions of transfer learning, meta-learning, and even fixed model with no adaptation are all equivalent (with the loss).
We hope this work inspires the community in meta-learning and machine learning to construct benchmarks from a problem-centric perspective - that go beyond only large scale data sets and instead quantitative metrics for the construction of such research challenges.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Alessandro Achille, Glen Mbeng, and Stefano Soatto. The Dynamic Distance Between Learning Tasks: From Kolmogorov Complexity to Transfer Learning via Quantum Physics and the Information Bottleneck of the Weights of Deep Networks. 2018. URL abs/1810..00 02440
org/
Alessandro Achille, Glen Bigan Mbeng, and Stefano Soatto. Dynamics and Reachability of Learning Tasks. 2019.
Alessandro Achille, Giovanni Paolini, Glen Mbeng, and Stefano Soatto. The Information Complexity of Learning Tasks, their Structure and their Distance. 2020.
Alessandro Achille UCLA, Michael Lam AWS, Rahul Tewari AWS, Avinash Ravichandran AWS, Subhransu Maji UMass, Stefano Soatto UCLA, and Pietro Perona Caltech. TASK2VEC: Task Embedding for Meta-Learning Charless Fowlkes UCI and AWS. Technical report, 2019.
Shunichi Amari and Hiroshi Nagaoka. Methods of information geometry. 191, 4 2007. doi:
10.1090/M0NO/1111 URL httt::tt///wwt ama..amm..... otm. wawgg//moo///// 191
Sébastien M R "Amold, Praateek Mahajan, Debajyoti Datta, Ian Bunner, and Konstantinos Saitas"
Zarkias. "leam2leeann A library for Meta-Learning research". aug 2020. URI. "http://arx:
org/abs/ 2008. 12284
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho. Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Kohd, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin Thomas, Florian Tramèr, Rose E. Wang, and William Wang. On the Opportunities and Risks of Foundation Models. aug 2021. htt.:httt://aaaa
org/ abs 2 1 108. 0 07258v2
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Kerbert-Vosss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam Mccandlish, Alec Radford, Ilya Sutskever, and Dario Amodei Openai. Language Models Few-Shot Learners. Technical report, 2020.
Qi Chen, Changjian Shui, and Mario Marchand. Generalization Bounds For Meta- Meta-Learning: An Information- Theoretic Thhoreition-Thheretiiic Analysis. sep 2021. doi: 24445550/arxii211094555555 URL https
//arx//. //axxiv.org/abs/2100. 14595v2
Wei-Yu Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A Closer Look at Few-shot Classification. 7th International Conference on Learning Representations, ICLR 2019, 2019. URL BtLltttpp//arxii org/ abs 1904.04233
Yinbo Chen, Xiaolong Wang, Zhuang Liu, Huijuan Xu, and Trevor Darrell. A New deta-Baseline for Few-Shot Learning. Technical report, 2020. URL hittpa://githbbe com/
10

---

Under review as a conference paper at ICLR 2023
François Chollet. On the Measure of Intelligence. 2019. [BLLLLLp:///xxxx..2/////22 01547
Tristan Deleu, Tobias Würfl, Mandana Samiei, Joseph Paul Cohen, and Yoshua Bengio. Torchmeta: A Meta-Learning library for PyTorch, 2019. http///ccooss.eaaa.....////////.
Available at: https://github.co http://hww...coom/iissmee///////
Giulia Denevi, Massimiliano Pontil, and Carlo Ciliberto. The Advantage of Conditional Meta-
Learning for Biased Regularization and Fine-Tuning Advances in Neural Information Processing Systems, 2020- D000-beeeeeer aug 2020. ISSN 10495258. doi: 10448550/axiv:2008.10888 URL
atttss://axxie org/ 2008. 10857v1
Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training Deep Bidirectional Transformers for Language Understanding. NAACL HLT 2019 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies Proceedings of the Conference, 1:4171 4186, oct 2018. URL httph:tts//attt
org/ abs 181081110000 04805v2
Guneet S. Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A Baseline for Few-Shot Image Classification. 2019. UtK.:ptt:///arrii.ooo//////10000
Frances Ding, Jean-Stanislas Denain, and Jacob Steinhardt. Grounding Represennatiion Similarity with Statistical Testing. 2021. URL htt.:t/s///..bb..oo//sssss/d// 0etp:m/misss/mmmmmrrr.....
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. 2017. URL http:://aax....orraaa/////// 03400
Katelyn Gao and Ozan Sener. Modeling and Optimization Trade-off in Meta-learning. Advances in Neural Information Processing Systems, 2020-December, oct 2020. ISSN 10495258. doi: 10. 1044550/arxi..210..22222 12916. URL https: URLL:ttpss///aatt...rr//aa///// /ttp://bbbb220....111122 12916v2
Micah Goldblum, Steven Reich, Liam Fowl, Renkun Ni, Valeriia Cherepanova, and Tom Goldstein. Unraveling Meta-Learning: Understanding Feature Representations for Few-Shot Tasks. 37th International Conference on Machine Learning, ICML 2020, 17117766811755::3555--3 feb 2020. 027.53333
doi: 10.485500/mxxv22002.0757.. BRLLtttps://arx.voortt
Yunhui Guo, Noel C. Codella, Leonid Karlinsky, James V. Codella, John R. Smith, Kate Saenko, Tajana Rosing, and Rogerio Feris. A Broader Study of Croms--Dominn Few-Shot Learning. dec 2019. BtLpttp: pettp://arxxy: org. /tsb//191., 1911.9911111. C7200
Jessica Hamriek and Shakir Mohamed. LEVELS OF ANALYSIS FOR MACHINE LEARNING. 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. Proceedings of the IEFE Computer Society Conference on Computer Vision and Pattern Recognition, 2016-Deeember7700778, dec 2015. ISSN 10636919, doi: 10.1109/CVPR. 2016.90. /
03385v1
Shaoli Huang and Dacheng Tao. All you need is good representation: A multi-level and classifier- centric representation for few-shot learning. 2019. URL httt:::/aaxxv..rgeee..
12476
Simon Komblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of Neural Network Representations Revisited. Technical report, may 2019. Uttp:tttp://rroeed.nng
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. ImageNet Classificatiin with Deep Convo- lutional Neural Networks. 2012. UR.
Ramnath Kumar, Tristan Deleu, and Yoshua Bengio. The Effect of Diversity in Meta-Laaming jan 2022. doi: 10.48850aaxiv.2201..1 ut/h

---

Under review as a conference paper at ICLR 2023
Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenenbaum, and Samuel J. Gershman. Building Machines That Learn and Think Like People. Behavioral and Brain Sciences, 40, 2016. URL
httt::/arxiv org/ has//ab///// / 1604 00289
Kaizhao Liang, Jacky Y Zhang, Boxin Wang. Zhuolin Yang, Oluwasanmi Koyejo, and Bo Li. Uncovering the Connections Between Adversarial Transferability and Knowledge Transferability.
2021.
David Marr. Vision: A Computational Investigation into the Human Representation and Pro- cessing of Visual Information. Phenomenology and the Cognitive Sciences, 8(4):397, 1982. ISSN 15687759. URL http::://ipppeeeeeettsee wttp:ss.ii.eeedeeceeeeeee woaett//lll//teee/////////
Etype 12227===
Brando Miranda. An empirical study of the properties of meta-learning presentation. Illinois
Digital Environment for Access to Learning and Scholarship (IDEALS), dec 2020a. URL https
/ www. idea s 1 wwwlinniiii.................il.. hwtpseeeccheahaaceeee//////////e
Brando Miranda.
An empirical study of the properties of meta-learning presentation. Illinois Digital Environment for Access Learning and Scholarship (IDEALS), 2020b. URL https
www. idea htw.:l1loooos..cchhaaddee///////
Brando Miranda, Yu-Xiong Wang. and Sanmi Koyejo. Does MAML Only Work via Feature Re-use? A Data Centric Perspective. dec 2021. URL [RRLttts:///axiiii org/ abs 2112. 13137v1
Tom M Mitchell. The Need for Biases in Learning Generalizations by The Need for Biases in Learning Generalizations. 1980.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing Atari with Deep Reinforcement Learning. 2013.
Ari S. Morcos, Maithra Raghu, and Samy Bengio. Insights representational similarity in neural networks with canonical correlation. Technical report, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. feb 2021. URL
https:://rrxt wrg. wtw.r///////////// 00020011
Aniruddh Raghu, Maithra Raghu, and Samy Bengio. Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML. Technical report, 2020. URL httts:://axiiiii
heg/abs// 1909. 09157
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretabilitty Technical report, 2017.
Sachin Ravi and Hugo Larochelle. Optimization as model for few-shot learning. Technical report, 2017.
Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. An Online Learning Approach to Interpo- lation and Extrapolation Domain Generalization. feb 2021. doi: 10.485500arxx.2100..1322 13128.
URL httl:/ttt///xiii/// httpp:::///xvvv../ org/ abs/2102. 1tt.22122..11..... 13128v2
Shai Ben-David Shai Shalev-Shwartz. Understanding Machine Learning: From Theory to Algorithms.
Cambridge University Press, 2014.
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature 2016 529:7587, 529(7587):484 489, jan 2016. ISSN 1476-4687. doi: 10.10388nature 16961. URL https: www. nat ure. com/ atticces/nattuee
12

---

Under review as a conference paper at ICLR 2023
Yonglong Tian, Yue Wang. Dilip Krishnan, Joshua B. Tenenbaum, and Phillip Isola. Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?, 2020.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta- Dataset: A Dataset of Datasets for Learning to Learn from Few Examples. 2019. URL http:
Ruohan Wang, Massimiliano Pontil, and Carlo Ciliberto. The Role of Global Labels in Few- Shot Classification and How to Infer Them. aug 2021. doi: 10448550/axix22108.00535. URL
https://arxiv. http://///axii..o//aaaba.ee/e wttggaa/////a///// 211.201088.005.
David H Wolpert and William G Macready. No Free Lunch Theorems for Optimization. JEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION, 1(1):67, 1997.
Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, Yang Gao. Tsinghua University, U C Berkeley, Shanghai Qi, and Zhi Institute, Mastering Atari Games with Limited Data. oct 2021.
URL wwwph:ttss/attttteee org/ wwwp//sssi.. 00210v1
13

---

Under review as a conference paper at ICLR 2023
A FURTHER DISCUSSIONS
We'd d like to emphasize that our synthetic experiments are promising because we can systematically differentiate meta-learning algorithms from transfer learning algorithms which supports our action- able suggestion to: 1. use the diversity coefficient to effectively study meta-learning and transfer learning algorithms, and 2. to use the diversity coefficient to design better benchmarks. In addition, this problematizes the observations that fo-proto-MAMI. in meta-data set (Triantafillou et al., (019) better than transfer learning solutions since our synthetic experiments show MAML not better than USL in the high diversity regime. To further problematize, we want point out that meta- learning methods are not better than transfer learning as observed by (Guo et al. 2019) - as observed in our synthetic experiments. We hypothesize however that the scenarios in (Triantafillou et al.) 2019) are different (Guo et al.) 2019). The first one focuses on the same meta-training and meta- testing conditions, while the latter focuses cross-domain. We hypothesize that the cross-domain scenario might benefit from a meta-learning which lower variance (e.g., a fixed embedding (Tian) al. 2020)) which might explain why sophisticated meta-learning solutions might perform worse on the cross-domain setting as observed in (Guo et al. (201)). Further research is needed in both benchmarks especially from problem centrie perspective with quantitative methods like the ones we suggest.
In addition, we hypothesize that diversity might be good proxy to predict the difference between meta-learning and transfer learning methods. More precisely, we conjecture that in a low diversity setting meta-learning methods are equivalent at meta-fest time to transfer learning methods but their difference increases as the diversity of tasks in a benchmark increases. In the high diversity regime we conjecture that the difference between meta-learning and transfer learning methods increases as the diversity increases. We are optimistic that meta-learning algorithms might outperform transfer learning methods, once we start comparing them in more thoughtfully designed benchmarks. currently designed are too
is possible that despite our efforts, meta-learning algorithms sophisticated and in fact lead to meta-overfitting, as shown in previous work (Miranda et al. 2021).
We'd like emphasize, that up until now, the meta-leaming community has evaluated meta-learning algorithms in benchmarks that might not be the most appropriate. We conjecture high diversity benchmarks are more appropriate, since they might capture the meta-learning inductive prior: high diversity means that adaptation is required by construction. Thus, we conjecture that previous conclusions should be taken with a grain of salt until more in depth study can be made in the high diversity regime- especially with benchmarks with real world data that have been analyzed extensively with metrics like the diversity coefficient that we propose. We conjecture that we can finally do meta-learning research effectively given that regime where meta-learning and transfer learning methods can be differentiated has been discovered and previous low diversity benchmarks have been understood.
We also conjecture that meta-learning research is different from classical machine learning research. Historically, a seminal paper is the one where AlexNet was proposed (Krizhevsky al., (0012). In that time we had low performance on a fixed task c.g., Imagenet and couldn't 't even interpolate the data (i.e., reach zero train error). We conjecture that meta-learning is different because if we have diversity so large where all possible tasks are incorporated, then we should reach the no-free lunch theorem regime (Wolpert and Macready 1997] where all algorithms should perform the same on average. Therefore, we hypothesize that a deliberate and quantitative efforts to design benchmarks essential. A great example of such attempt is the Abstraction and Reasoning Corpus (ARC) benchmark [Chollet] 2019) which was made very thoughtfully with Artificial General Intelligence (AGI) in mind. We conjecture meta- meta-learning is the most promising path in that direction, and hope this work inspires the design of benchmarks that lead to actionable and deliberate attempts to make progress to build such AGI technologies.
B RELATED WORK (CONTINUED)
The meta-learning literature is growing quickly and hope to provide a wider coverage here.
In terms of benchmarks, we'd like to start with the ARC benchmark (Chollet 2019). ARC was designed with AGI in mind arguably the ultimate meta-learner. Its focus is primarily on visual reasoning using program synthesis techniques. We hypothesize that it's very promising path but our
14

---

Under review as a conference paper at ICLR 2023
work inspires extension that go beyond program synthesis approaches. The meta-data set benchmark is an attempt to make the data set for few-shot learning at a larger scale and more diverse (Triantafillou et al. 2019). The main difference of their work and ours is that we propose quantitative metric to measure the intrinsic diversity in the data and go beyond data set size or number of classes. They Poo-Proto-MAML. is capable of beating transfer
also showed that meta-learning algorithms learning. However, they also showed transfer learning baselines are in fact quite difficult to beat. The IBM Cross-Domainn few-shot learning benchmark (Guo et al. 2019) is a l'accinating benchmark to evaluate meta-learning algorithms. Their central premise however is to transfer from source domain to a different target domain- instead of our setting where tasks are created from the same meta- distribution. This is why their paper is considered, in addition to few-shot learning, cross-domain benchmark. We believe this is an essential scenario to think about, but consider it different from our setting or the setting of meta-data set. We'd also like to emphasize that they do not employ metric like our diversity coefficient that quantitatively assesses the diversity of their benchmarks. These two last benchmarks, although fascinating, are missing the essential quantitative analysis of the data itself we are trying to propose.
The work by (Chen et al., 2021> give to the best of our knowledge - the first non- vacuous generaliza- tion bounds for the (supervised) meta-learning setting. Their statements apply to a non-convex loss function and use stability theory at the task level. The bound depends on the mutual information on the input data vs the output data of the meta-learner. The results, although fascinating, are not built to separate classes of meta-learning like our work attempts to do empirically.
The work by (Wang et al. 2021) proposes the idea of global labels as a way to indirectly optimize for the meta-learning objective for a fixed feature extractor. Global labels is equivalent the concept we call USL in this paper. They show that pre-training (i.e. using USL/global labels) provides excellent meta-test results- including with their method (named MeLa) that can infer global labels given only local labels provided at in episodic meta- veeta-triining. Their theoretical analysis depends on a fixed feature extractor, instead of considering the whole end-to-end meta-learner as a whole - meaning two different deep learning models cannot be used in their analysis. Therefore, their analysis fails to separate how different feature extractors might be trained, c.g. comparing USL vs MAML directly in end-to-end fashion. In contrast, we instead tackle this question head on theoretically M (with limited results) but instead show that the feature extractors indeed are different empirically
The work by (Denvv et al. 2020) proposes theoretical treatment of meta-learning using meta- learners with closed-form equations derived from ridged regularization using fixed features. They formulate the conditional and unconditional formulation using side information for the task (e.g. the support set) and show the conditional method is superior. In relation to our work, they do not provide characterizations of the role of neural network doing end-to-end meta-learning (in their empirical or theoretical analysis). In contrast, our findings make an explicit effort in understanding the role of the neural network in meta-learning in an end-to-end fashion through emperical analysis. Another contrast is that their results are highly theoretical, while ours focus on empirical results. In addition, their results are on synthetic experiments and do not explore their findings in the context of modern few-shot learning benchmarks like Minilmagenet or Cifar-fs.
The work by (Goldblum et al., 2020) provide strong evidence that adaptation at test time is best done when the meta-trained model matches the adaptation it was meta-trained with. This is shown because their classically pre-trained nets cannot perform better than the MetaOpt models with any fine-tuning method. However, their results cannot beat (Tian et al. 2020) and thus does not help separate the role of meta-training and union supervised learning (USL). Their Resnet12 results do provide further support to our hypothesis that large enough neural networks all perform the same, since 78.63 (Goldblum) vs 79.74 (RFS) have very close errors, in line with our findings. However, we hypothesize it is not due to the model size in accordance with our experiments 2
The work by (Gao and Sener 2020) provides theoretical bounds of when the expected risk of MAML and DRS (Domain Randomized Search) by bounding the gradient norm. DRS attempts to model USL but fails to do so completely, because USL is capable of modeling adaptation because the final layer is capable of adaption. Thus, it does not address the capabilities of the feature extractor being able to learn all the information needed to meta-learn Concisely, their analysis is not capable of separating performance of MAML and USL. Even if it hypothetically could, their analysis remains an upper bounds (with assumptions). This raising the question if their method truly explain the observations that transfer learning methods like USI. beat meta-learning methods. In addition,
15

---

Under review as a conference paper at ICLR 2023
they do not provide in depth empirical analysis with respect any real few-shot learning benchmarks like Minilmagenet or Cirfar-fs.
The work by (Kumar et al. 2022) provides exploration of the effects of diversity in meta-learning. The main difference with our work is that they focus mostly on sampling strategies, and it's effect on diversity, while we focused on the intrinsic diversity in the benchmarks themselves.
The work by (Rosenfeld et al. 2021) provides a theoretical analysis on the difference between interpolation and extrapolation transfer learning (and domain generalization). We believe this type of theory may be helpful as an inspiration to explore why in the high diversity regime there seems to be difference between the performance of meta-learning and transfer learning methods.
related line of work (Miranda 2020b a) first showed that there exist synthetic data sets that are capable of exhibiting higher degrees of adaptation as compared to the original work by (Raghu et al.) 2020). The difference is that they did not compare MAML models against transfer learning methods like we did here. Instead, they focused on comparing adapted MAML models vs. unadapted MAML models.
C CONVERGENCE OF LEARNING CURVES FOR FAIR COMPARISON
C.I CONVERGENCE OF LEARNING CURVES FOR MINIIMAGENET AND CIFAR-FS
In this section, we have the plots showing the learning curves achieving convergence for the models Note, the learning curves for the models trained with MAML look noisier
used in figure [ and 2 because the distributed training reduces the size (meta) batch size for logging purposes. In addition, due to episodic meta-traing veta-training, (meta) batch sizes have be smaller compared batch sizes used in USL.
Figure 4: Plot showing convergence of 5CNN Minilmagenet.
Figure Plot showing convergence of 5CNN on Cifar-fs.
Figure 6: Plot showing convergence of Resnet12 on Minilmagenet.
16

---

Under review as a conference paper at ICLR 2023
Figure 7: Plot showing convergence of Resnet12 12 on Cifar-fs.
Figure 8: Plot showing convergence of a custom 3-layer fully connected network, for all synthetic Gaussian benchmarks tested. Each curve represents a different synthetic Gaussian benchmark tested on either MAML (left plot) or USL (right plot). The curves then color-coded by the value of the Task2 bace task diversity coefficient of the Gaussian benchmark tested in that particular run.
D EXPERIMENTAL DETAILSS
D.1 EXPERIMENTAL DETAILS ON MINTIMAGENET AND CIFAR-FS
We will explain the details for the four models we trained on figure 1 and 2
Summary: We trained five layer CNN (5CNN) and Resnet on both Minilmagenet and Cifar- fs to convergence. We used the Adam optimizer with learning rate le-3. We used the standard
Minilmagenet and Cifar-fs data augmentations as provided in ("Arnold et al. 2020) matching (Tian
al. 2020).
Experimental Details for 5CNN on Minilmagenet: We used the five layer CNN from (Finn et al. 2017 Ravi and Larochelle] 2017). We used 32 filters as used in previous work. We used the Adam optimizer with learning rate le-3 for both MAML and USL. We used no scheduler. We trained the USL model for 1000 epochs. We trained the MAML model for 100,000 episodic iterations (outer loop iterations). We used a batch size of 128 for USL and a (meta) batch size of 8 for MAML. For MAML we used an inner learning rate of le-1 and 5 inner learning steps. We did not use first order MAML. It took 3 hours 5 minutes 5 seconds to train USL to convergence with a single GPU. It took 1 day 6 hours 21 minutes 8 seconds train MAML. convergence with 4 NVIDIA GeForce GTX TITAN X GPUs.
Experimental Details for Resnet12 for Minilmagenet: We used the Resnet provided by (Tian ct al. 2020). We used the Adam optimizer with learning rate le-3 for both MAML and USL. We used the same cosine scheduler as in (Tian et al.) 2020p for USL and cosine scheduler for MAML. We trained the USL model for 186 epochs. We trained the MAML model for 37,800 episodic iterations (outer loop iterations). We used batch size of 512 for USL and a (meta) batch size of 4 for MAML. For MAML we used an inner learning rate of 1e-1 and 4 inner learning steps. We did not use first order MAMI.. took day 17 hours minutes seconds to train USL to convergence with a single dgx A100-SXM4-4000
GPU. The MAML model was trained with Torchmeta (Deleu et al. 2019) which didn't support multi gpu training when we ran this experiment, so we estimate took 1-2 weeks to train single GPU. In addition, it was with earlier version of our code, unfortunately did not record the type of GPU but suspect was either an A100, A40 or Quadro RTX 6000.
Experimental Details for 5CNN for Cifar-fs: We used the five layer CNN from Finn et al. 2017 Ravi and Larochelle 2017) provided by ("Aroold et al. 2020). But used 1024 filters instead of 32
17

---

Under review as a conference paper at ICLR 2023
(to speed up convergence). We used the Adam optimizer with learning rate 1e-3 for both MAML and USL. We used the same cosine scheduler as in (Tian et al. (020) for MAML and no cosine scheduler for USL. We trained the USL model for 1000 epochs. We trained the MAML model for 100,000 episodic iterations (outer loop iterations). We used a batch size of 256 for USI. and a (meta) batch size of 8 for MAMI.. For MAML we used an inner learning rate of 1e-1 and 5 inner learning steps. We did not use first order MAML. It took 10 hours 43 minutes 31 seconds to train USL to convergence with a single GPU dgx 1100-SXM-40GGB It took 2 days hours 26 minutes 27 seconds to train MAML to convergence with 4 Quadro RTX 6000 GPUs.
Experimental Details for Resnet12 for Cifar-fs: We used the Resnet provided by (Tian et al. 2020). We used the Adam optimizer with learning rate 1e-3 for both MAML and USL. We used the cosine scheduler used in (Tian et al. 2020) for both USL and MAML.. We trained the USI. model for 200 epochs. We trained the MAML model for 75,500 episodic iterations (outer loop iterations). We used a batch size of 1024 for USL. and a (meta) batch size of for MAMI.. For MAML. we used an inner learning rate of le-1 and 5 inner learning steps. We did not use first order MAML. It took 45 minutes 54 seconds to train USL to convergence with a single GPU. It took day 19 hours 29 minutes 31 seconds to train MAML convergence with 4 dgx A100-SXM4440GB GPUs
Why the Adam optimizer? We hypothesize that the Adam optimizer is the most appropriate optimizer for fair comparison for various reasons. First, the Adam optimizer is widely used making our results most relevant and broadly applicable. Adam is generally a stable optimizer especially for sophisticated meta-learning algorithms like MAML. It is not uncommon to have SGD result in exploding gradients or end up diverging especially for MAML. Most importantly however, we hope to stay faithful whenever possible to how modern transformer models are -eeined--ecautee they have been shown to be good meta-learners, egg., gpt-3 is often cited as zero-shot learner (Brown) al., (020). These type of models do use more complicated learning schemes besides only Adam (e.g., warm-ups, decay rates etc.) but we hypothesize using Adam is a good first step. We conjecture that the small benefits that SGD might provide are negligible compared to the stability that Adam provides, especially as the scale of the data sets starts to increase. Without Adam we conjecture it would be hard to even perfectly fit the data for large scale data sets as it's usually done in Deep Learning. This was definitively true in our own experiments. Therefore, we decided to use Adam for our experiments, since it would be too hard to use SGD reliably at scale or with sophisticated meta-learning algorithms.
D.2 EXPERIMENTAL DETAILS ON N-WAY GAUSSIAN TASKS
We used a custom 3- 3-layer fully connected network derived from Learn2Learn's OmniglotFC model ("Amold et al. 2020), with parameters input_si sizee 1, output_sis = 5, and hidden layer sizes sizes = 128, 128.. We used the Adam optimizer with learning rate le-3 for both MAML and USL, but did not use a cosine scheduler for either USL or MAML. We trained the USL model for 100 epochs. We trained the MAML model for 14,000 episodic iterations (outer loop iterations). We used a batch size of 100 for USL and (meta) batch size of 100 for MAML. For MAML we used an inner learning rate of 1e-1 and 5 inner learning steps. We did not use first order MAML. It took 19 minutes 24 seconds to train USL to convergence with a single Titan X. It took days and 13 hours 6 minutes 27 seconds to train MAML to convergence with single Titan X.
D.2.1 EXPERIMENTAL DETAILS OF EMPIRICAL EQUIVALENCE FIGURE IN MAIN BODY
Experimental details for figure 3 MAML models were trained with 5 inner steps. MAML5 and MAML10 10 indicate the adaptation procedure at test time. Results used (meta) batch-size of 500 tasks and 95% confidence intervals. As the diversity of the benchmark increases, the Gaussian tasks are sampled further away from the origin. Note, as the diversity increases, the difference between USL and MAMI. increases (in favor of USL).
D.3 EXPERIMENTAL DETAILS ON 5CNN TEST PERFORMANCE EQUIVALENCE ON MINIIMAGENET
For figure 2 we used Logistic Regression (LR) with LBFGS with the default value for the 12 regularization parameter given by Python's 's Skleam. Note that an increase in inner steps from 5 to 10 with the MAML5 trained model does not provide an additional meta-test accuracy boost, consistent
18

---

Under review as a conference paper at ICLR 2023
with previous work (Miranda 2020a Note that the fact that the MAML5 representation matches the USL representation when both use the same adaptation method is not surprising given that: 1) previous work has shown that the distance between the body of an adapted MAML model is minimal compared to the unadapted MAML (which we reproduce in in the green line) and 2) the fact that a MAML5 adaptation is only 5 steps of MAMI. while L.R fully converges the prediction layer. We want to highlight that only the MAML5 model achieved the maximum meta-test performance of 0.6 with the MAML5 adaptation suggesting that the USL and MAML5 meta-learning algorithms might learn different representations. For USL to have fair comparison during meta-test time when using the MAML adaptation, we provide the MAML final layer learned initialization parameters to the USL model (but any is fine due to convexity when using a fixed feature extractor). This is needed since during meta-training USL is trained with a union of all the labels (64) - so does not even have the right output size of 5 for few-shot prediction. Meta-testing was done in the standard 5-way, 5-shot regime.
E FEATURE EXTRACTOR ANALYSIS OF USL AND MAML
Figure 9 significant difference between the feature extractor layers of a MAMI, trained model vs. a union supervised learned model.
Regrosnntation distance with metric SUCCA (Ieatune layer only)
Recresentaation distencce with metric PWCCA (trature layer only)
Adotel
Muptis MANE VS Adipted MAMe
MANL Adipttdd MANL
0.
00
Dittrne
02-
02041
Clro
Cowt
1.0
Adgted MAM
1.0
6.
08
Figure 9: Shows the significant difference between the feature extractor layers of a MAML trained model vs. a union supervised learned model - especially in contrast to the small change in the adapted MAML model (green line). This figure suggests that although benchmark diversity is small, a meta-learned representation still learns through a different mechanism than a supervised learned representation. Note that the green line is our reproduction of previous work (Raghu et al.) 2020) that showed that MAML trained model does not change after using the MAML adaptation. They term this observation as "feature re-use".
F CLASSIFICATION LAYER ANALYSIS OF USL AND MAML SHOW PREDICTIONS ARE SIMILAR
The main contribution of this section is to show the similarity of the predictions of USL and MAML through figure I0 This is additional evidence to our claims ttrough different metric than accuracy difference that USL and MAML are empirically equivalent. For the background on SVCCA, PWCCA, CKA, and, OPD section L
19

---

Under review as a conference paper at ICLR 2023
1.0
Aciggted Hose adgpted MAME
10
ediuted Hoad Adiiped MANA.
0.0
00-
layer
Ligar
Regresencation disencce with motni LICCA
Appresoriation distance with motric OPO
02
02
0.e1 O1e2
Deed
Heat
0w01 Dser2
Dees Heal
The classification layer of transfer learning and MAML5 model decrease in distance - implying similar predictions. More precisely, an initialization trained with 5 inner steps (MAML55 has an increasingly similar head (classifier) after adaptation with MAML.5 compared to the classifier layer of the Union Supervise-Lcarmn the final layer. particular, the USL model has been adapted with Logistic Regression (LR) with LBFGS with the default value for the 12 regularization parameter given by Python's 's Sklearn (as in (Tian et al. 2020).. We showed this trend with four different distance metrics SVCCA, PWCCA, LICKA, and OPD as referenced in section 2 Observe that according to PWCCA, the distance between the predictions is zero. This is true because the distance of classification layer (indicated "head" in the figure) is zero. The architecture used here is five layer CNN in (Finn al. 2017; Ravi and Larochelle 2017) with their same setup. The benchmark used for this analysis is Minilmagenet.
Figure 10:
(USL) model that has been adapted only at
G BACKGROUND OF FEW-SHOT LEARNING BASICS
The goal of few-shot learning to learn to classify from a limited set of training samples. few-shot benchmark is utilized evaluate few-shot learning algorithms and typically contains many classes and smaller number of samples per class. Typically, few-shot learning algorithms learn in episodes, where in each episode, task consisting of train (or support) set and a held-out validation (or query) set is sampled. In particular, a task is n-way k-shot classification problem, means that the support and query sets each consist of classes sampled from the benchmark, and each of the n classes are represented by A shots examples. The learner uses the support set to adapt to the task, and the query set to evaluate the performance on the given task.
H SYNTHETIC GAUSSIAN BENCHMARK AND N-WAY GAUSSIAN TASKS
We create a series of synthetic few-shot benchmarks, where each Gaussian benchmark B is defined by four parameters B- (pm, Fm, fl,,, σs). To form the dataset of our benchmark, we first sample 100 meta-train, 100 meta-test, and 100 meta-validation classes, where class 1 < < 300 is a Gaussian parameterized by (Helasss Oclass;) where
Pelass, ~ N(p,, am)) Telass, ~ (N Rx)
Then, for each class i, we sample 1000 data points (I 1) ... (I;,1000: i) where each datapoint (æ, y) is composed of a input value E IR and class label 1 E y < 300. The input values Ti, .1,1....11,000
20

---

Under review as a conference paper at ICLR 2023
each sampled from class i's class distribution:
Fi,1 ... .., 1000 1000 N (Melass. Telass.)
Having defined our dataset underlying our benchmark, we may now sample individual tasks from our benchmark. Each task in our benchmark is 5-way, 10-shot that is, each task formed by first sampling 5 ways from the benchmark dataset, then sampling 10 shots from each of the 5 ways. The goal of each task is to correctly predict which of the ways an input value æ E falls into. We conducted experiments using 7 different benchmarks, with each benchmark defined by four parameters and its corresponding Hellinger distribution diversity coefficient and Vesk2Vec task diversity coefficient, as listed in Table
Benchmark Parameters
(pm, m. Mss (s)
Hellinger-based Distribution Diversity
Task2Vec-based Task Diversity
(0, 0.01, 1, 0.01) (0, I, I, 0.01)
7.475e-05 + 4.891e-07
0.247 + 1.04e-3
0,183 1.24e-3
0.271 = 1.15e-3
0,574 + 2.28e-3
0.393 :: 1,79e-3
(0, 10, 0.01)
0.860 1.75e-3
0.470 2.35e-3
(0, 20, 1, 0.01)
0.929 + 1.31e-3
0.533 + 2.47e-3
(0, 30, 1, 0.01)
0.952 1.10e-3
0.537 = 2.57e-3
(0, 1000, 1, 0.01)
0.998 + 2.07e-4
0.546 + 2.74e-3
Benchmarks of increasing diversity are created by increasing on, or the standard deviation of the class mean. This table also shows that as the ground truth diversity increase so does the Vec diversity iimplying the Task2Vec diversity is good proxy to the groun truth distribution based diversity. A larger om increases the variance of the class means, making their respective class distributions farther apart average and causing both the Hellinger- based distribution diversity and Bask2Vec-baseer task diversity coefficients to increase. We varied om from 0.01 to 1000 and fixed all remaining benchmark parameters to obtain 7 different Gaussian benchmarks. The corresponding Hellinger-hasee hased distribution diversity coefficients were obtained by numerically approximating the expected Hellinger distance between two classes sampled from the benchmark and computing the 95% confidence interval of the approximation. We also computed Task2Vec-based task diversity coefficients as an alternative measure to diversity using random 3-layer fully connected probe network described in Section [ Figure [2] visualizes the Task2 Vec task diversities among the synthetic benchmarks via heatmap showing the relative pairwise distance between sampled tasks.
Table 3:
DISTRIBUTION- BAASED DIVERSITY METRICS
In addition the task diversity methods (such as Tesk2 that we chose as measure of diversity across our experiments in our main paper, we would like to introduce an additional class of diversity metries that call distribution diversity. Unlike task diversity, which quantifies diversity through the expected distance between any two distinct tasks sampled from the benchmark, distribution diversity quantifies diversity through the expected distance between any two distinct distributions that underlie the benchmark. In our synthetic Gaussian experiments, we define the distribution diversity of our Gaussian benchmark as the expected Hellinger distance between two distinet Gaussian class distributions sampled from the benchmark we describe the calculation of the distribution diversity of our synthetic Gaussian benchmark more detail in Section 0
J HELLINGER DIVERSITY CORFFICIENT AND HELLINGER DISTANCE
An alternative metric the Task2Vec-basee task diversity metric is the Hellinger- bellinger-based based distribution diversity metric. The Hellinger-based distribution diversity of our Gaussian benchmark is obtained by computing the expected Hellinger distance between any two classes sampled from the benchmark. That is, for some benchmark parameterized by B (Jmm) m Ma az), the diversity coefficient is given by
div( B) E Har (Hm Bo oo (F N(2,, 02))]
21

---

Under review as a conference paper at ICLR 2023
Diverity tffect rerformane offorence of USL MAN.
Dierdyy in Ferformance Offferecce US. MAML
He Inger vs Task2Veec Diveratyy Synthenic Caussian
Mantt
0.50
N40
0.20
0.30
Helliger CwwriiyyCufffeeee
3.
0.
Hellgger Oiversty Coeficeet
0.4
6.6
1.
Shows the strong relation between Hellinger distribution diversity and the Task2Ver task diversity coefficients, as both coefficients may be used interchangeably as a measure the diversity of given synthetic Gaussian benchmark The first two plots show the relation between the Hellinger-basee based distribution diversity of a synthetic Gaussian benchmark and the benchmark's performance the MAML5, MAML10, and USL methods. These first two plots noticeably similar to Figure 3 (where Task2 Besk2Veebbased task diversity was used as measure diversity instead of Hellinger-based based distribution diversity), which indicates that Hellinger-basse based distribution diversity metric also serves as good proxy for task diversity. The rightmost plot shows a strong positive correlation between Hellinger- basee distribution diversity and Task2Vec task diversity (Pearson r 0.990).
Figure 11:
100
08
10 11
33 16
30
131
10
0.0
Heatmaps show how benchmarks with larger Task2Vec-based task diversity coefficient show more heterogeneity between sampled tasks Each heatmap below shows the pairwise distance between fifteen 5-way, 10-shot few-shot learning tasks sampled from the various synthetic Gaussian benchmarks described in Table 3 Note that as d, (the standard deviation of the class mean) increases, the distance between
Figure 12:
tasks becomes larger average and more varied, which can be seen as the heatmaps become more heterogeneous. This increase in expected distance among different tasks in turn increases the Task2Vec-based task diversity coefficient, which summarizes the average distance between any two tasks. From left to right, top to bottom, the benchmarks tested have parameters om 0.01, 1, 20, 30, 1000 and Task2 Bakk2Vec-basee based task diversity coefficient parameters div 0.247, 0.271, 0.393, 0.533, 0.537, 0.546.
22

---

Under review as a conference paper at ICLR 2023
where H² denotes the squared Hellinger distance metric and N( T1), ((2, 02) denote the distributions of the two classes sampled from the benchmark. The Hellinger-baa based distribution diversity metrie provides an intuitive, madel-agnostic characterization of the diversity of a benchmark the larger the diversity, the less similar any two classes within the benchmark are, and the easier it is to distinguish between classes within the benchmark are, and the harder it is to distinguish between two classes due to larger overlap between the two classes' distributions.
classes. Conversely, the lower the diversity, the more similar any two
Note that the closed-form equation for the Hellinger distance between the two class distributions N (1) (P2,, σ2) is given by
20102 V of oz
-1
(e1-19)2 ofto
(N) =1 -1-
However, there is no simple closed-form equation for computing the diversity div(B) itself. As result, we computed the diversity coefficient as a numerical approximation by repeatedly sampling two classes from the benchmark distribution and calculating the Hellinger distance between the two classes. These samples ultimately provide 95% confidence interval that represents the expected Hellinger distance between two classes sampled from the benchmark.
We also compared our Hellinger-based distribution diversity coefficient with the Task2Vec-basss task diversity coefficient for each of the synthetic Gaussian benchmarks tested in Table 3 We observe strong positive correlation between the Hellinger-
distribution diversity and Task2 Vec- based task diversity coefficients according to Figure 11 indicating that the Hellinger- -aaee distribution diversity serves as a effective proxy for task diversity when the number of ways and shots of all tasks fixed.
K ANALYSIS OF DISTRIBUTION OF TASK DISTANCES IN FEW-SHOT LEARNING BENCHMARKS
K.1 HEAT MAPS SHOW Low DIVERSITY AND HOMOGENBITY OF TASKS FROM MINIIMAGENET AND CIFAR-FS
In this section, show the heat maps showing the distances between 5-way, 20-shot few-shot learning tasks from Minilmagenet and Cifar-fs in figure [3 and [4] We used 20-shots because we do not need separate the data into support and query set to compute the diversity coefficient. We show that tasks sampled from these benchmarks create not only a low diversity coefficient on average, but also at the level of individual distances between pairs of tasks. In addition, the heat map's uniform coloring reveals that it is also justifiable to call the tasks from these benchmarks homogeneous. Low diversity is shown because the distances are between 0.07-0.1 given that max is 1.0 and minimum is 0.0.
K.2 HISTOGRAMS OF DISTANCES OF TASKS THE SYNTHETIC GAUSSIAN BENCHMARK, MINIÍMAGENET AND CIFAR-FS
In this section, we show the histograms of the cosine distances between pairs of tasks for the Gaussian Benchmark, Minilmagenet and Cifar-fs. The main purpose of this is to argue that (relatively small) sample of the tasks sufficient to estimate population statistics like the expected distance between tasks i.c. diversity coefficient.
For case exposition of the argument, consider the case where we have 500 distance from a large population of size (04) = 7, 624, 512. The goal to argue that 500 samples are enough to make strong statistical inferences about the population even if it's large as 624, 512. If we assume the distribution of the data is Gaussian, then we expect to see a single mode with an approximate bell curve. Therefore, if we plot the histogram of task pair distances of the 500 tasks and this then we can infer our Gaussian assumption is approximately correct. Given that we do see that in infer our assumption is approximately correct. This implies we can
figures 15 16. 17 then
23

---

Under review as a conference paper at ICLR 2023
1M
Figure 13: Shows homogeneity and low diversity of 5-way, 20-shot tasks from Minilmagenet using the Vesk2Vec distance (Achille UCLA et al. 2019). The top left heat map uses a Resnet18 pre-trained on Imagenet to compute the Task2 distance between tasks. The top right heat map uses Resnet18 with random weights to compute the Tesk2Vec distance between tasks. The bottom left heat map uses Resnet34 pre- pre-trained on Imagenet compute the Tesk2 distance between tasks. The bottom right heat map uses Resnet34 with random weights on Imagenet to compute the Tesk2Vec distance between tasks. Homogeneity is shown because of the uniform color shown in the heat map. Low diversity is shown because the distance is between 0.07-0.12 given that max is 1.0 and minimum 0.O. Note the diagonal is exactly zero because is comparing the same tasks. The axis indices indicate the arbitrary name for the tasks. Indices between heat maps do not indicate the same task. We used the cosine distance between task Task2Vec embeddings.
make strong statistical assumptions about the population in particular, that we have a good estimate of the diversity coefficient using 500 samples. Additionally, in histograms also discard the presence of outlier tasks.
L BACKGROUND ON DEEP NEURAL NETWORK DISTANCE METRICS
To compute the distance between neural networks we use the distance versions of Singular Value Canonical Correlation Analysis (SVCCA)
Distances for Deep Neural Network Feature Analysis:
(Raghu et al. 2017). Projection Weighted Canonical Correlation (PWCCA) (Morcos et al. 2018). Linear Centered Kernel Analysis (LINCKA) (Kornblith et al. 2019) and Orthogonal Procrustes Distance (OPD) (Ding et al. 2021). These distances are in the interval (0, 1] and are not necessarily a formal distance metric but are guaranteed to be zero when their inputs are equal and nonzero otherwise. This is true because SVCCA. PWCCA, LINCKA are based on similarity metrics and OPD already a distance. Note that we use the formula d(X, , Y) = sim (X, Y) for our distance metrics where sim is either SVCCA, PWCCA. LINCKA similarity metric and X, Y are matrices of activations (called layer matrices). The distance between two models is computed by choosing a layer and then comparing the features/activations after adaptation for that layer given batch of tasks represented as a support and query set.
L.1 NEURON VECTORS
The representation of neuron d in layer I is the vector 2 ex)(x) N of activations for set of N examples, where X RNDD is the data matrix with N examples.
24

---

Under review as a conference paper at ICLR 2023
6.1
10
812
5583P
Figure 14: Shows homogeneity and low diversity of 5-way, 20-shot tasks from Cifar-fs using the Task2Vec distance. The top left heat map uses a Resnet18 pre-trained on Imagenet compute the Task2 Vec distance between tasks. The top right heat map uses a Resnet18 with random weights to compute the Task2 Vec distance between tasks. The bottom left heat map uses Resnet34 pre-trained on Imagenet to compute the Tesk2 distance between tasks. The bottom right heat map uses a Resnet34 with random weights on Imagenet to compute the Task2 Vec distance between tasks. Homogeneity is shown because of the uniform color shown in the heat map. Low diversity is shown because the distance is between 0,07-0.12 given that max 1.0 and minimum is 0.0. The axis indices indicate the arbitrary name for the tasks. Indices between heat maps do not indicate the same task. We used the cosine distance between task Task2Vec embeddings.
1500
19000
1200
1508
500
2200
0320
E10
0316
0113
2120
CoTT Detancc Thaind CEARFS
Dembuttio
4000
2508
2008
20/0
20590
100 030 MDO COM 030 AEO 5477
Figure 15: Histogram of distances of 5-way, 20-shot tasks from Cifar-fs using the Task2Vec distance. This plot justifies the use of subsample of the population to estimate the diversity coefficient because of its approximate Gaussian distribution. For the full argument, see the main text, section K.2 The top left histogram uses Resnet 8 pre-trained on Imagenet to compute the Vesk2Vec distance between tasks. The top right histogram uses Resnet18 with random weights to compute the Vec distance between tasks. The bottom left histogram uses Resnet34 pre-trained on Imagenct to compute the Task2 Vec distance between tasks. The bottom right histogram uses Resnet34 with random weights Imagenet compute the Task2 Vec distance between tasks.
25

---

Under review as a conference paper at ICLR 2023
D10184501 010004144 BAB.AA DRGA Rastec22 www.eteeeeeetteeee Preftaneed ragertee Distrtacianoo lendlee Ostari Rasturta hancer
2000
Duttistion baa2Ve Ditiaees Restocal Taaked
Diriibation bakWWe Diraaces Festeri4 fancer tes Minii
4000
1800
6.13
$11
6.12
0.990
C.M00
6/00
6CT9
B075
Obrarcs
Figure 16: Histogram of distances of 5-way, 20-shot tasks from Minilmagenet using the Task2Vec distance. This plot justifies the use of subsample of the population estimate the diversity coefficient because of its approximate Gaussian distribution. For the full argument, see the main text, section K.2 The top left histogram uses a Resnet18 pre-trained on Imagenet to compute the Vesk2Vec distance between tasks. The top right histogram uses Resnet18 with random weights to compute the Task2 Vec distance between tasks. The bottom left histogram uses a Resnet34 pre-trained Imagenet compute the Tesk2Vec distance between tasks. The bottom right histogram uses Resnet34 with random weights on Imagenet to compute the Task2Vec distance between tasks.
L.2 LAYER MATRIX
layer matrix L for layer / is matrix of neuron vectors 20 (X) (X) E R with shape, [AV, D:] i.c. L R R.D. In other words, the layer matrix is the subspace of N spanned by its neuron vectors 2d (X). In short, L is the layer matrix ==b, E R.D.D with neuron vector z/-
L.3 CCA
Canonical Correlation Analysis (CCA) is well established statistical technique for comparing the (linear) correlation two sets of random variables (or vectors random variables). In the empirical case, however, one computes the correlations between two sets of data sets (e.g. matrices X E N, and Y E N, with N examples and D1, D2 features or layer matrices).
True distribution based Canonical Correlation Analysis (CCA): What call true distribution based CCA is the standard CCA measure using the true but known distribution of the data p" (x) and p" (y). In this case, CCA searches for pair of linear combinations a", b" of variables (or vectors of random variables) x [X,,..,<< XD.] and = [Y....., YD. that maximizes the Pearson correlation coefficient:
of random
(x ((b b))]
Σx,y
a' b²,b" -arg max a.b Ex (a aT Ey (aa y)
arg max w2
Ex,xa Ey.b
where Ex, Ex, = Cou X, = - [X,Y;] for centered random variables). All these can replaced by empirical data matrices in the obvious way.
the (true) covariance and variance matrices respectively (e.g.
L.4 SVCCA
At a high level, SVCCA a similarity measure two matrices that aims in removing redundant neurons (i.e. redundant features) with the truncated SVD by keeping 0.99 of the variance and then measure the overall similarity by averaging the top C' CCA values.
SV: Given two matrices L1 N.D, L2 N.D2 (e.g. layer matrices) first reduce the effective dimensionality of the matrix via low rank approximation E N. I.2 E R.N by choosing
26

---

Under review as a conference paper at ICLR 2023
Desrbbation Task2Vec sitaaces, Gauasion (0.110001
13.0
10.0
58
839
Coline Diitanoe between Tril Pairs
Ditribution 1ssk2ve
10.31.0011
200
15.
150
100
23-
25
0.28
8.45
Dotamcs Task
Dedrbbrtm of Taec2Wer Ortwnnes 10 Councion ai (0,701800))
315
58
Cod DitDicn setwern Part
Ditribution Yaak2Wer Dratanoes Gazm (8.1800...000)
15
0.
Figure 17: Histogram of distances of 5-way, 20-shot tasks from the Synthetic Gaussian bench- mark using the Task2Ve Vec distance. This plot justifies the use of subsample of the population to estimate the diversity coefficient because of its approximate Gaussian distribution. For the full argument, see the main text, section K.2 The meta parameters generating tasks for each benchmark
are denoted by B (0, x, 1, 0.01) where a is in the list [0.01, 1,3, 10, 20, 30, 1000) indicating the mean to generate the mean of the Gaussian tasks. For full details of the synthetic Gaussian benchmark, see section
the top & singular values that keeps 0.99 of the variance. In particular, for each layer matrix, L; keep the top D/ singular values (and vectors) such that Σラ layl 0.99 E/ Irank(L.) layl-
SVCCA: SVCCA is statistical technique for the measuring the (linear) similarity of two sets of data sets L1 E IR N,D; L2 E I N.Da (e.g. data matrices, layer matrices) by first reducing the effective dimensionality of the matrix via a low rank approximation I/ E NNND It E N.D (e.g. by choosing the top i singular values that keeps 0.99 of the variance) and then applying the standard empirical CCA to the resulting matrices. This is repeated C' = min(D', D2) times and the overall similarity of the matrices is computed the average CCA: sveca sim( Io)) 1 C Po
Concretely:
1. Get the D' components that keep 0.99 of the variance (i.e. D' such that ΣDi layl 2
0.99 Irank(L.) (o,1).
2. Get the SVD: U1 E1, VT SVD((T) and U2, E2, Vg SVD(L2)
27

---

Under review as a conference paper at ICLR 2023
3. Then produce the SVD dimensionality reduction by L = LıVi[1 k]] R. and
L22 = [2[11 k] E R D, where Vi[1: Djj gets the top D, columns of a layer matrix
4. Get the CCA of the reduced layer matrix: [pee== - CCA(L Lg) where,
min(D(,D{)
5. Finally return the mean CCA: secca = č Ec Pe, where is the k-th CCA value of the reduced layer matrix.
L.5 PWCCA
At high level, PWCCA was developed increase the robustness (to noise) of SVCCA in the context of deep neural networks. In particular, Maithra et al. (Morcos et al. 2018 noticed that when the performance of the neural networks stabilized, so did the set of CCA vectors (or principle neuron vectors) related to the network stabilized on the data set in question. Thus, they suggest to give higher weighting to the canonical correlation Pe of these stable CCA vectors in particular to the ones that are similar to the final output layer matrix, e.g. L1. Note this is simpler than trying to track the stability of these CCA vectors during training and then give those higher weighting.
Formally let L1 be the layer matrix I2's...: = E N. with neuron vectors z for some layer 1. Recall that the k-th left CCA vector for layer matrix L1 defined as follows, T. = Li = L1 ( 2 u..) where ag the eth CCA direction and u, is the e-th left singular value from the matrix M L1,
PWCCA:
3 EL L2 E UAVT. Then, PWCCA can be computed as follows:
1. Calculate the CCA vectors T. L L, (E7, u.) u.) and explicitly orthonormalize with Gram- Seam--chhidtt for numerical stability.
2. Compute the weight ae of how much the layer matrix J,1 is account for by each CCA vector 3y with equation de (he L1) = E (@ee </) where z/ is the c-th column of the layer matrix L1
3. Normalize the weight indicating how much each CCA vector h. accounts for Lı and denote (11
it with, aākā L1) D (o(
4. Finally return the mean CCA weighted by aā L1): pwoca = EE ae LI)pe where C' = min( D2 D().
The original authors could have used the right CCA vectors, i.c. ÿe L2by L2 vz) and in fact the details of their code suggest they choose the one that would have lead to less values removed by SVD. This choice seems to already be robust noise, as shown (Morcos al., 2018). Note that the CCA vectors St. Ük are of size N and thus could be viewed as the principle neuron vectors that correlate two layers L1, L22. With this view, PWCCA computes the mean CCA normalized by of the principle neuron vectors are account for the output layer matrix the most.
L.6 CCA FOR CNNS
The input to CCA are two data matrices, but CNNs have intermediate representations that are 4D tensors. Therefore, some justification is needed in how to create the data matrices needed for computing CCA for CNNs. Note that it's the same reasoning for both SVCCA and PWCCA.
Each channel as the dimensionality of the data matrix: One option is to get the intermediate representation of size [M, C, H, W] and get layer matrix of size [M HW, C?. Thus, M MWW is the effective number of data points and the channels (or number of filters) the effective dimensionality of the (layer) matrix. In this view, each patch of image processed by the CNN is effectively considered data point. This view is very natural because it also considers each filter as its own "neuron" which seams reasonable considering that each filter uniquely responds each stimulus (e.g., data patch). This view results in HW images for every sample the data set (or batch) of size M and C effective neurons.
Although the original authors suggest this metric as a good metric mainly for comparing two layers that the same hypothesize it is also good for comparing different layers (as long as the
28

---

Under review as a conference paper at ICLR 2023
effective number of data points match for the two layer matrices). The reason is that CCA tries to compute the maximum correlation of two data sets (or sets of random variables) and assumes no meaning in the ordering of the data points and assumes no process for generating each individual sample for the set of random variables, thus meaning that this metric (CCA) can be used for any two layers in a matrix. Overall, in this view, we are comparing the representation learned in each channel.
Each activation as the dimensionality of the data matrix: One option is to get the intermediate representation of size [M, C, H, W] and get a layer matrix of size [M, CHWJ. Thus, M is the effective number of data points (which matches the number of samples in the data set or batch) and therefore each activation value is the effective dimensionality of the (layer) matrix. In this view, each activation is viewed as neuron of size M and we have CHWW effective neurons for each activation. The authors suggest this metric for comparing different layers (potentially at different depths). However, because CCA assumes no correspondence between the data points nor the same dimensionality in the data matrix- - we hypothesize this way to define the data matrix is as valid as the previous definition for comparisons between any models at any layer. One disadvantage however is that it will often result in data matrices that are very large due to CHHW being very large - which results in artificially high CCA similarity values. Potential ways deal with it are noticing that there is no correspondence between the data matrices, so a cross comparison of every data point with every other data point in CCA is possible (resulting is (M2) comparisons for the empirical covariance matrix). A Alternatively one can pool in the spatial dimensions [H, W] resulting in potential smaller layer matrices e.g. of shape [M, CJ] with a pool over the entire spatial dimension. For these reasons and the fact that we hypothesize an image patch being its own image we prefer to interpret the number of channels the natural way to compare CNNs so that the layer matrices results of size [M HWWWO C).
Subsampling of representations for channels as dimensionality: In this section, we review the subsampling we did when comparing the representations learned in each channel, i.c. the layer matrix has size [ HW, C). The effective number of data points M HW will often be much larger than needed (e.g. for 16 data samples M = 16 and H W = 84 results in M HW = 112, 896). especially compared to the number of filters/channels (c.g. C = 64). Previous work (Raghu et al.) Morcos
Morcoso et al. 2018) suggest using the number of effective data points to be from 5-10 times the size of the dimensionality in a layer matrix of size [N', D' that means N' = 10D'. Based on our reproductions of that number, we choose N' 20D' which results in NHW 20C
L.7 CENTERED KERNEL ALIGNMENT (CKA)
At a high level, CKA is based on the insight that one can first measure the similarity between every pair of examples each representation separately and then use the similarity structure to compute an overall similarity metric. In our case, we can treat the examples as the neuron vectors and compare all neuron vectors using some kernel function. Usually this will end in a kernel matrix of size M, M' where M and M' are the number of examples. In case, they would be D, D' for the number of neurons of each layer matrix. Note, the layers matrices can correspond to neurons of different layers in neural network.
Linear CKA: We use the linear kernel function as used in previous work (Korbblith et al. 2019 Ding al. 2021). Given two layer matrices X1 RR..D: and X₂ E R N.Dr for layers T,Y', compute the linear kernel X2 X2 to get the D; by Dr kernel matrix indicating the (linear) similarity per neuron vector for the two layers. Then obtain a single distance value we compute the Frobenius norm of the kernel matrix and subtract by one after normalization:
|ATBBER B
(i =1 ||AT A BlI
Note that depending on how the examples in matrices A, B are organized the cross-produc could be computed with AB" instead. Other kernel functions have been tested (c.g., the RBF kernel) for CKA but similar results are obtained, resulting in linear CKA being the most popular CKA method (Ding et al. 2021 Kornblith et al. 2019) to the best of our knowledge.
29

---

Under review as a conference paper at ICLR 2023
L.8 ORTHOGONAL PROCRUSTES DISTANCE (OPD)
At a high level, the orthogonal Procrustes distance computes the distances between two matrices after using for the best orthogonal matrix that tries to match the two. Usually this is done after centering and dividing by the Frobenious the matrices, i.e. normalizing the matrices. In addition, previous work (Ding et al. 202I) finds that OPD a better metric detecting changes that matter functionally and robust against changes that do not matter.
OPD: Formally, the Orthogonal Procrustes Distance the smallest distance between two matrices X and Y (with columns as the vectors in question) found by finding the orthogonal matrix Q which most closely maps A to B. Therefore, the OPD distance is the distance value from solving the orthogonal Procrustes problem:
Q dopn(X,Y) min || - YQUI
where || || is the Frobenius norm. When matrices are normalized (centered and divided by their Frobenious norm) this is called the general Procrustes problem. However, the closed for equation we used is the following:
doPD ( Y) (IX11 + | - 2
(6)
where || I|I is the nuclear norm, i.e. the sum of singular values Σ, oAA;; = ||||, The division by is to guarantee that the OPD distance is between [0, 1] instead of [0, 2]. We do the standard normalization of the matrices before computing the OPD distance - by centering and dividing by the Frobenious norm of the matrix. This is done because the orthogonal matrix in the orthogonal Procrustes problem does not allow for translation or rescaling of the matrices. Therefore, this normalization enforces invariance to this type of transformations - i.c. we don't 't want large OPD values due to rescaling or translation (and even if present, the orthogonal matrix wouldn't be able to reflect it).
Therefore, the final equation for OPD use is:
doPD (opp(X,Y)=14 10
Why OPD? We use OPD due to the findings of (Ding et al. 202I). They find that OPD is a more robust metric (compared SVCCA, PWCCA, and CKA) because it is sensitive to changes that affect real functional behavior (so it detects changes to behavior that "matter") and it's specific against changes that do not. As a summary, some the evidence that they provide for this is that OPD is able to detect when 0.75 of the principal components are removed, while CKA cannot detect removal of principal components until 0.97 are removed. CCA like metrics on the other hand are not specific- even random initialization noise overwhelms the distances it reports, while OPD is more robust to this random noise. For the last point, this means that even if we compare two different layers with CCA, the noise will dominate the distance reported instead of the difference caused by comparing different layer.
L.9 CORRECTLY USING FEATURE BASED DISTANCES
When comparing two layers of neural network using two layer matrices, one needs to be careful with the number of data points (or batch size) being used. This is because metrics like CCA intrinsically formulated as optimization and if the number of examples is not larger than the number of dimensionality of the examples - then the similarity can be pathologically be perfect (e.g., the distance is zero when it's actually not zero). Therefore, we follow the suggestions by the original authors of SVCCA (Raghu et al. (017)) and always use at least 10 times more examples than there are features for our feature based comparisons. We call this value the safety margin. To illustrate this idea, we produce two random matrices and compare how the similarity (SVCCA) values varies as a function of the dimensionality of the data and the number of points. Since the two matrices are completely random, we know they should not be very similar and thus SVCCA should report a high similarity value (or low distance value). Therefore, we can see in figure 18 how as the dimensionality increases, the similarity value approaches perfect similarity of 1.0. In figure 19 we the number of points increases, we approach a smaller similarity - closer to the true similarity for random matrices.
how as
30

---

Under review as a conference paper at ICLR 2023
In general, given two matrices X, Y E R',''' with the number of (effective) data points M' and (effective) dimensionality (number of features) D' - we want the number points to larger by safety factor s. Formally, it must satisfy this inequality to avoid the pathological case for feature based distances:
D' &M'
88)
where we suggest use S 10 (as used in previous work (Raghu et al. 2017)) Note the effective number data points used and dimensionality can be different depending on how one reshapes the CNN tensors to produce layer matrices explained in section [.6] For example, if one uses the channel as the dimensionality (i.e., use the image patches as an effective data point) then one has to obey the following inequality:
C S sMHW
where M is the batch size, H, W is the height and width of the images, and C' the number of filters/channel for the current layer. This means that for given architecture processing images of given size that the only parameter we can change to make the above inequality true is the batch size M.
Features (D) Sim (SVCCA)
sirlantly (w0c4)
B- 300
0200
0000
10000
number of prebhiie of mmmeniin (3)
Figure 18: Shows that as the dimensionality of a random data matrix inereases - the SVCCA similarity approaches the pathological case by falsely reports the similarity is perfect. The green line indicates when the number examples and dimensionality are equal (and equal to 300). D denotes the dimensionality of the simulated data and B the size of the batch size/number of points.
(SVCCA)
siiierlty (002a)
0.6
2000
4000
8000
3000
10000
mumber of dasa points (n)
Figure 19: Show how to avoid the pathological case when using feature based similarities by increasing the number of data points (or batch size). In particular, the number of data points in two random data matrix increases the true similarity approaches the true low similarity value. The green line indicates when the number of examples and dimensionality are equal (and equal to 300). D denotes the dimensionality of the simulated data and B the size of the batch size/number of points.
31

---

Under review as a conference paper at ICLR 2023
M A STATISTICAL DECISION VIEW OF THE DIFFERENCES BETWEEN SUPERVISED LEARNING AND META-LEARNING
Recent work in meta-learning implies that feature-reuse might be all we need to solve modern few- shot learning benchmarks (Tian et al. 2020). However, what it also reveals is our poor understanding of meta-llearingg algorithms. Therefore, in this section, take the most foundational perspective to formulate and analyze meta-lcarning algorithms by analyzing them from an optimal statistical decision theory perspective?
We hope that this can help clarify the results from (Tian et al. 2020) and therefore help meta-learning researchers design better meta-leariigg benchmarks and meta-learning algorithms.
M.1 SUPERVISED META-LEARNE LEARNING PROBLEM SET-UP
In this section, we introduce the notation for supervised meta-learning, Intuitively, we seek to find function that minimize the expected risk over tasks and the data the tasks. To formalize it, we will use three formulations:
Monolithic meta-learner: for a monolithic decision rule g (or meta-learner), we want to find the optimal g by minimizing the supervised meta-learning expected risk:
Ratono (g) = "er-p(tBBa.yytteee
[[((2,)) +), v)]
(10)
where g a single monolithic function, p(r) is the true but unknown distribution of tasks, p(a, y ()) is the true, but unknown distribution of data pair given task and (ær, y) is the data pair of input and target value sampled from task.
Meta-learned meta-learner for meta-learned decision rule we usually have adaptation rule A (e.g. SGD in MAML) and a function approximator (e.g. a neural network) and minimize the follow over both:
R h) =
[[A(h, (=)) y)1
(11)
p(T) is the true but unknown distribution tasks, p(z, y r) is the true, but unknown distribution of data pair given a task and (x, y) is the data pair of input and target value sampled from task.
Fixed representation meta-learner without adaptation: one can also solve 10 using a single decision rule f that does not take the task input follows:
RsL(f) = ET-p(rr Ee. [(f(z),
(12)
where f a function to be adapted (e.g. a neural network), p(T) is the true but unknown distribution of tasks, p(z, y (T) is the true, but unknown distribution of data pair given task r and (x, y) is the data pair of input and target value sampled from a task.
Fixed representation meta-learner with a final adaptation layer: one can also solve 10 using single feature extractor that does not take the task 7 as input with a feature extractor g:
RSLA(f.9)
[L((f(T) g)(z), v)]
(13)
where g is the feature extractor from the raw inputs (e.g. a neural network), f the final layer adapted (e.g. linear layer), p(T) is the true but unknown distribution of tasks, p(x,, | () is the true, but unknown distribution of data pair given a task and (x. y) is the data pair of input and target value sampled from task.
Remark M.I. Note that in practice, the meta-learner does not usually take the full task 7 as input, but instead a train and test set (often referred to as support set and query set) sampled from the task r.
The goal of this work is to clarify the difference between T2 and 11 under the framework of statistical decision theory. Arguably the most important comparison between and 13 left for future work.
M.2 MAIN RESULT: DIFFERENCE BETWEEN THE SUPERVISED LEARNED AND META-LEEARED DECISION RULE
The proof sketch is as follows: we first show the optimal decision rules for both supervised learning and meta-learning when minimizing the expected meta-risk from equations [2 and 11 and then
32

---

Under review as a conference paper at ICLR 2023
highlight that the main difference between them is that the meta-leamed solution act optimally if it identifies the task T while the supervised learned solution has no capabilities of this since it learns an average based on tasks instead.
Theorem M.2. The minimizer to equation 77 is:
A(h, ()) yaur = Byyplularr)) [y]
(14)
where Fylu = By-ppulaar)) [y] and is the squared loss UÚ. v) = (ū - y)
Proof. The proof the same as the standard decision rule textbook proof but instead of minimizing it wise w.r.t we minimize it point-wise w.r.t. (Iz, (I, In particular, we have:
RML(A, h) Er-p(r) lE y-p(aal)))) [((A(,, (x),, v)l
min E, Tx-p(r) B (urr(ul..)) (A(h, ()) - y)
A.
without loss of generality (WLOG) and for clarity of exposition consider the special case for discrete variables:
min ΣP(T) Ep(a I A.A
p(T)
(A(h,TT)(s - y)
At this point we notice we can minimize the above point wise w.r.t (æz, 7) and ignore h. To do that, take the derivative R(A, h) with respect to A(h, (x) because that A(h, (x) E and set it to zero:
dA(h, (xx) B Ruppiule,)) [(A(h, () y) =0
Bymp(ule,)) [(A(h, (z) y)]
E By-pla)arr) [A(h, (x)) = By-r(ularr) lv]
A(h,T) =E By-p(ulas)) [y] Fyler
as desired.
Corollary M.3. For monolithic meta-learner defined in section M. the solution to the meta supervised learning problem the same as in equation 14 for the squared loss ((..y) y) = (g y)² iee. g(T,2) -yya.r -By-p(gla,r) [y]-
Proof. Proof is trivial, replace A(h, (x) with g(T, x) since h is not used. In this case, there is no difference with having an adaptation rule A equipped with another function and a monolithic meta-learner g.
Theorem M.4. The minimizer to equation 12
f(x)
Eypp(r x) [
(15)
where Uyjer = (yy-p(war.)) [y] and is the squared loss M(g v) = (0 - y)
Proof. WLOG, consider the minimizer of equation 12 in the discrete case. particular, have:
Rst(S) Er<p(r) Ez (x. [(f(x)), w)]
min Ey Bupp Byy [((xx) y
min Σ Erpp(typPx B) Byppllerr) [('(x) -
Note we can minimize the above point-wise w.r.t. x only (and not also w.r.t. as we did in proof M.2 Thus, we have want:
f(x) = min B)<p(r)(( Eyrpplyy))))) [(((x) - y)
f(z)E
33

---

Under review as a conference paper at ICLR 2023
at this point it is interesting to observe the visadvantage of supervised learning methods with fixed functions without dependence on the task is that they are forced to consider all task 7 at once. We proceed to take derivatives as in proof M.2 but with this objective:
E P())))
T) pppoll)) [(((z) y)a
d df(x)
E P (T T) yym((lar)) [(f(x) y) =0
( B) [f(x)] = E Br~p(r)p(at T)Eyypp(llas) [v]
f(x) [p(a r) E Brwo(r)Pa r) Byy-p(uaar)))) [g]
p(a 7)
f(x) = [())=Erap(r) E7pp(r)
E Bemp(vla [y]
(16)
Bymp(r) [p(x [)]
proceed by noticing that E2 Expp(r) (p(2 r)] = p(x)) thus:
p(<: 7)
[(x) ETnp(r)
Ey-p(ule,7) [w]
p(x))
p(x [T)
f(z) Σp(r p(T)
[T [Bm-p(ule,r) [v]]
p(x²)
f(x) Σ p(T) pla;, T)
[ By-p(ola,r) [0]]
p(x)
p(T)
f(x) = Σ p(x, T)
[ (Eymp(ula))) [v]]
p(z)
f(x) Σ p(x (T)
F pppplr))))) [v]
[(x) Bymp(atr) (FyMiUla
[(x) E. Eynp(e 7) MylazT]
required by the rightmost RHS of equation 15
Theorem M.S. The minimizer in equation 15 reduces to an expectation only over w.r.t. p(T) of Myler under benchmarks that are balanced. Formally
f(x) BTrpp(r) [olerr] ETwp(()) [Ey-p(ulerr) [v]]
(17)
where Hyjerr = Byrppolle,) [y] and under assumption A1: p(x T) is a constant, i.e. p(æ () kxt R, Vx X, VT I and is the squared loss ((}.8) (g y)
Proof. Recall equation 15
f(x) = Er-p(r m)
Tylz,r
due to Bayes's rule have p(7 ()) p(r)m(alr) and equation p(x)
15 can be re-written follows:
((( T)
f(x) = Erpp(()
p(x)
under assumption Al we have that p(x) 7) does not depend function of x Thus, have:
p(x) Σ p(T)p(( p(x (T) Σ p(r) p(x 7)
Thus we have:
p(x) p(x) Ty
f(z) = ETpp(r)
Vulzer
f(x) = Erap(T) [yleer]
required.
34

---

Under review as a conference paper at ICLR 2023
Remark M.6. Note that assumption Al holds for the common Minilmagenet few-shot learning data set, where p(z [T)
600
Remark M.7. In addition, because all classes are equally likely (e.g. p(class) 61 for the meta-train set) we have P(T) is the same constant independent of the task т. Proof in the appendix, lemma M.8
Theorem M.8. If the tasks are equally likely, then equation 17 becomes an average over conditional predictions over all tasks. Formally, if p(T) + then equation 17 becomes:
[(x)
1 T
Σ
My Mylz,r
(18)
under the squared loss I(). y) = (g) - y)?
Proof. Since (xx = ETo((r)
Bylnr then, plugging p(T) completes proof.
Remark M.9. is interesting to note that without adaptation or dependence on the task t being solved, the supervised learned meta-learner is suboptimal compared to the meta-leamed solution. The proof is simple, and it follows because the meta-learned decision rule was chosen to minimize each term individually, but the supervised learned decision is not of that form. Proof in appendix M.IT. Unfortunately, note that this does not necessarily apply to previous work (Tian et al., 2020).
Remark M.10. Note that remark M.9 does not apply to work (Tian et al. 2020) because that work does depend on a task T during meta-test time by adapting the final layer even if the representation fixed.
Remark M.11. The supervised learning decision rule is suboptimal compared to the meta-learned decision rule.
M.3 THE SUPERVISED LEARNING SOLUTION IS EQUIVALENT TO THE META-LEARNING SOLUTION WHEN THERE IS LOW TASK DIVERSITY
The main idea is that because all tasks are very similar (task diversity is low) essentially means that 7 is not truly an input to the adaptation rule or monolithic meta-learner). Equivalently, the problem is essentially single task problem, so the task is implicitly an input to any method used. Therefore, since the task conditioning does not exist, then the optimization problem is the same for the meta-learned solution and when there is fixed supervised learning feature extractor.
Sketch argument:
Theorem M.12. Assume TI for any tasks in T and the data sets are balanced (i.e. same number of images x for each task). Then we have the meta-learned solution is the same as the supervised learning solution with shared embeddings: fx( A(fmi, (z))
Proof. Consider the optimization problem, for supervised learning:
min Ex (arrprr)e-pppttt Besptalr A.
Burppully Burp(lle.)) [(A(h, r)(x) - y)²
If every pair of tasks is equal, it means their distributions are equal p(a, r) = p(x, y) (mean-
ing t be ignored). Thus, the solution to the supervised learning problem is: fx) E, E.EEpaa.y [v] = (p(aa)) [v] = Vr.- Now for the meta-learning problem we have: A(ʃml: (x)) = "f2,r = (yr-p(ule.r) [v] but due to every pair of tasks being equal means p(æ, 7) p(x, (i.e.
all task share the same distributions) we have: A(fmi, (x) Eywp(ula,)) = 15 [v] Df=
which is the same as the solution as in fat- Thus fx((x) = A (fmt,T)(1))
Remark M.13. Proofs were presented in the discrete case clarity, but it trivial to expand them to the continuous case e.g., using integrals instead of summations.
N SUMMARY OF COMPUTE REQUIRED
We used an internal compute cluster with wide varied of GPUs. We used Titan X GPUs for most five layer CNN experiments. We used A40 and dgx-A100 A10 GPUs for Resnet12 experiments, with 48 GB and 40 GB GPU memory respectively. We did notice that the Resnet12 1 architecture used
35

---

Under review as a conference paper at ICLR 2023
from previous work (Tian et al. 2020) required more memory than Resnet18 and Resnet34 used in Task2 Vec (Achille UCLA et al. 2019). By requiring more memory, we mean we did not have many memory out of bound issues with Cessetl8/Resmeee but did have memory issues with Resnet 22. In addition, our episodic meta-learning training for MAMI. used Learn21 's (" ("Arnold et al. 2020) distributed training speed up experiments. Experiments took 1 1-2 weeks with MAML in a single GPU to potentially 2-3 days with multiple GPUs (we used 2, 4 to 8 GPUs depending on availability). For synthetic experiments used Titan GPUs with 16GB of GPU memory. Experiments took around 1-2 days on average with single GPU. For more precision check the experimental details section
36


Reviews:
[{"review_id": "f0x12TL6sNU", "reviewer": "Reviewer_nDz4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Model agnostic meta-learning (MAML) and transfer learning (TL) are popular techniques for learning under data scarcity such as  few-shot learning. While the effectiveness of MAML over TL has been shown for medical image analysis tasks, the verdict on classical benchmark datasets in this domain is still unclear. This paper presents a systematic study comparing MAML and TL under fair conditions (architecture, optimizer, etc.) across diverse tasks. The authors introduce a diversity coefficient to measure task diversity using Task2Vec representations. Using this measure, the authors show that TL and MAML results are comparable outcomes when the task diversity is low. This observation holds even as the model size changes.\n", "summary_of_the_review": "Overall, the paper is an interesting attempt to contextualize the behavior of MAML and USL algorithms. While I agree that task diversity is an important metric for evaluation, I find the work needing additional experiments. There are interesting patterns in the results, but I am afraid the experiments are not complete enough to draw conclusions. ", "strength_and_weaknesses": "Strengths\n\nThe earlier work by Raghu et al, did compare MAML against multi-class (USL) and multi-task learning. However, their analysis was restricted to studying the quality of features learned through different training regimes. The key difference between Raghu etal’s work and this paper is the training of an additional classifier (adapt head only). The feature extraction backbone is not very influential in the low-task diversity setting as logistic regressors trained using MAML and USL feature extraction backbones result in comparable performance. This is an interesting observation and main strength of the paper (though I have some reservations as elucidated later)\n\n[Raghu et al] Aniruddh Raghu, Maithra Raghu, and Samy Bengio. Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML. Technical report, 2020. URL https://arxiv. org/abs/1909.09157.\n\nThe diversity metric based on the Hellinger distance of the Task2Vec representations is quite intuitive and sound.\n\nThe authors conduct experiments using benchmark datasets for few-shot learning. The variety of distance metrics such as SVCCA, PWCCA, and the likes reinforces the observations from the earlier experiments. \n\nWeaknesses\n\nThe diversity metric is computed using only the Resnet backbones (Table 1). I wonder if the similarity of the backbone architecture is a cause for the similar diversity results? Experimenting with diverse architectures (perhaps VGG and densenet variants) might strengthen the observation. \n\nMy primary concern is the task that the authors have used for quantifying diversity. The results in Table 1 and 2 are from 5-way 5-shot tasks (though I understand that 20 shots were used to estimate the diversity score in Table 1). I believe that a 5-way task is insufficient to draw out the diversity across tasks, especially when the tasks are sampled from miniimagenet or CIFAR-FS datasets. I would urge the authors to try out 10-way or even higher N-way tasks, with the same number of shots. If the authors were keen to stick to a 5-way task, I would have expected results from a 5-way 1-shot setting as well. The current set of results, though suggestive of a trend, are not convincing enough to draw an inference.\n\nI like the experiment on increasing the model size as a function of the number of filters. The results are presented in Figure 2. However, here too, I find that the authors have stopped the experiment at a critical juncture. I would have liked to see runs beyond 30 filters. There is an increasing trend in favor of USL. This should ideally get amplified with larger model size. It is difficult to conclude by stopping at 30.\n\nI also find results presented in Figure 3 to be quite strange. How can the meta-test accuracy increase with higher Task2Vec diversity coefficient? One would have expected the accuracy to actually reduce with harder tasks, the current trend seems to be quite counter intuitive as both MAML and USL have higher accuracies on difficult tasks. Further, similar to the previous observation, the authors stop at 0.5, when things start to get interesting - the difference between MAML and USL is apparent here. I would urge the authors to continue further (as indicated in the last subplot of Figure 3).\n\nWhile using synthetic data helps to control task diversity, I would urge the authors to try a more natural setting of higher task diversity - cross domain-few shot learning tasks. This should give better insights into behavior of both MAML and USL.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_XRr7wbohiC", "reviewer": "Reviewer_NZ7N", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a novel diversity metric that measures the dissimilarity between few-shot learning tasks. The authors find that popular few-shot learning benchmarks - MiniImagenet and Cifar-fs, score low on the proposed diversity metric. The authors then compare the performance of MAML to transfer learning in these two benchmarks and find that under standardized hyperparameters, MAML and transfer learning are statistically similar in performance across many model configurations. Thus, the authors conclude that the similarity in performance between MAML and transfer learning is largely due to the low task diversity.\n", "summary_of_the_review": "In balance, I think the paper does not make a sufficiently novel claim to meaningfully contribute to our collective knowledge, thus I recommend rejection.", "strength_and_weaknesses": "Overall, I appreciate the thoroughness in the execution and presentation of the experiments in this work. The ablation over multiple probe networks lend credence to the robustness of the proposed metric over the choice of the probe network. The hyperparameters and training details of each model used in the experiments are well documented. As such, the paper states convincingly that the performance of MAML and transfer learning are statistically equivalent on MiniImagenet and Cifar-fs. \n\nThe writing quality of the paper is fairly strong. The authors went into great detail to explain background works, including formulations for the feature similarity metrics, in the appendix, which makes the work nicely self-contained. There is no noticeable ambiguity in any of the formulations/algorithms used in the paper. The proofs regarding similarity of a MAML-type algorithm and a “fixed representation meta-learner without adaptation” are correct, albeit straight-forward.\n\nThere are some problems that I notice in this work, which I list in the order of severity. \n\n1. The claim that MiniImagenet and Cifar-fs has low diversity is stated without context. While indeed the proposed diversity coefficient can range from 0 to 1, it is not clear where do real meta-learning datasets actually lie on this 0-to-1 scale. Without this context, one cannot claim that a value of 0.1 or 0.06 should be considered “low”. \n\n2. The authors suggest that their experiments point towards the following conclusion: “if the task diversity is low, then transfer learned solutions might fail to outperform meta-learned solutions”. In close examination, this claim is in fact vacuous (or unsupported, depending on the interpretation of if). Even if we assume the claim regarding “low” diversity to be correct, the experimental results (on real datasets) in this paper only indicates that transfer learned solutions fail to outperform meta-learned solutions in “low” diversity tasks, but this could be the case regardless of whether task diversity is low. The observations have only shown co-occurrence (“roses are red, sky is blue”), but a non-vacuous claim at least requires correlation (“roses are more red when sky is blue”), which requires negative examples (“roses are not red, sky is not blue”). \n\n3. The synthetic experiment in this paper may refute the above issue as it shows transfer learning out-performing MAML (the Gaussian experiment) in “high” task diversity settings. Yet, the control variable (task diversity) has a significant confounding variable that could directly explain the observation (USL beating MAML), the variable being task difficulty. The authors adjust the variance of the distribution of class means (sigma_m) to create a range of benchmarks with different diversity values. However, increasing the variance of class means also increases the average distance between the class means, which in turn increases the margin in classification boundary between each class for any given 5-way classification episode, thereby making the classification problem easier. This can be easily observed in figure 3, where the meta-test accuracy of both methods increases consistently with increased diversity, indicating a decrease in difficulty. We can then postulate that USL out-performs MAML in the 3 benchmarks on the right-end of the graph not due to diversity, but due to difficulty. A likely explanation is that the L-BFGS algorithm used for optimizing the last layer is fully converged, while MAML-5’s 5 gradient steps are not fully converged. When the classes are so well separated (e.g. when inter-class variance is 20 times that of intra-class variance), a linear model (i.e. last layer of a mlp) can be trained to convergence on very few examples without fear of overfitting, as the likelihood of outliers is vanishingly small. This explanation predicts that using L-BFGS on the last layer with the feature-extractor learnt by MAML should close the performance gap, as the difference is in the meta-test time optimization, not MAML vs USL. In addition, this experiment highlights the fact that the proposed diversity metric is sensitive to task difficulty, which should be an unwanted side effect. The authors should conduct experiments in settings where the diversity varies while the difficulty is controlled, to illustrate how MAML and USL responds to various diversity levels.\n4. While the authors do not make any strong claims w.r.t their theoretical analysis, I want to point out that the analysis misses the point by analyzing the learner trained by equation 12 instead of equation 13. Without any task adaptation or conditioning, it is rather obvious that the decision rule can only learn the mean prediction over all tasks (as pointed out by the authors themselves). This decision rule is entirely useless, as it will only learn trivial solutions when the assignment between classes and labels is shuffled (e.g. all standard benchmarks). On the other hand, the result of when equation 11 is equivalent to 12 in the zero task diversity setting is weak in its current form - a much more useful result would be to show that the two learners tend towards the same when diversity converges to zero.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Qa2A13v0Ak", "reviewer": "Reviewer_NP6p", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies when and how meta-learning algorithms should be used to solve few-shot classification tasks. The authors introduce a “diversity coefficient”, which they use to measure how varied a set of tasks is. They show that (some) standard few-shot classification have low diversity — in other words, all tasks are easy so both MAML and transfer learning perform equally well. From those experiments, they conclude that meta-learning and transfer learning methods yield equivalent performance on few-shot classification tasks.", "summary_of_the_review": "- Main strength:\n    - Use of Task2Vec to measure task similarity in few-shot classification.\n- Main weakness:\n    - Lack of novelty — most insights in the paper are either well-known or obsolete by now.", "strength_and_weaknesses": "- Strengths:\n    - The use of Task2Vec to compute diversity is novel in the context of few-shot classification (but Achille et al. already used it for similarity across vision tasks).\n    - I appreciated the experiments on the toy Gaussian experiments: it clearly shows a strong correlation between the proposed diversity coefficient, ground truth Hellinger distance, and the meta-test accuracy.\n- Weaknesses:\n    - More thorough validation: while the paper focuses on MiniImageNet and CIFAR-FS (which are well-known to have similar train and test tasks), I wonder if the same results would hold on FC100 and TieredImageNet. The latter benchmarks are specifically designed to increase task diversity, and I wonder if the proposed metric would be able to quantify this diversity. Second, I would like to see a comparison of methods to measure task diversity. Three come to mind:\n        - First, the authors could use the ones they mention (eg, CCA, CKA) to measure similarity between the obtained representations.\n        - Second, Dhillon et al., 2019 propose a measure of task difficulty, which can be used to measure diversity (along the axis of difficulty).\n        - Third, Ethayarajh et al., 2022 (Understanding Dataset Difficulty with V-Usable Information) also propose a measure of difficulty between datasets, but this measure actually builds on the information contained in a dataset (akin to the diversity in this paper).\n    - Missing discussion of prior art: several important prior works are missing from the discussion in this paper. In its current form, I am not sure that the paper offers novel insights.\n        - Dhillon et al., 2019 shows the same results as Tian et al., 2020, and so does Wang et al., 2019 (SimpleShot: Revisiting Nearest-Neighbor Classification for Few-Shot Learning).\n        - Arnold and Sha, 2021 (Embedding Adaptation is Still Needed for Few-Shot Learning) also show that transfer learning methods shine when train and test tasks are similar, but MAML has the upper hand whenever they are dissimilar. Collins et al., 2020 (How Does the Task Landscape Affect MAML Performance?) compare the settings under which MAML is competitive and can outperform a pretrained feature extractor.\n        - Several recent works (incl. at ICLR this year) have carefully studied MAML and drastically improved its performance on few-shot classification. See for example: Ye and Chao, 2022 (How to Train Your MAML to Excel in Few-Shot Classification).\n    - Presentation: the paper needs significant work in terms of presentation to be up to the ICLR standards. Some examples:\n        - Equation 2 and 3 have formatting issues.\n        - Citation: Achille UCLA et al., 2019 should not have UCLA.\n        - Figure 1 needs improvements to be legible. It is also misleading — in few-shot classification, we typically use 600 or 2000 tasks to compute confidence intervals. Doing so would significantly shrink the error bars.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HIQfioZlv6", "reviewer": "Reviewer_Bxys", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the question of whether transfer learning (USL) is superior to meta-learning (MAML) for few-shot classification (multi-task), as suggested by recent studies.\nIt shows that the two methods have identical performance when task diversity is low, while their performance is different when task diversity is high.\nIt proposes a new measure of task diversity and argues that previous studies had not compared the two methods fairly.\nIt also shows that predictions of USL and MAML are similar although they learn different representations.\n\n", "summary_of_the_review": "Overall the strengths slightly overweight the weaknesses, but the paper could be improved a lot.", "strength_and_weaknesses": "Strengths\n\n- The debate around whether transfer learning or meta-learning should be used for few-shot classification is interesting and not resolved.\n- Experiments are convincing: they run with multiple architectures, model sizes, datasets, metrics and have several controls.\n\n\nWeaknesses\n\n- Showing differences between USL and MAML for high task diversity is crucial for testing the central hypothesis of this paper.\nYet, this is done only in synthetic Gaussian data in section 4.5.\nIt would be good to have a high diversity benchmark with real data, instead of Gaussians.\nThis could be done, for example, by separating the 100 classes of CIFAR-FS into hierarchies, with separate tasks for animals, vehicles, etc.\nOne way of doing this is described in https://ojs.aaai.org/index.php/AAAI/article/view/20590\n\n- I find it very surprising that USL is better than MAML for high task diversity. \nMAML is designed to deal precisely with this situation, while USL is not.\nThe authors should discuss this result in much more detail.\n\n\nMinor points\n\n- I don't understand why Eq.2 needs to be in this paper, it doesn't seem to be used anywhere.\n\n- \"distances based on Task2Vec are good approximations to the ground truth distance of task distribution.\" \nI don't understand this sentence, FIM is a metric in the space of parameters (or parametric distribution), not in the space of tasks.\nWhat is the \"ground truth distance of task distribution\" anyway?\nOverall, section 3.3 includes several unclear (and maybe wrong) statements, this section could be entirely removed.\n\n- Results in Table 2 are not entirely clear to me. \nHow can USL perform worse than chance (<20%) in any circumstance?\nWhy results depend so much on the adaptation method used?\n\n- \"the two methods are equivalent in a statistically significant way\".\nThis statement is repeated several times in the paper but is incorrect. \nThe correct statement is: \"differences in the two methods are not statistically significant\"\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject