The Effect of diversity in Meta-Learning
Authors: Ramnath Kumar
Tristan Deleu
Yoshua Bengio
Venue: ICLR 2022 Submitted (2022)
Keywords: Meta-Learning
Few-shot learning

Abstract: Few-shot learning aims to learn representations that can tackle novel tasks given a small number of examples. Recent studies show that task distribution plays a vital role in the performance of the model. Conventional wisdom is that task diversity should improve the performance of meta-learning. In this work, we find evidence to the contrary; we study different task distributions on a myriad of models and datasets to evaluate the effect of task diversity on meta-learning algorithms. For this experiment, we train on multiple datasets, and with three broad classes of meta-learning models - Metric-based (i.e., Protonet, Matching Networks), Optimization-based (i.e., MAML, Reptile, and MetaOptNet), and Bayesian meta-learning models (i.e., CNAPs). Our experiments demonstrate that the effect of task diversity on all these algorithms follows a similar trend, and task diversity does not seem to offer any benefits to the learning of the model. Furthermore, we also demonstrate that even a handful of tasks, repeated over multiple batches, would be sufficient to achieve a performance similar to uniform sampling and draws into question the need for additional tasks to create better models.

Under review as a conference paper at ICLR 2022
THE EFFECT OF DIVERSITY IN META-LEGRNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Few-shot learning aims learn representations that can tackle novel tasks given a small number of examples. Recent studies show that task distribution plays a vital role in the performance of the model. Conventional wisdom that task diversity should improve the performance of meta-learning. In this work, we find evidence to the contrary; we study different task distributions on a myriad of models and datasets evaluate the effect of task diversity meta-learning algorithms. For this experiment, we train on multiple datasets, and with three broad classes of meta-learning models Metric-based (i.e., Protonet, Matching Networks), Optimization-basee Bayesian meta-learning models (i.e., CNAPs).. Our experiments demonstrate that the effect of task diversity on all these algorithms follows a similar trend, and task diversity does not seem to offer any benefits the learning of the model. Further- more, we also demonstrate that even handful of tasks, repeated over multiple batches, would be sufficient achieve a performance similar to uniform sampling and draws into question the need for additional tasks to create better models.
(i.e., MAML, Reptile, and MetaOptNet), and
INTRODUCTION
It is widely recognized that humans can learn new concepts based very little supervision, i.e., with few examples (or "shots"), and generalize these concepts to unseen data as mentioned by Lake et al. 12. Recent advances in deep learning, on the other hand, have primarily relied on datasets with large amounts of labeled examples, primarily due to overfitting concerns in low data regimes. Although the development of better data augmentation and regularization techniques can alleviate these concerns, many researchers now assume that future breakthroughs in low data regimes will emerge from meta-learning, or "learning to learn." " Here, we study the effect of task diversity in the low data regime and its effect on various models. In this meta-learning setting, model is trained on handful of labeled examples project examples of different classes and generalize this knowledge to unseen labels at test time. Although this setting is often used to illustrate the remaining gap between human capabilities and machine learning, we could argue that the domain of meta-learning is still nascent. The domain of task selection has remained virtually unexplored in this setting.
time under the assumption that it will learn how to correctly
Conventional wisdom is that the performance of the model will improve as we train more diverse tasks. To test this hypothesis to its limits, we define various task samplers which either limit task diversity by selecting subset of overall tasks improving task diversity using approaches such as Determinantal Point Processes (DPPs) proposed by Macchi (1975).
Our contributions in this work are as follows:
We show that, against conventional wisdom, task diversity does not significantly boost performance in meta-learning. Instead, limiting task diversity and repeating the same tasks over the training phase allows the model to obtain performances similar to models trained on Uniform Sampler without any adverse effects.
We also show that increasing task diversity using sophisticated samplers such as DPP or Online Hard Task Mining (OHTM) Samplers do not significantly boost performance. In- stead, the dynamic- DDP Sampler harms the model due to the increased task diversity.
We empirically show that repeating tasks over the training phase can perform similarly to model trained on the Uniform Sampler, achieving similar performance with only

---

Under review as a conference paper at ICLR 2022
fragment of data. This key finding questions the need to increase the support set pool to improve the model's performance.
RELATED WORKS
Meta-leaming formulations typically rely on episodic training, wherein an algorithm adapts to task, given its support set, to minimize the loss incurred on the query set. Meta-learning methods differ in terms of the algorithms they learn, and can be broadly classified under four prominent and Bayesian-based approaches. Metric- methods such as Koch et al. (2015); Vinyals et al. (2016); Snell et al. (2017); Sung et al. (2018) operate on the core idea similar to nearest neighbors algorithm and kernel density estimation. These methods also called non-parametric approaches. Model-based methods such Santoro et al. (2016); Munkhdalai & Yu (2017) depend on a model designed specifically for fast learning, which updates its parameters rapidly with a few training steps, achieved by its internal architecture con- trolled by another meta-learner model. Generic deep learning models learn through backpropagation of gradients, which are neither designed to cope with small number of training samples nor con- verge within a few optimization steps. To address this, Optimization-bnl -based methods such as Ravi & Larochelle (2016); Finn al. (2017); Nichol al. (2018) were proposed, which were better suited to learn from a small number of samples. However, all the above approaches are deterministic and are not the most suited for few-shot problems that are generally ambiguous. Hence, Bayesian-bee based methods such as Yoon et al. (2018); Requeima al. (2019) were proposed which helped address the above issue.
classes: Metric-based, Model-based, Optimization-basee
Although research in meta-learning models has attracted much attention recently, the effect of task diversity is virtually unexplored in the domain of meta-learning, However, task sampling and task diversity have been more extensively studied in other closely related problems such active learn- ing. Active learning involves selecting unlabeled data items in order to improve an existing classifier. Although most of the approaches in this domain based on heuristics, there are few approaches to sample a batch of samples for active learning. Ravi & Larochelle (2018) proposed an approach to sample a batch of samples using protonet as the backbone architecture. The model tries to maximize the query set, given support set and unlabeled data. Other works such as Hsu et al. (2018) proposed a framework named CACTUs, which samples tasks/examples using relatively simple task construction mechanisms such as clustering embeddings. The unsupervised representations learned via these samples lead to a good performance on various downstream human -penan-eee tasks.
Although nascent, few recent works aim to improve meta-learning by explicitly looking at the task structure and relationships. Among these, Yin et al. (2019) proposed an approach to handle the lack of mutual exclusiveness among different tasks through an information-theored objective. In addition, several popular meta-learning methods Lee et al. (2019); Snell et al. (2017) improve the meta-test performance by changing the number of ways shots of the sampled meta- training tasks, thus increasing the complexity and diversity of the tasks. Other works such as Liu et al. (2020a) proposed approach to sample classes using class paiss-pair-bssed sampling and class- based sampling. The Class-pair based Sampler selects pairs of classes that confuse the model the most. The class-based Sampler samples each class independently and does not consider the task's difficulty as a whole. Our OHTM sampler is similar to the Class-pair based Sampler. Other works such as Liu et al. (2020b) propose to augment the set of possible tasks by augmenting the pre- defined set of classes that generate the tasks with varying degrees of rotated inputs as new classes. Other works such as Setlur et al. (2020) look at the structure and diversity of tasks specifically through the lens of support set diversity, and show that, surprisingly, reducing diversity (by fixing support set) not only maintains-but in many cases, significantly improves-the performance of meta-learning. This experiment is very similar to our No Diversity Task Sampler if the size of the support set is equal to the number of classes per task. However, in this work, we extend their work on MetaOptNet, Protonet to many other models and myriad of samplers to better understand task diversity in meta-learning. To the best of our knowledge, we are the first to study the effect of task diversity in meta-learning to this extent.
regularized
2

---

instances provided Under review as a conference paper at ICLR 2022
P
V
4
P
E
Task
12
A
F
4P
4
P
2
B
B
Tasl
B
4
3
Y
17
L
3
6
CP
B
Task
+
B
P
C
B
Batch
Batch 2
Batch 3
Batch
Batch 2
Batch 3
Batch
Batch
Batch
Uniform sampler
(b) No Diversity Task sampler
(c) No Diversity Batch sampler
Figure I: Illustration of (a) the Uniform Sampler, (b) the No Diversity Task Sampler, and (c) the No
Diversity Batch Sampler.
3 BACKGROUND
Here, we review some of the fundamental ideas required to understand our few-shot learning exper- iments better.
3.1 EPISODIC FEW-SHOT LEARNING
In episodic few-shot learning, an episode is represented as a K-way, N-shot classification problem where N is the number of examples per class and K is the number of unique class labels. During training, the data in each episode is provided as a support set S = {(I1,1 V1.1),..., (IN.K, UN. k)} where Vijj E RDP is the i-th instance of the j-th class, and yj E {0, 1 is its corresponding one-hot labeling vector. Each episode aims optimize function f that classifies through a "query" set Q. containing instances of the same class as S. This task is difficult because N is typically very small (e.g, to 10). The classes change every episode. The actual test set used to evaluate model does not contain classes seen in support sets during training. In the task-distribution view, meta-learning
general-purposs learning algorithm that can generalize across tasks and ideally enable each new task to be learned better than the last. We can evaluate the performance of w over a distribution tasks p(7). Here we loosely define a task to a dataset and loss function T (D,C), Learning how to learn thus becomes:
min E C(D:w) Trpp(T)
where C(D; w) measures the performance of model trained using w on dataset D and p(T) indicates the task distribution. In this experiment, extend this setting such that we vary the task diversity in the train split to study the effects on test split, which remains to use uniform or random sampling for tasks.
3.2 DETERMINANTAL POINT PROCESSES (DPPs)
DPP is a probability distribution over subsets a ground set y. where we assume 21, and N = 1 An L-ensemble defines a DPP using real, symmetric, and positive- definite matrix I. indexed by the elements of y. The probability of sampling subset Y AAC y can be written as:
P(Y A) LAt
(2)
where LA [iglice is the restriction of L to the entries indexed by the elements of A. As L is positive semi-definite, there exists d x N matrix v such that I TT vp where d < N. Using this principle, we define the probability of sampling as:
P(Y A) x det LA Vol(((V))ee)),
(3)
3

---

Under review as a conference paper at ICLR 2022
B
Thek
P
Thck 2
1
6
4
B
D
0
Pp
Batch 1
Batch 2
Batch 3
Thsk
7
B
Batch
Batch 2
Batch 3
(a) No Diversity Tasks per Batch sampler
(b) Single Batch Uniform sampler
Figure 2: Illustration of (a) the No Diversity Task per Batch Sampler, and (b) the Single Batch Uniform Sampler.
where the RHS is the squared volume of the parallelepiped spanned by (V. In Eq. 3, V is defined as the feature vector of element i, and each element Lij in L is the similarity measured by can verify that DPP places higher probabilities dot products between elements and Hence,
on diverse sets because the more orthogonal the feature vectors are, the larger the volume paral- lelepiped spanned by the feature vector is. In this work, these feature embeddings represent class embeddings, which are derived using either pre-trained protonet model or the model being trained as discussed in Sec. 3.3.
In DPP, the cardinality of a sampled subset, |A|, is random in general. A k-DPP is extension of the DPP proposed in the work of Kuhn et al. (2003), where the cardinality subsets are fixed as k (i.e., |A| =). In this work, we use k-DPPs as an off-the-shelf implementation to retrieve classes that represent task used in the meta-learning step.
3.3 TASK SAMPLING
In this work, experiment with eight distinct task samplers, each offering different level of task diversity. To demonstrate the task samplers, we use a 2-way classification problem with a meta- meta-batch batch size of and denote each class with a unique alphabet from the Omniglot dataset.
Uniform Sampler This is the most widely used Sampler used in the setting of meta- leatnnl...ning... The Sampler gives equal probability to every task and is intuitively random sampler. An illustration of this Sampler shown in Figure
No Diversity Task Sampler In this setting, we uniformly sample one set of the task at the begin- ning and propagate the same task across all batches and meta-batches. Note that repeating the same class over and over again does not simply repeat the same images/inputs as we episodically retrieve different images for each class. An illustration of this Sampler shown in Figure
No Diversity Batch Sampler In this setting, we uniformly sample one set of tasks for batch one and propagate the same tasks across all other batches. Furthermore, we shuffle these tasks to enforce that the model does not overfit. An illustration of this Sampler is shown in Figure 1.
No Diversity Tasks per Batch Sampler In this setting, we uniformly sample one set of tasks for a given batch and propagate the same tasks for all mnea-batches. We then repeat this same principle for sampling the next batch. Furthermore, we shuffle these tasks to enforce that the model does not overfit. illustration this Sampler shown in Figure
Single Batch Uniform Sampler this setting, we set the meta-batch size to one. This Sampler intuitively the same diversity task per batch sampler, without the repetition of tasks. This
4

---

Under review as a conference paper at ICLR 2022
w
V
A
Meta-Learning algorithm
Meta-Learning algorithm
?
M
Toe
M
VI
kDDP
Hard Took
Hard tasks
k-DPP
Datch
Batch
Batch 2
Batch
(a) Online Hard Task Mining sampler
(b) Static DPP sampler
(c) Dynamic DPP sampler
Figure 3: Illustration of (a) Online Hard Task Mining Sampler, (b) the Static DPP Sampler, and (c) the Dynamic DPP Sampler.
Sampler would be an ideal ablation study for the repetition of tasks the meta-learning setting. An illustration of this Sampler is shown in Figure 2.
Online Hard Task Mining Sampler This setting is inspired by the works of Shrivastava et al. (2016) where they proposed OHEM, which yielded significant boosts in detection performance on benchmarks like PASCAL VOC 2007 and 2012. However, to reproduce OHEM for meta-learning, we only apply the OHEM sampler for half the meta-batch size and uniform sampler for the remain- ing half. This approach would allow us to involve many tasks and not restrict us to only known tasks. Furthermore, avoid OHEM the initial stages, we sample tasks with a uniform sampler until the buffer of tasks seen by the model becomes sufficiently big, say 50 in our case. An illustration of this Sampler is shown in Figure 3.
Static DPP Sampler Determinantal Point Processes (DPP) have been used for several machine learning problems such as the works of Kulesza & Taskar (2012). They have also been used in other problems such the active learning settings in the works of BIyık et al. (2019) and mini-batel sampling problems in the works of Zhang et al. (2019). These algorithms have also inspired other works in active learning in the batch mode setting, such Ravi & Larochelle (2018). In this setting, we use DPP as an off-the-shelf implementation to sample tasks based on task embeddings. These task embeddings are generated using our pre-trained protonet model. The DPP instance is used to sample the most diverse tasks based on these embeddings and used for meta-learning. An illustration of this Sampler is shown in Figure 3.
Dynamic DPP Sampler In this setting, we extend the previous sDPP setting such that the model training generates the task embeddings. The Sampler is motivated by the intuition that selecting the most diverse tasks for given model will help learn better. Furthermore, to avoid DPP in the initial stages, we sample tasks with a uniform sampler until the model becomes sufficiently trained, say 500 batches in our case. An illustration of this Sampler is shown in Figure 3.
4 EXPERIMENTS
The experiment aims to answer the following questions: (a) How does task diversity affect meta- learning? (b) sophisticated samplers such as OHEM or DPP offer any significant boost in per- formance? (c) Are there any rule of thumb or general good practices when it comes to sampling tasks?
To make exhaustive study the effect of task diversity in meta-learning, we train on four datasets: Omniglot Lake et al. (2011), minilmagenet Ravi & Larochelle (2016). tieredImageNet Ren et al. (2018), and Meta-Dataset Triantafillou al. (2019). With this selection of datasets, cover both simple datasets, such as Omniglot and minilmageNet, as well as the most difficult ones, such leered and Meta-Datased We train three broad classes of meta-learning models

---

Under review as a conference paper at ICLR 2022
Ounigglt wrway 1-shot
60
MAMI.
Profenet
Mitcching
MetaClptNet
CNAPS
miniknagerNet S-waay I-shot
70
600
Aceunicy
60
MAML
Reptile
Protonet
Matching Net.
MetaOptNee
CNAPs
Uniform
Divunity
Divensiyy
Deversity Tasks
OHTM d-DPP
Figure 4: Average accuracy on Omniglot 5-way 1-shot & minImageNet 5-way 1-shot, with 95% confidence interval. All samplers are poorer than the Uniform Sampler and are statistically signifi- cant (with a p-value = 0.05). We use the symbol * to represent the instances where results are not statistically significant and similar the performance achieved by Uniform Sampler.
Ornniglat 20-my I-shot
100
70
50
MAML
Reptile
Matching
MetaOptNes
CNAPs
Unifarmm
No Divensity Tuale
No Diversity Batch
No Diversity Taske/Bateh
Single
OHTTM
&DPP
d-DPP
Figure 5: Average accuracy on Omniglot 20-way I-shot, with 95% confidence interval. We denote all samplers that are worse than the Uniform Sampler and are statistically significant (with a p-value p 0.05) with v, and those that are significantly better than the Uniform Sampler with A.
these datasets Metric-bused (i.e., Protonet, Matching Networks), Optimization- (i.e., MAML, Reptile, and MetaOptNet), and Bayesian meta-learning models (i.e., CNAPs). More details about the datasets which were used in our experiments are discussed in App. A.1. More details about the models and their hyperparameters are discussed in App. A.2. We created a common pool of 1024 randomly sampled held-out tasks to test every algorithm in our experiments to make an accurate comparison. For all experiments, we assessed the statistical significance our results based on a paired-difference t-test, with p-value 0.05.
4.1 RESULTS
In this section, we present the results of our experiments. Figure 4 presents the performance of the six models on the Omniglot and minilmageNet under different task samplers in the 5-way 1-shot setting. Table the Appendix presents the same results with higher precision.
We also reproduce our experiments on the 20-way 1-shot setting the Omniglot dataset to establish that these trends are shared across different settings. Figure presents our performance of the mod- els under this setting. Furthermore, the results on the 20-way 1-shot experiments presented in Table 3 in the Appendix with higher precision. To further establish our findings, we also present our result notoriously harder datasets such as tieredImageNet and the performance of the models on the tieredImageNet. Again, Table 2 in the Appendix presents the same results with higher precision.
Figure 6 presents

---

Under review as a conference paper at ICLR 2022
sireillangeee S-way 1-eboo
Reptile
Matching
MetaOptNNet
Meta-Dataset (Traffee Sigus) S-way I-shot
30
20
MAML
Reptile
Prosomet
Matching Nec.
MetaOptNet
CNAPs
Meta Datacet (MSCOCO) S-way 1 shot
50
Acouray
30
MAML.
Repille
Prosomet
Matching Net.
MetaOptNNe
CNAPSs
Diversity
Diversity
Diversity Thsks/Tatch
Single
Figure 6: Average accuracy on tieredImageNet 5-way I-shot, Meta-Dataset Traffic Sign 5-way 1-shot & Meta-Datasel MSCOCO 5-way 1-shot, with 95% confidence interval. We denote all samplers that are worse than the Uniform Sampler and are statistically significant (with a p-value p 0.05) with and those that are significantly better than the Uniform Sampler with A.
Figure presents the performance of the models on the Meta-Dataset Traffic Sign and Meta-Dataset MSCOCO datasets. We only present the results on Traffic Sign and MSCOCO the Meta-Datased as these two sub-datasets are exclusively used for testing and are an accurate representation of the generalization power the models when trained with different levels of task diversity. Other results on the Meta-Dataset are presented in Table 4. We empirically show that task diversity does not lead any significant boost the performance of the models. In the subsequent section, we discuss some of the other key findings from our work.
DISCUSSION
In this section, we diseuss few empirical results from our experiments and shed light on some of the key findings from our research.
Poor performance by NDT Sampler The lowest performance is consistently obtained by the No Diversity Task Sampler, which reasonable since the model only sees one task throughout its training. What is fascinating is that just one task is sufficient for the model to reach reasonably decent performance in most cases. We notice instances where NDT Sampler performs very well on a few sub-datasets of the Meta-Dataset. This can be explained by the fact that the model has only be trained for single sub-dataset and has relatively less class variability (noise) when compared to training multiple sub-datasets.
Poor performance by Single Batch Uniform Sampler Consequently, the Single Batch Uni- form Sampler does perform poorly on most datasets, including Omniglot, minilmageNet, and tieredImageNet. This is reasonable since the model is trained a tiny pool of the dataset. How- ever, we notice instances where the Sampler performs very well on a few sub-datasets of the Meta- Dataset. We hypothesize that training fewer samples keeps the model unaware of the inherent

---

Under review as a conference paper at ICLR 2022
class variability generated by training diverse datasets and aids better performance in the case of Meta-Dataset.
Disparity between Single Batch Uniform and NDTB Sampler Another exciting result is the Disparity between Single Batch Uniform Sampler and No Diversity Tasks per Batch Sampler. As mentioned earlier, the only difference between the two samplers is that tasks are repeated in the latter. However, this repetition seems to offer great deal of information to the model and allows the model to perform on par with the Uniform Sampler. It might be possible that the Single Batch Uniform Sampler obtains the performance observed by the No Diversity Tasks per Batch Sampler trained for enough epochs. However, it would be safe to comment that the convergence of the model is significantly faster in the latter. Thus, repeating tasks might help speed up the convergence of the model when we have a fixed and handful amount of data. However, the same is not valid for models trained on Meta-Dataset. Although both samplers are trained over similar pool of datasets, the NDTB sampler sees more data and might lead to more inherent class variability generated by training on diverse datasets. This might explain why repeating tasks leads lower performance in this case.
Disparity between s-DPP and d-DPP Sampler We also note that s-DPP and d-DPP samplers do not offer any boost in performance when compared the regular Uniform Sampler. Furthermore, there seems to be a significant disparity between these two samplers. We believe that d-DPP, which computes the most diverse tasks at regular intervals, harms the model with the diverse tasks since we observe that the model's performance degrades over epochs. For example, consider the scenario where the model is trained on tasks involving dogs and tractors. This task is relatively easy to learn and would not require the model to fine-tune great deal. However, during test time, suppose our task involves classifying cats and dogs; this would be a problem since the model has not learned the intricacies of the two classes. Thus, diversity seems to do more harm than good in this case. The best example this is observed by Matching Networks in Omniglot 5-way 1 1-shot setting as shown in Figure 10, where each instance of diverse sampling harms the model significantly.
Limitation of samplers with DPP backbone Samplers such as s-DPP and d-DPP, which use DPP to sample diverse tasks, require task embeddings of every class in the dataset. Computing these task embeddings, although intensive, might be sustainable for small datasets such as Omniglot, minilmageNet, and rieredImageNet. However, computing the task embedding for every class of dataset as large as the vetaaDatasett is nearly impossible due to the time and memory constraints. Hence, in our experiments on the Meta-Dataset, we do not run the model using the s-DPP and d-DPP Sampler and only report the findings from the remaining samplers.
OHTM Sampler offers no significant performance boost The OHTM Sampler is quite sophisti- cated since it regularly samples diverse tasks, as well as selects the most challenging tasks to improve the model. It is needless to say; the model requires more computational power and time than the Uniform Sampler. However, the OHTM Sampler offers no significant boost in performance when compared to the Uniform Sampler in the case of Omniglot, minilmageNet, and tieredImageNet. However, for Meta-Dataset, we notice that the OHTM Sampler sometimes leads to improved per- formance. This finding is quite puzzling since the Sampler works similarly across all datasets and does not address the inherent class variability generated from training on diverse datasets such as the Meta-Dataset. The behavior of the OHTM Sampler on Meta-Datasets warrants further research.
Comparison between NDTB, NDB, and Uniform Sampler From our experiments, we also no- tice that the No Diversity Tasks per Batch Sampler and No Diversity Batch Sampler are pretty similar to the Uniform Sampler in terms of performance. This would suggest that the model trained only a data fragment can perform similarly to that trained on the Uniform Sampler.
Abnormal run of matching networks d-DPP (20-way 1-shot) In our run on the matching net- works with the d-DPP Sampler under the 20-way 1-shot setting, we ran across a peculiar error. The prototypes generated by the matching networks were sometimes not fit to be used by the d- DPP Sampler to sample 20 unique classes. The reason is that the rank of the matrix generated using the embeddings was lower than the required number of classes per task (i.c., 20). To create a workaround for this sole experiment, we chose to sample 5 diverse classes time and append them
8

---

Under review as a conference paper at ICLR 2022
to create the task. We hypothesize that the prototypes created by matching networks are unsuitable for downstream tasks and warrant further research regarding this behavior.
Poor performance of MAML on Meta-Dataset In our experiments on Meta-Dataset, we notice that MAML performs significantly worse than other models. Some of the reasons for this disparity of performance when compared to Triantafillou et al. (2019) have been discussed in detail in Appendix A. Furthermore, we hypothesize that the poor performance of MAML can be attributed to the way its adaptation process works: MAML learns within episode weights of the model before adapting to set of new tasks via meta-update outer loop update. This new task is again sampled from the pool of tasks and is a different set of data altogether. In most cases, where the model trained on only one dataset, this intuition would make sense and lead to nigh-performing models. However, in the case of Metta-Dttaee,, where the new set of tasks might be of entirely different datasets or domains, this approach tends to do more harm to the model rather than aid. In the work of Triantafillou et al. (2019), the authors adapted MAML such that it focuses on learning the within-episode initialization 0 of the embedding network so that it can be rapidly adapted for a new task. This allows the model to learn from a variable number of ways and shots per episode.
Peculiar behavior with MetaOptNet model Compared all other models, MetaOptNet seems to be immune to the effects of task diversity to a great extent. The convergence of the model seems to follow a general pattern and achieve similar performance across task distributions except for the Single Batch Uniform Sampler and No Diversity Task sampler. Furthermore, we do not observe the expected pattern of d-DPP Sampler, where the performance drops upon mining diverse tasks. We present the convergence graph of the MetaOptNet model on Omniglot 5-way 1-shot run in Figure 11 in the Appendix with an added smoothing factor of 1.
General Trend From our experiments, we notice that there are generally two classes of samplers: High Performing Samplers and Low Performing Samplers. The High Performing Samplers include No Diversity Batch, No Diversity Tasks per Batch, Uniform, OHTM, and s-DPP Sampler. The Low Performing Samplers include No Diversity Task, Single Batch Uniform, and d-DPP Sampler. This trend is shared across all datasets and models. There are some perturbations in ranking within the two classes, but the High Performing Samplers tend to perform better than the Low Performing Samplers.
CONCLUSION
In this paper, we have studied the effect of task diversity in meta-learning. We have empirically shown that task diversity does not lead to any significant boost in performance in meta-learning. In- stead, limiting task diversity and repeating the same tasks over the training phase allows us to obtain similar performances to the Uniform Sampler without any significant adverse effects. Furthermore, We also show that sophisticated samplers such as OHEM or DPP samplers do not offer any signifi- cant boost in performance. In contradiction, Sampler hampers the performance of the meta-learning model. Our experiments using the NDTB and NDB empirically show that a model trained on even tiny data fragment can perform similarly to a model trained using Uniform Sampler. This is a crucial finding since this questions the need to increase the support set pool to improve the models' performance. We believe that the experi- ments we performed lay the roadwork to further research for the effect of task diversity domain in meta-learning and lay some groundwork and rules of thumb for task sampling for meta-learning.
notice that increasing task diversity using the d-DPP
REPRODUCABILITY STATEMENT
In this paper, work with four different datasets Omniglot, minilageNNet, fieredlmageNet and Meta-Dataset. Additional details about setting up these datasets is available in Appendix A. Fur- thermore, experiment with six different models MAML., Reptile, Protonet, Matching Networks, MetaOptNet, and CNAPs. All these models were run after reproducing from their open-source codes. Additional details about setting up these models are available in Appendix A.2.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Erdem BıyIk, Kenneth Wang, Nima Anari, and Dorsa Sadigh. Batch active learning using determi- nantal point processes. arXiv preprint arXiv:1906.07977 07975, 2019.
Mírcea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3606-3613, 2014.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pp. 1126-1135. PMLR, 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. arXiv preprint 1806. 02817, 2018.
Marta Garnelo, Dan Rosenbaum, Christopher Maddison. Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International Conference on Machine Learning, pp. 1704-1713. PMLR, 2018.
Sebastian Houben, Johannes Stallkamp, Jan Salmen, Marc Schlipsing, and Christian lgel. Detection of traffic signs real-world images: The german traffic sign detection benchmark. In The 2013 international joint conference neural networks (IJCNN) pp. 1-8. Ieee, 2013.
Kyle Hsu, Sergey Levine, and Chelsea Finn. Unsupervised learning via meta-learning. arXiv preprint arXiv:1810.02334 2018
Jonas Jongejan, Henry Rowley, Takashi Kawashima, Jongmin Kim, and Nick Fox-Gieg. The quick, draw!-ai experiment. Mount View, CA, accessed Feb, 17(0018):4 2016.
Gregory Koch, Richard Zemel, Ruslan Salakhutdinov, et al. Siamese neural networks for one-shot image recognition. In /CML deep learning workshop, volume 2. Lille, 2015.
Alexandre Kuhn, Ad Aertsen, and Stefan Rotter. Higher-order statistics of input ensembles and the response of simple model neurons. Neural computation, 15(1):67-101, 2003.
Alex Kulesza and Ben Taskar. Determinantal point processes for machine learning, arXiv preprint 1rXiv:1207.6083, 2012.
Alexandre Lacoste, Boris Oreshkin, Wonchang Chung, Thomas Boquet, Negar Rostamzadeh, and David Krueger. Uncertainty in multitask transfer learning. arXiv preprint 1rXiv:8806.07528, 2018.
Brenden Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua Tenenbaum. One shot learning of simple visual concepts. Proceedings of the annual meeting of the cognitive science society, volume 33, 2011.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10657-10665, 2019.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Chenghao Liu, Zhihao Wang, Doyen Sahoo, Yuan Fang, Kun Zhang, and Steven CH Hoi. Adaptive task sampling for meta-learning, In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVIII 16, pp. 752-769, Springer, 2020a.
Jialin Liu, Fei Chao, and Chih-Min Lin. Task augmentation by rotating for meta-learning. arXiv preprint arXivv2003.0088 00804, 2020b.
Odile Macchi. The coincidence approach to stochastic point processes. Advances in Applied Prob- ability, 7(1):33-222, 1975.
10

---

Under review as a conference paper at ICLR 2022
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- tine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013.
Tsendsuren Munkhdalai and Hong Yu. Meta networks. In International Conference Machine Learning, pp. 2554-2563. PMLR, 2017.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-leamiing algorithms. arXiv preprint 1.Xiv:1803302990 2018.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pp. 722-729, IEEE, 2008
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. 2016.
Sachin Ravi and Hugo Larochelle. Meta-learning for batch mode active learning. 2018.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard $ Zemel. Meta-learning for semi-supervised few-shot classifica- tion. arXiv preprint arXiv:1803.0060 2018.
James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard E Turner. Fast and flexible multi-task classification using conditional neural adaptive processes. Advances in Neural Information Processing Systems, 32:7959-797777 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-55)) 2015.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with memory- -augeeedd neural networks. In International conference on machine learn- ing. pp. 1842-1850. PMLR, 2016.
Brigit Schroeder and Yin Cui. Fgvex fungi classification challenge 2018. com//isippedidffycxxsuy comp, 9, 2018.
github.
Amrith Setlur, Oscar Li, and Virginia Smith. support set diversity necessary for meta-learning? arXiv preprint 2rXiv:2011.14044 2020.
Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. Training region-basee based object detectors with online hard example mining. In Proceedings the IEEE conference computer vision and pattern recognition, pp. 761-769, 2016.
the
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. arXiv preprint 1003v:1703..51111 2017.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales.
Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pp. 1199-1208, 2018
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, et al. Meta-dataset: dataset of datasets for learning to learn from few examples. arXiv preprint 1903:0990000099 2019.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29:3630-366 3638, 2016.
Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
Mingzhang Yin, George Tucker, Mingyuan Zhou, Sergey Levine, and Chelsea Finn. Meta-learning without memorization. arXiv preprint 19/2v:192.003820 2019.

---

Under review as a conference paper at ICLR 2022
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 7343-7353, 2018.
Cheng Zhang, Cengiz Öztireli, Stephan Mandt, and Giampiero Salvi. Active mini-batch sampling using repulsive point processes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 5741-5748, 2019.
12

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 DATASETS
Omniglot, minilmageNet and rieredlmageNet were extracted using pytorch-meta, The Meta-Dataset was downloaded using the setup information from the official repository: https: //ghthbb. com/google research/mee wttpceaaatmaaaaaaattttataeaa
Omniglot Omniglot is a benchmark dataset proposed by Lake et al. (2011) for few-shot image classification tasks. Omniglot dataset consists of 20 instances and 1623 characters from 50 different alphabets. We experiment with both 5-way 1-shot and 2-way 1-shot in this work.
minilmageNet minilmageNet is another benchmark dataset proposed by Ravi & Larochelle (2016) for few-shot image classification tasks. The minilmageNet dataset involves 64 training classes, 12 validation classes, and 24 test classes. We run under the setting 5-way 1-shot for this experiment.
lieredl tieredImageNet is notorious as a difficult benchmark dataset proposed by Ren al. (2018). The tieredImageNet dataset is larger subset of ILSVRC-12 with 608 classes (779,165 images) grouped into 34 higher-level nodes in the ImageNet human-curated hierarchy. This set of nodes is partitioned into 20, 6, and 8 disjoint sets of training, validation, and testing nodes, and the corresponding classes form the respective meta-sets. As argued in Ren et al. (2018), this split near the root of the ImageNet hierarchy results in more challenging yet realistic regime with test classes that are less similar to training classes. We run under the setting 5-way 1 1-shot for this experiment.
Meta-Dataset Meta-Dataset notorious as a difficult benchmark dataset proposed by Triantafil- lou et al. (2019). The Meta-Dataset is much larger than any previous benchmark and is comprised of multiple existing datasets. This invites research into how diverse data sources can be exploited by meta-learner and allows us evaluate more challenging generalization problem to new datasets altogether. Specifically, Meta-Dataset leverages data from the following 10 datasets: ILSVRC-2012 (Russakovsky al. (2015)). Omniglot (Lake et al. (2011)), Aireraft (Maji al. (2013)), CUB-200- 2011 (Wah et al. (2011)), Describable Textures (Cimpoi et al. (2014)), Quick Draw (Jongejan et al. (2016)), Fungi (Schroeder Signs (Houben et al. (2013)) and MSCOCO (Lin et al. (2014)). There exist few classes with fewer image samples than 16. This becomes an issue since we need 1 for training and 15 for testing. We repeat some of the support images for these classes to make up for the lack of examples. Since the number of such classes is minimal, we justify this use as this solution cannot significantly increase the model's performance. Furthermore, unlike previous experiments variable number of ways and shots during training, we train with a fixed number of ways and shots. Furthermore, unlike the works in Triantafillou al. (2019), we do not perform any pre-training to help aid the model. We helieve these are the main attributions for the disparity of our performance. However, since we are focused on the relative performance of the samplers for given model, this discrepancy would not affect our study of task diversity in any manner. We again run under the setting 5 5-way 1-shot for this experiment.
Cui (2018)), VGG Flower (Nilsback & Zisserman (2008)) Traffic
this dataset which use
A.2 MODELS
This section describes some of the models we used for our experiments and the hyperparameters used for their training.
A.2.1 PROTOTYPICAL NETWORKS
Prototypical Networks proposed by Snell et al. (2017) constructs a prototype for each class and then classifies each query example as the class whose prototype 'nearest" to it under Euclidean distance. More concretely, the probability that query example x= belongs to class k is defined as:
exp(-11g(x)) -c*ll3)
ply" = k|x", Exe(U....)) exp(- 11g(x*) cx. 1x.1))
(4)
13

---

Under review as a conference paper at ICLR 2022
Where C₁ is the "prototype' for class k: the average embeddings of class k's support examples.
HYPERPARAMETERS In our experiments on Omniglot and minilmageNet, and magred under 5-way, I-shot setting, learning rate of 0.001. We use an Adam optimizer to make gradient steps and a StepLR scheduler with step size 0.4 and gamma 0.5. The same hyperparameters are used for training our model on Omniglot under 20-way 1-shot setting.
the model for 100 epochs with batch size of 32 and meta-
We use the same parameters the minilmageNet to train our model on the Meta- Dataset under 5-way 1-shot setting. However, we run with a batch size of 16 rather than 32 to accommodate the extensive training period and memory constraints.
A.2.2 MATCHING NETWORKS
Matching Networks proposed by Vinyals et al. (2016) labels each query example as a cosine setance-weighted linear combination of the support labels:
[s
ply" = kk = Σ ax , ( 1=1
(5)
where 4 the indicator function and a(x" xi) is the cosine similarity between g(x") and g(at),
softmax normalized over all support examples I3, where 1 S i (S).
We had trouble reproducing the results from matching networks using cosine distance since the convergence seemed to be slow and the final performance dependent the random initializa- tion. This is similar to what is observed by other repositories such as https: gi t thub, com/ oscarknagg/ Eettthot. shot. Since we are focused on the relative performance of the samplers for given model, this discrepancy would not affect our study of task diversity in any manner.
HYPERPARAMETERS In our experiments on Omniglot, minilmageNet,, and rieredImageNet under a 5-way, 1 I-shot setting, we run the model for 100 epochs with a batch size of 32 and an Adam opti- mizer with a meta-learning rate of 0.001 and weight decay of 0.0001. The same hyperparameters are used for training our model on Omniglot under a 20-way 1-shot setting.
We use the same parameters as the minilmageNet to train our model on the Meta-Datasel under 5-way 1-shot setting. However, we run with batch size of 16 rather than 32 to accommodate the extensive training period and memory constraints.
A.2.3 MAML
MAML proposed by Finn al. (2017) uses linear layer parametrized by W and b on top of the embedding function g(:: f) and classifies query example as:
p(y*|x" S) softmaz(b' W=((=";")
where the output layer parameters W and b' and the embedding function parameters 0' are obtained by performing a small number of within-episode training steps on the support set S, starting from initial parameter values (b, W,
HYPERPARAMETERS In our experiments on Omniglot, minilmageNet, and rieredImageNet under a 5-way, 1-shot setting, we run the model for 150 epochs with a batch size of 32, with the Adam optimizer with a meta-learning rate of 0.001, number of inner adaptations as and step size 0.4. For our experiments on Omniglot under the 20-way 1-shot setting, we set the step size of 0.1 and the number of inner adaptations 5, batch size of 16, and kept all other hyperparameters constant.
We use the same parameters as the minilmageNet to train our model on the Deta-Dataset under 5-way I-shot setting. However, we ran with a batch size of 16 rather than 32 and only trained for 100 epochs to accommodate the extensive training period and memory constraints.
A.2.4 REPTILE
Like MAML, Reptile proposed by Nichol et al. (2018) learns an initialization for the parameters of neural network model, such that when optimize these parameters at test time, learning is fast
14

---

Under review as a conference paper at ICLR 2022
Le., the model generalizes from small number of test tasks. Reptile converges towards a solution that is close (in Euclidean distance) to each task 7's manifold of optimal solutions. Let & denote the network initialization, and W è + Aø denote the network weights after performing some sort of update. Let W denote the set of optimal network weights for task т. We want to find & such that the distance D(Ø, W7) is small for all tasks:
1 min Er1, D(
(7)
The official repository seems to train the model with 5-way 15-shot and test the model on a 5-way I-shot. However, we do not consider this to be an accurate study for the effect task diversity. In work, train and test the model in 5-way 1-shot setting to ensure fair and accurate comparison with other models. We believe this to be the source of discrepancy in our performance scores. Since focused the relative performance of the samplers for given model, this discrepancy would not affect our study of task diversity in any manner.
HYPERPARAMETERS In our experiments on Omniglot and miniImageNeet under a 5-way, 1-shot setting, we run the model for 150 epochs with a batch size of 32, learning rate of 0.01, meta- learning rate of 0.001, and number of inner adaptations as We use the SGD optimizer of inner steps and Adam optimizer for the outer steps. For our experiments on tieredImaaeNeet we change the number inner adaptations to 10, keeping all other hyperparameters constant. For our experiments Omniglot under the 20-way 1-shot setting, set the meta-learning rate to 0.0005 and the number of inner adaptations to 10 and kept all other hyperparameters constant, Furthermore, only run the model for 50 epochs due to the very high training time.
We use the same parameters as the tieredImageNet to train our model on the Meta-Dataset under a 5-way 1-shot setting. However, extensive training period and memory constraints.
with batch size of 16 rather than 32 to accommodate the
A.2.5 CNAPs
Conditional Neural Adaptive Processes proposed by Requeima et (2019) is able to efficiently solve new multi-class classification problems after an initial training phase. The proposed approach, based on Conditional Neural Processes (CNPs) mentioned in Garnelo et al. (2018), adapts a small number of task-specific parameters for each new task encountered at test time. These parameters are conditioned on a set of training examples for the new task. They do not require any additional tuning to adapt both the final classification layer and feature extraction process. This allows the model to handle various input distributions. The CNPs construct predictive distributions given xx as:
ply"|=" "vu'=",,,,'") D7) p(y"I= 0, 1 (o(DD))
where f are global classifer parameters shared across tasks, T are local task-specific parameters, produced by a function (.) that acts of D (.) has another set of global parameters & called adaptation network parameters. 0 and are the learnable parameters in the model.
HYPERPARAMETERS In all our experiments with CNAPs, we run the model for ten epochs with a batch size of 16 and meta-learning rate of 0.01.
A.2.6 METAOPTNET
MetaOptNet proposed by Lee et al. (2019) proposes linear classifier as the base learner for meta- learning based approach for few-shot learning. The approach uses a linear support vector machine (SVM) to learn a classifier given a set of labeled training examples. The generalization error is computed on a novel set of examples from the same task. The objective is to learn an embedding model & that minimizes generalization (or test) error across tasks given base learner A. Formally, the learning objective is:
min [ metta (Destt 00, 0),), where A(D ; 6)
The choice of base learner A has a significant impact on the above equation. The base learner that computes 0 A(Diest; () has be efficient since the expectation has to be computed over
15

---

Under review as a conference paper at ICLR 2022
distribution of tasks. This work considers base learners based on multi-class linear classifiers such as SVM, where the base learner's 's object is convex. Thus, the base learner can be simplified as:
0 A(D Φ) arg min min Σx |w + CEn S.i subject to:
(we)
(10)
Wy. (o( (o 21 8yn Snz Vn, k
where Dirnan = ((In. C' is the regularization parameter and 6... is the Kronecker delta func- tion.
Furthermore, the official repository seems to train the model with a 5-way 15-shot and test the model on 5-way I-shot. However, we do not consider this be accurate study for the effect of task diversity. In our work, we train and test the model in a 5-way 1-shot setting to ensure fair and accurate comparison with other models. We believe this to the source of discrepancy in our performance scores. Since we are focused on the relative performance of the samplers for given model, this discrepancy would not affect our study of task diversity in any manner.
HYPERPARAMETERS In our experiments on Omniglot, manel and liered under 5-way, 1-shot setting, we run the model for 60 epochs with a batch size of 32 and a meta-learning rate of 0.01. We use an SGD optimizer with momentum of 0.9 and weight decay of 0.0001 make gradient steps. We also use LambdaLR scheduler to train our model. The same hyperparameters are used for training our model on Omniglot under 20- 20-way 1-shot setting.
We use the same parameters as the minilmageNet to train our model on the 5-way 1-shot setting. However, we run with a batch size of 16 rather than 32 accommodate the extensive training period and memory constraints.
under
A.3 ADDITIONAL RESULTS
A.3.1 RESULTS ON REGRESSION
We further experiment on the few-shot regression regime to maintain our findings classification. For this problem, we trained on three few-shot regression datasets: (i) Sinusoid (Finn et al. (2017)), (i)SSnnusiid & Line (Finn et al. (2018)), and (iii) Harmonic (Lacoste et al. (2018). Table 1 presents our results using the mean square error metric on the 5-shot and 10-shot setting for all three datasets. Figure and Figure 8 presents For both MAML and Reptile, we use the same network, with two hidden layers, each of size 40.
results the Sinusoid dataset specifically for easier comparison.
HYPERPARAMETENS For both MAML and Reptile, we train the model for 150 epochs, and use the same hyperparameters Omniglot. Only in the case of MAML, use step size of 0.001 for Sinusoid and Sinusoid & Line dataset, instead of the traditional 0.4.
Results We again observe that samplers that limit diversity, such as NDB, NDT/B samplers achieve performances similar to the uniform sampler with no adverse effects. Astonishingly, we also notice that the NDT sampler seems to outperform Furthermore, similar our previous findings, samplers that increase diversity, such the OHTM sampler, do not perform very well in this domain either, as shown in Figure and Figure 8. We can- not train using Drp-oriented samplers for regression since the data is continuous, and computing features or representations for the entire range would be theoretically impossible.
other samplers in the regression domain.
A.3.2 RESULTS ON CLASSIFICATION
In this section, we present the results with higher precision from our earlier experiments in Table and Table 3. Subsequently, mentioned earlier in Figure 10 and Figure 11. We also present the results of our models on the Meta-Dataset in Table 4.
also plot convergence curves aid better visualizations of findings
Statistical Results
We compare the performance of different models to the Uniform Sampler. All samplers are poorer than the Uniform Sampler and are statistically significant with confidence interval of % We use the symbol to represent the instances where the results not statistically
16

---

Under review as a conference paper at ICLR 2022
S-shot
10shoo
Datuset
Sampler
MAML
Reprle
MAML
Repale
Uniform Sampler
No Diversity Task Sumpler
0.79 = 0.00
0,35 + 0.35
0.54 0.04
0.12 = 0.01
Sinusoid
Diversity Batch Sampler
0.77 0.08 5
0.37
0.04
0.56 0.04
0.13 4 0.01
No Diversity Tasks Batch Sampler
0.92 0.00
0.37
13324000
0.12 0:01
Single Batch Uniform Sampler
1.94 0.11
0.71
= 0.06
1.39 = 0.07
0.25 = 0.02
OHTM Sampler
+ 0.08
0.79
0003
Uniform Sampler
4,08 a: 0.32
3,74 3.07
2,78 0.22
0.66 0.13
No Diversity Task Sampler
3.91 1: 0.32
2,80
0.30
2.70
0.70 1 0.13
Sinusoid and Line
No Diversity Batch Sampler
4.07 4: 0.34
2.18
0.27
0.22 2.66
0.69 4: 0.11
Diversily Tasks per Batch Sampler
2.82
2.73
Single Batch Uniform Sampler
6,62 + 0.57 V
5.52
4.45
8.48 + 0.94
1.19 - 0:15
OHTM Sampler
4,43 1.33
3.67
0.51
3,36 0.28
1.19 0315
Unitorm Sampler
1,07 = 0.07
1,20 +:00:000 0.08
1.07 2001200 0.07
1.05 0.07
Diversity Sampler
0.07
1.07
0.07
Harmonic
No Diversity Batch Sampler
1.11 + 0.07
1,18 = 0.09
1,03 0.06
1.88 0.07
Diversity Tasks per Batch Sampler
1.14 0.07
1.18
1.05 0.07
0.07
Single Batch Uniform Sampler
1,19 0.08
1,10
1,06 0.07
1.8 0.07
OHTM Sampler
1.13 = 0.08
1.18 = 0.08
1.42 0.15
1.10 0.07
Table 1: Performance metric of our models on different task samplers the few-shot regression setting.
64-
-3-
-14
(a) Uniform
(b) No Diversity Task
(c) No Diverrsity Batch
18-
2.5
(d) Diversity Tasks/Batch
(e) Single
(I) OHTM
Figure 7: Few-shot adaptation for the simple regression task on Sinusoid 5-shot dataset. The ground truth is denoted by "_". The predicted output of MAML and Reptile are denoted by "__." and respectively. The training points used for computing gradients denoted by
significant and similar to the performance achieved by uniform Sampler. We only observe few cases where sampler performs significantly better than the Uniform Sampler, which we represent using the symbol E. To assess statistical significance, we used paired-difference t-test, with a p- value p 0.05.
Comparisons with SBU Sampler Previously, we compared the performance of NDTB and SBU samplers. However, due to the possibility of unfair comparisons, limited by the number of itera- tions of the SBU sampler, we extend the SBU Sampler to propose two new samplers: (1) SBU- unbounded sampler This sampler is similar to the SBU sampler but is run for more iterations. However, this sampler allows the model to be trained on more classes than our traditional SBU sam- pler. Since this remains unfair comparison, propose more appropriate sampler. (2) SBU-bounded sampler This sampler is again similar to the SBU sampler but ran for more iterations with bounded pool of tasks, similar the NDTB
this a higher bound/ideal performance and
17

---

Under review as a conference paper at ICLR 2022
64-
-1
(a) Uniform
(b) No Diversity Task
(c) No Diverrsity Batch
14
-20
-0
(d) No Diversity Tasks/Batch
(c) Single
(f) OHTM
Figure 8:
Few-shot adaptation for the simple regression task on Sinusoid 10-shot dataset. The
ground truth denoted by b_" The predicted output of MAML and Reptile are denoted by "__." and " respectively. The training points used for computing gradients is denoted by
sampler or the SBU sampler. Our results from these experiments are presented in Figure Table shows the same results with higher precision.
We notice that training for more iterations using samplers such as SBU-unbounded and SBU- bounded leads boost in performance. However, also observe that the NDTB sampler mains to perform better in most cases. With this, we maintain our previous findings. Furthermore, also notice peculiar behavior where the SBU-bounded sampler performs better than the SBU- unbounded sampler in most cases. This finding is interesting because the SBU-unbounded sampler has access to data, and the model being trained should lead to better representations and per- formance. However, we notice that repeating the tasks and fixing the number of classes achieves the same performance with little adverse effects.
Dataset
Sampler
MAM/
Repatle 94.54
Protenet
Matching Networks
MeteQpttNrr
CNAPY
Uniferm Sampler
Hiussyy Sampler
1130
Diverity Sumpler
0.05
Omniglat
Single
Simpler
Sample:
07.74
93.48
96.12
97.61
94.79
92.29
95.83
95.00
02:00
Sumplea
Diverity Sumpler
Samporr
Mimilmaggne
Batch
Sumplet
Sampler
4000
39.00
URT'
0044
4308
47.1
00192000
4240 2005
3DDPP
3561 1030
35.01 055
34.77 00000
43.15 20st
3487 1103
60.00 1070
Sumpler
Sampler
12117
Amagenet
3077
3800
4500
05.0 1007
5006
30.10
w-19p Sampler
$1.15
50.00
49422
44.8
49.6 0.06
61.88
ZDPP Sumpler
34.00 032
34,78
0.5
40377 0006
33.9
03.01
Table 2: Performance metric of our models on different task samplers in the 5-way 1-shot setting.
18

---

Under review as a conference paper at ICLR 2022
Dalisel
Sampler
MAML
Reptlee
Pritonet
Moichng Networks
MeinOpeNe
CNAPS
Unifoem Sampler
91.25 0.23
W000 s: 0.23
9172 8:30
74.02 0.36
9219 = 1.22
No Divensty Tisk Sumpler
83.39 0:28
99.49 0:33
85.84 i 6.27
26.50 6.33
6
73.82 + 0.M
No Diversty Butch Sampler
39.07 2:22
03:25
W8 = 0,20
71.77
19.36 = 022
Omsnigle
No Diverdly Tgais Batch Sampler
Singk Bates USffeem Stmpler
90.77 0.23
82.45 0.031
1..510..11
90399
0:21
1335 = 6,19
9200 6:20
54.01 6.40
61.31:0411
59.39 0.34
7081 0.35
77.54 = 0.37
5919 = 0..
OHTM Sampler
91.25 0:221
89.92
0:22
91.33 a 0.20
72.20
0.38
91.56
89.51 à 0.25
DDF
2.3
90190
72.00
V1.47
90195
DVY Sampler
3725 2.35
108F 20335
VOLTI
Table 3: Performance metric of our models on different task samplers in the 20-way 1-shot setting.
Datant
Sampler Sampler
MAME
Reptile
Proionet
Macciey Natwarks
MeteDgNNer
CNAPY
29.02
23.85
Meta- Datasee (ILSVRC)
No Mitrrrrrryy Toak per lins Kamppir
No DNveRRY Beac Sanqier
23244 136
1091
1700
24.1
25.25
1541 1131
27 111
2014
3430 11N
Sirph mimmm Nemple
1TAR 100
27.99
2800 1:115
16 1.4
Surpler
933
Unifirn Saugler
3000 = 009
36.07 $.300
1270 2.
77.61 1664
$7.00 =n 1033
Ditesity Sumpler
Meta- Dutaset (Ounlghht))
No
16.9 1044
+230
11.10
Hirgh Ia::3 Tmison Nangle
99.1
15.19 6m
0131
11.95 100
11.01 17076
15:15
CHTTM Somple
M1 5700
74.95 1095
78.46
0116
Tiilfem S Sumpler
1131
40A1
28:31
52.12 0.11
39.64
No Di lisk Sumple
2.35
20.70 2:30
23.55 10.00
31.27
Meta
No Divasryy Bach Natigter
Sampler
25.17
2709
0:036
Staple
Nample
20.9
QRTTM Simper
2000 1:44
1100 2044
Tlillemn Sumpler
4029
11.16
17.03
Nn di Tirk Semple
2700 1.3%
15:25 2BW
Mets-Datostt (Birda
No Wiventy Baych Sangler
25.0 2033
31.08
Direraty Tikk Back Sarppet
Steple
CITTM
45.15
1033
30.19
31.35 100
Unifeem
33,33:
Diersiyy list
2035
1536
1511
Mots-istawrt (Teecture)
Divenity lioks per Batch Samplee
No Disevity Buch Sampler
23 935
935
24.34
20:00::00 0.66
26.55 1635
16 000
25.55 133
13.44 a 6.3
54.95 0:35
23.99 0.34
31.77
17.46 - 0.0
Stsple Hiles Libonn Sample
26.77
25332
12.30
Simple
Unileen Sampler
35.11
51.90
56.17
38.54
Mette-Ditust (Quick Draw)
No Divenity Tisks Bash Sumpler
No Biverry
35.31 961
55 4
50.77 0-65
47.41 049
3650=
18:00 2.00
41.34 6 0.53
Siegle Baich Unifom Semple
2:.00:00
45.52 0:0
48.00 1005
51.06 9.07
20.20 14.55
CRTTM Surpar
0930
3037
Usiforn Sangler
13.04 10.6
20.43 2.42
34.45 2.3
24.55 2.33
40.37 =
Hiarsdyy laxt Sample
27.00
1637 1.75
Meta-Dattaet (Funai)
Diversty Tials Bual Sumpler
10.35
3120
2532
Bigglt Bat3 Unidon Sangle
30112
S131
30100 1:44
25.00 IS
CRITM Sampler
31.15
35,3 1-46
36.40
1.56
41.05
Unifien Sampler
10.8 601
65
50.98 0.055
58.64 607
58.29 6.007
65.12 M88
Ni Nir Tial Sumple
15:00 1100
27.90
21056
MottaeD (VGG Bloweri
No Marniry Rich Simple
Kampier
40.35 10.9
1110
41.95
Unioon
$4.35
ORTM Sumper
50555 105
3607 2000
5535 688
6251 075
Uniforn Sampler
45,35
30.9
39,39
2035
52.15
Nu Divesity Taak Sample
25 945
32465
30.44 1.00
33.81 2.51
35.77 2057
40.24 155
Mets Datiset (Traftic Signs)
Disersyy Tink per lava Sampler
No Hitemny dedh sanrler
11
10700
15.0
Sirgle
Sample
Sumplr
41.56
Sampler
36.97
2031
31:52000
33.0
Dy
11446
Meta- Dathert (MSCOCC)
No lifereryy libb por line Namper
No Diseniy Bacc Sumpler
29.55 1100
2508 0.47
16.99 1035
11.85 6X1
1-11 1001
20.3 138
N0W 1951
1644 100
20.55 16
31.97 1131
1489 1351
1276 1191
birgle Bata Unzome Sangle
11,07 2..3
1E47 1153
20000
34.5 637
27.15
OHTM Sumpler
0:30
37.33
31.00
33.54
33.87
Table 4: Additional Performance metrics of our models on different task samplers in the 5 5-way
--shot setting.
Dutuse
MANL
Acynic
Protene
Macking
MenrtpoNer
CINLSS
No Divamiiyy
Sumpler
07
9435
97.18 0.2
Omniglot
SIBC
Unitorn
VKt6
9334
85.00 633
98,95
97.18
95.76
3011
75 #
SBUl
V7
97.5 933
V0%
V7.8
Divenity lasks per Racch Sumple
4K17
41,49
47.73 1001
4.54 053
50.00
Minilasagenet
inboodded
Unitom Simpler
11.65
22
3500
11.35
39.00
3121
No Diverity lasks par Runch Sumple
40.2 649
50.77
6.67
46:90 0.67
45.15 om6
43005 0.6
62.11 6335
Ticred-Inugceet
Single Batch Laitem Sampler
SIB uncorded
11774
2 006
41.8
3082
000
4.3
500000000
38.24
2000
4110 2004
39.0 mMM
$1.8 4.
36.21
$000
SBU Somnded
403 45.57 600
51.11 100
Table 5: Performance metric of our models on SBU oriented task samplers in the 5-way 1-shot setting.
19

---

Under review as a conference paper at ICLR 2022
Omniglltt b-mxy 1-shot
100
Aunrary
70
MAML
Replile
Protomet
Maschinn Nel.
MetaOO
CNAP:
misiImageNet 5 S-way 1-shoo
60
Accurayy
40
20
MAML
Reptile
Protonct
Matching Net. tieredInaggeNet S-way L-shot
MetaOptNet
CNAPs
60
60
Reptile
Matching
MetaOptNet
No Divensit Tacks, Batell
Single
Single-anbotmueed
bouneee
Figure 9: Average accuracy on Omniglot 5-way 1-shot, miniImageNet 5-way 1-shot, & tieredlmageNet 5-way 1-shot with a 95% confidence interval. We denote all samplers that are worse than the Uniform Sampler and are statistically significant (with p-value p 0.05) with v, and those that are significantly better than the Uniform Sampler with A.
Matching Notworks Omnigiot S-way
0.8
0.4
Stop
6-00P
Figure 10: Convergence curve of Matching Networks model on Omniglot 5-way 1-shot.
20

---

Under review as a conference paper at ICLR 2022
MetaOptNot:: Omniglot 5-way 1-shot
0.96
0.9
0.85
0.75
Step
10
20
30
Divenily
Diverily
Stagle
Diversity
Figure 11: Convergence curve of MetaOptNet model on Omniglot 5-way 1-shot.
21


Reviews:
[{"review_id": "knu4utjqBr3", "reviewer": "Reviewer_UoR2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Significance: The paper follows an existing line of work that empirically shows task diversity (in the training phase) does not help with meta-learning. This is an important step towards a better understanding of meta-learning. \n\nNovelty: The key finding that task diversity may not be beneficial for meta-learning has been proposed and studied by Setlur et al. (2021). The paper extends the same findings to (a) different meta-learning algorithms, (b) different task samplers, and (c) different datasets. For this reason, I find the novelty of the paper limited.\n\nThere are some technical ambiguities in the paper, as discussed below.\n1. The explanation of the empirical results is not insightful and convincing enough. In section 5, the authors attribute the poor (or good) performance of various diversity samplers to noise (or lack of thereof) introduced when training on multiple sub-datasets. This reason is not particularly insightful or useful to other researchers and needs more detail. For example:\n(a) what noise the authors are referring to? There is no noise introduced in the problem formulation in section 3.1.\n(b) how does diversity in tasks introduce noise?\n(c) how does this noise affect the performance of meta-learning?\n(d) can we perform ablation studies to confirm that the samplers’ poor performance is indeed caused by noise?\n2. Some choice of task samplers in section 3.3 lacks motivation. For some task samplers, e.g., Online Hard Task Mining and Static DPP, the authors provide citations to various works that propose the use of these samplers. However, for the remaining samplers, the paper does not discuss why these samplers are good candidates to compare with the uniform sampler.", "summary_of_the_paper": "The paper studies how the diversity of tasks in the training phase affects the performance of meta-learning algorithms. The paper finds negative evidence, which is consistent with Setlur et al. (2021). Compared with the existing work, the paper performs more extensive experiments with different meta-learning algorithms, different task samplers, and different datasets.", "summary_of_the_review": "The paper provides an extensive set of empirical evidence to demonstrate that task diversity (during training) is not beneficial for meta-learning. The insights and conclusions drawn from these empirical experiments, however, are not so convincing and helpful, i.e., it does not tell other researchers how to rectify the problem or how to design better meta-learning algorithms.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DsgbUYe26lv", "reviewer": "Reviewer_wVFn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In this paper, the authors investigate the effect of task diversity in the training process of meta-learning. The findings indicate that increasing task diversity during the meta-training process does not boost performance. They evaluate the performance on four few-shot image classification datasets. \n\nThough the authors investigate several samplers, they only conduct the experiments on N-way K-shot few-shot image classification tasks. The findings do not surprise me since the meta-training process of N-way K-shot image classification essentially learns a better representation using all meta-training data samples (see the discussions in Baseline++, Meta-Baseline). Thus, using a less diverse sampler does not change the number of training samples, and thus it may not hurt the performance. \n\nTo make the findings more convincing and exciting, I suggest that the authors conduct the experiments on much more diverse tasks, e.g., regression tasks and noisy data. It would also be interesting to conduct qualitative analysis on toy tasks to understand why task diversity does not benefit the performance.\n\n[1] Dhillon, Guneet S., Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. \"A baseline for few-shot image classification.\" arXiv preprint arXiv:1909.02729 (2019).\n\n[2] Chen, Yinbo, Zhuang Liu, Huijuan Xu, Trevor Darrell, and Xiaolong Wang. \"Meta-Baseline: Exploring Simple Meta-Learning for Few-Shot Learning.\" In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9062-9071. 2021.", "summary_of_the_paper": "In this paper, the authors investigate the effect of task diversity in the training process of meta-learning. The findings indicate that increasing task diversity during the meta-training process does not boost performance. They evaluate the performance on four few-shot image classification datasets. ", "summary_of_the_review": "I would recommend rejection since the authors only investigate few-shot image classification tasks. More analysis and experiments are needed for such an analysis paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7HRr30fx0nR", "reviewer": "Reviewer_MHHW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In this work, the authors provide an empirical evaluation of various sampling mechanisms which can be used to create few-shot tasks during an episodic training regime that's common with meta-learning training. The list of sampling strategies include simple methods like only using a fixed set of tasks in each episode and more complicated strategies like online-hard-negative-mining. Through experiments on multiple popular few-shot learning datasets, the authors show which sampling methods do work well and which of them do not.\n\nStrengths\n--------------\n* Narrative is clear and the paper is enjoyable to read. Various pictorial representations of different sampling strategies make it easy to follow.\n* Experimental setup is quite exhaustive, from both the sampling mechanism point of view and also the baseline datasets chosen. \n\nWeaknesses\n------------------\n* Lack of Novelty -- There is no clear novel contribution in this paper either from a theoretical or empirical perspective. The authors evaluated a series of existing task-sampling mechanisms using a bunch of existing meta-learning algorithms on few-shot benchmarks. It would have been good to see some novelty in terms of the sampling mechanism -- if uniform sampling is really the best one and that is what practitioners generally use as far as I know, then this analysis does not provide a lot of value.\n\n* Lack of Explanation and Unfair Comparisons -- The paper does not do a good job in explaining why some of the sampling methods work better than others and in some cases, the explanations are trivial. For example, the NDT sampler performs poorly because it is only trained on a subset of data compared to the uniform sampler and therefore, is expected to provide weaker performance. However, the NDB sampler performs almost similar to uniform although it also works with limited amount of data. Why is that? What is inherently different between the subset of data selected by these two methods? One explanation I can think of is that for NDB, the initial batch contains some representations from a majority of classes in which case, its comparison to NDB is again unfair (which only contain images from a limited classes). Another similar topic is why convergence of SBU is slower than NDTB? Is it because in NDTB, each task is repeated K-times and the model has simply performed K-times more optimization steps compared to SBU? In that case, performance of SBU vs NDTB should be compared when SBU has run for k-times more epochs. ", "summary_of_the_paper": "In this work, the authors propose various task-sampling strategies for an episodic meta-learning setup and compare their performances against a standard uniform sampling. Through experiments across diverse benchmark datasets, the authors empirically show which of these methods underperform the uniform sampling and which of those perform at-par. ", "summary_of_the_review": "Although the empirical setup is well designed and the paper is easy to follow, the lack of novelty in the contribution and the apparent lack of clarity in the explanations/analysis led to my non-acceptance for this paper at this point. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper set out to show that increasing task diversity during meta-training process does not boost performance. The reviewers mostly  agreed (only reviewer wVFn dissented) that the empirical set up of the paper was convincing, but they also felt it over-emphasized empirics over a deeper understanding of the phenomena observed. In turn, this resulted in discussions around how the experiments and the explanations didn't fully prove that increasing task diversity does not help. Overall, the discussion and the additional analysis tools provided by the authors (such as the diversity metric) will greatly improve the paper.
Decision: Reject