Year: 2021

Abstract:
Few-shot learning aims to learn representations that can tackle novel tasks given a small number of examples. Recent studies show that task distribution plays a vital role in the performance of the model. Conventional wisdom is that task diversity should improve the performance of meta-learning. In this work, we find evidence to the contrary; we study different task distributions on a myriad of models and datasets to evaluate the effect of task diversity on meta-learning algorithms. For this experiment, we train on multiple datasets, and with three broad classes of meta-learning models Metric-based (i.e., Protonet, Matching Networks), Optimization-based (i.e., MAML, Reptile, and MetaOptNet), and Bayesian meta-learning models (i.e., CNAPs). Our experiments demonstrate that the effect of task diversity on all these algorithms follows a similar trend, and task diversity does not seem to offer any benefits to the learning of the model. Furthermore, we also demonstrate that even a handful of tasks, repeated over multiple batches, would be sufficient to achieve a performance similar to uniform sampling and draws into question the need for additional tasks to create better models.

heading: 1 INTRODUCTION
text: It is widely recognized that humans can learn new concepts based on very little supervision, i.e., with few examples (or ”shots”), and generalize these concepts to unseen data as mentioned by Lake et al. (2011). Recent advances in deep learning, on the other hand, have primarily relied on datasets with large amounts of labeled examples, primarily due to overfitting concerns in low data regimes. Although the development of better data augmentation and regularization techniques can alleviate these concerns, many researchers now assume that future breakthroughs in low data regimes will emerge from meta-learning, or ”learning to learn.” Here, we study the effect of task diversity in the low data regime and its effect on various models. In this meta-learning setting, a model is trained on a handful of labeled examples at a time under the assumption that it will learn how to correctly project examples of different classes and generalize this knowledge to unseen labels at test time. Although this setting is often used to illustrate the remaining gap between human capabilities and machine learning, we could argue that the domain of meta-learning is still nascent. The domain of task selection has remained virtually unexplored in this setting.
Conventional wisdom is that the performance of the model will improve as we train on more diverse tasks. To test this hypothesis to its limits, we define various task samplers which either limit task diversity by selecting a subset of overall tasks or improving task diversity using approaches such as Determinantal Point Processes (DPPs) proposed by Macchi (1975).
Our contributions in this work are as follows:
• We show that, against conventional wisdom, task diversity does not significantly boost performance in meta-learning. Instead, limiting task diversity and repeating the same tasks over the training phase allows the model to obtain performances similar to models trained on Uniform Sampler without any adverse effects.
• We also show that increasing task diversity using sophisticated samplers such as DPP or Online Hard Task Mining (OHTM) Samplers do not significantly boost performance. Instead, the dynamic-DPP Sampler harms the model due to the increased task diversity.
• We empirically show that repeating tasks over the training phase can perform similarly to a model trained on the Uniform Sampler, achieving similar performance with only a
fragment of data. This key finding questions the need to increase the support set pool to improve the model’s performance.

heading: 2 RELATED WORKS
text: Meta-learning formulations typically rely on episodic training, wherein an algorithm adapts to a task, given its support set, to minimize the loss incurred on the query set. Meta-learning methods differ in terms of the algorithms they learn, and can be broadly classified under four prominent classes: Metric-based, Model-based, Optimization-based and Bayesian-based approaches. Metricbased methods such as Koch et al. (2015); Vinyals et al. (2016); Snell et al. (2017); Sung et al. (2018) operate on the core idea similar to nearest neighbors algorithm and kernel density estimation. These methods are also called non-parametric approaches. Model-based methods such as Santoro et al. (2016); Munkhdalai & Yu (2017) depend on a model designed specifically for fast learning, which updates its parameters rapidly with a few training steps, achieved by its internal architecture or controlled by another meta-learner model. Generic deep learning models learn through backpropagation of gradients, which are neither designed to cope with a small number of training samples nor converge within a few optimization steps. To address this, Optimization-based methods such as Ravi & Larochelle (2016); Finn et al. (2017); Nichol et al. (2018) were proposed, which were better suited to learn from a small number of samples. However, all the above approaches are deterministic and are not the most suited for few-shot problems that are generally ambiguous. Hence, Bayesian-based methods such as Yoon et al. (2018); Requeima et al. (2019) were proposed which helped address the above issue.
Although research in meta-learning models has attracted much attention recently, the effect of task diversity is virtually unexplored in the domain of meta-learning. However, task sampling and task diversity have been more extensively studied in other closely related problems such as active learning. Active learning involves selecting unlabeled data items in order to improve an existing classifier. Although most of the approaches in this domain are based on heuristics, there are few approaches to sample a batch of samples for active learning. Ravi & Larochelle (2018) proposed an approach to sample a batch of samples using a protonet as the backbone architecture. The model tries to maximize the query set, given support set and unlabeled data. Other works such as Hsu et al. (2018) proposed a framework named CACTUs, which samples tasks/examples using relatively simple task construction mechanisms such as clustering embeddings. The unsupervised representations learned via these samples lead to a good performance on various downstream human-specified tasks.
Although nascent, a few recent works aim to improve meta-learning by explicitly looking at the task structure and relationships. Among these, Yin et al. (2019) proposed an approach to handle the lack of mutual exclusiveness among different tasks through an information-theoretic regularized objective. In addition, several popular meta-learning methods Lee et al. (2019); Snell et al. (2017) improve the meta-test performance by changing the number of ways or shots of the sampled metatraining tasks, thus increasing the complexity and diversity of the tasks. Other works such as Liu et al. (2020a) proposed an approach to sample classes using class-pair-based sampling and classbased sampling. The Class-pair based Sampler selects pairs of classes that confuse the model the most. The class-based Sampler samples each class independently and does not consider the task’s difficulty as a whole. Our OHTM sampler is similar to the Class-pair based Sampler. Other works such as Liu et al. (2020b) propose to augment the set of possible tasks by augmenting the predefined set of classes that generate the tasks with varying degrees of rotated inputs as new classes. Other works such as Setlur et al. (2020) look at the structure and diversity of tasks specifically through the lens of support set diversity, and show that, surprisingly, reducing diversity (by fixing support set) not only maintains—but in many cases, significantly improves—the performance of meta-learning. This experiment is very similar to our No Diversity Task Sampler if the size of the support set is equal to the number of classes per task. However, in this work, we extend their work on MetaOptNet, Protonet to many other models and a myriad of samplers to better understand task diversity in meta-learning. To the best of our knowledge, we are the first to study the effect of task diversity in meta-learning to this extent.

heading: 3 BACKGROUND
text: Here, we review some of the fundamental ideas required to understand our few-shot learning experiments better.

heading: 3.1 EPISODIC FEW-SHOT LEARNING
text: In episodic few-shot learning, an episode is represented as a K-way, N-shot classification problem where N is the number of examples per class and K is the number of unique class labels. During training, the data in each episode is provided as a support set S = {(x1,1, y1,1), ..., (xN,K , yN,K)} where xi,j ∈ RD is the i-th instance of the j-th class, and yj ∈ {0, 1}K is its corresponding one-hot labeling vector. Each episode aims to optimize a function f that classifies new instances provided through a ”query” set Q, containing instances of the same class as S. This task is difficult because N is typically very small (e,g, 1 to 10). The classes change every episode. The actual test set used to evaluate a model does not contain classes seen in support sets during training. In the task-distribution view, meta-learning is a general-purpose learning algorithm that can generalize across tasks and ideally enable each new task to be learned better than the last. We can evaluate the performance of ω over a distribution of tasks p(T ). Here we loosely define a task to be a dataset and loss function T = {D,L}. Learning how to learn thus becomes:
min ω E τ∼p(τ) L(D;ω) (1)
whereL(D;ω) measures the performance of a model trained using ω on datasetD and p(τ) indicates the task distribution. In this experiment, we extend this setting such that we vary the task diversity in the train split to study the effects on test split, which remains to use uniform or random sampling for tasks.

heading: 3.2 DETERMINANTAL POINT PROCESSES (DPPS)
text: A DPP is a probability distribution over subsets of a ground set Y , where we assume Y = {1, 2, ..., N} and N = |Y|. An L-ensemble defines a DPP using a real, symmetric, and positivedefinite matrix L indexed by the elements of Y . The probability of sampling a subset Y = A ⊆ Y can be written as: P (Y = A) ∝ det LA, (2) where LA := [Li,j ]i,j∈A is the restriction of L to the entries indexed by the elements of A. As L is a positive semi-definite, there exists a d × N matrix Ψ such that L = ΨTΨ where d ≤ N . Using this principle, we define the probability of sampling as:
P (Y = A) ∝ det LA = Vol2({Ψi}i∈A), (3)
where the RHS is the squared volume of the parallelepiped spanned by {Ψi}i∈A. In Eq. 3, Ψi is defined as the feature vector of element i, and each element Li,j in L is the similarity measured by dot products between elements i and j. Hence, we can verify that a DPP places higher probabilities on diverse sets because the more orthogonal the feature vectors are, the larger the volume parallelepiped spanned by the feature vector is. In this work, these feature embeddings represent class embeddings, which are derived using either a pre-trained protonet model or the model being trained as discussed in Sec. 3.3.
In a DPP, the cardinality of a sampled subset, |A|, is random in general. A k-DPP is an extension of the DPP proposed in the work of Kuhn et al. (2003), where the cardinality of subsets are fixed as k (i.e., |A| =k). In this work, we use k-DPPs as an off-the-shelf implementation to retrieve classes that represent a task used in the meta-learning step.

heading: 3.3 TASK SAMPLING
text: In this work, we experiment with eight distinct task samplers, each offering a different level of task diversity. To demonstrate the task samplers, we use a 2-way classification problem with a meta-batch size of 2 and denote each class with a unique alphabet from the Omniglot dataset.
Uniform Sampler This is the most widely used Sampler used in the setting of meta-learning. The Sampler gives equal probability to every task and is intuitively a random sampler. An illustration of this Sampler is shown in Figure 1.
No Diversity Task Sampler In this setting, we uniformly sample one set of the task at the beginning and propagate the same task across all batches and meta-batches. Note that repeating the same class over and over again does not simply repeat the same images/inputs as we episodically retrieve different images for each class. An illustration of this Sampler is shown in Figure 1.
No Diversity Batch Sampler In this setting, we uniformly sample one set of tasks for batch one and propagate the same tasks across all other batches. Furthermore, we shuffle these tasks to enforce that the model does not overfit. An illustration of this Sampler is shown in Figure 1.
No Diversity Tasks per Batch Sampler In this setting, we uniformly sample one set of tasks for a given batch and propagate the same tasks for all meta-batches. We then repeat this same principle for sampling the next batch. Furthermore, we shuffle these tasks to enforce that the model does not overfit. An illustration of this Sampler is shown in Figure 2.
Single Batch Uniform Sampler In this setting, we set the meta-batch size to one. This Sampler is intuitively the same as no diversity task per batch sampler, without the repetition of tasks. This
Sampler would be an ideal ablation study for the repetition of tasks in the meta-learning setting. An illustration of this Sampler is shown in Figure 2.
Online Hard Task Mining Sampler This setting is inspired by the works of Shrivastava et al. (2016) where they proposed OHEM, which yielded significant boosts in detection performance on benchmarks like PASCAL VOC 2007 and 2012. However, to reproduce OHEM for meta-learning, we only apply the OHEM sampler for half the meta-batch size and uniform sampler for the remaining half. This approach would allow us to involve many tasks and not restrict us to only known tasks. Furthermore, to avoid OHEM in the initial stages, we sample tasks with a uniform sampler until the buffer of tasks seen by the model becomes sufficiently big, say 50 in our case. An illustration of this Sampler is shown in Figure 3.
Static DPP Sampler Determinantal Point Processes (DPP) have been used for several machine learning problems such as the works of Kulesza & Taskar (2012). They have also been used in other problems such as the active learning settings in the works of Bıyık et al. (2019) and mini-batch sampling problems in the works of Zhang et al. (2019). These algorithms have also inspired other works in active learning in the batch mode setting, such as Ravi & Larochelle (2018). In this setting, we use DPP as an off-the-shelf implementation to sample tasks based on task embeddings. These task embeddings are generated using our pre-trained protonet model. The DPP instance is used to sample the most diverse tasks based on these embeddings and used for meta-learning. An illustration of this Sampler is shown in Figure 3.
Dynamic DPP Sampler In this setting, we extend the previous sDPP setting such that the model in training generates the task embeddings. The Sampler is motivated by the intuition that selecting the most diverse tasks for a given model will help learn better. Furthermore, to avoid DPP in the initial stages, we sample tasks with a uniform sampler until the model becomes sufficiently trained, say 500 batches in our case. An illustration of this Sampler is shown in Figure 3.

heading: 4 EXPERIMENTS
text: The experiment aims to answer the following questions: (a) How does task diversity affect metalearning? (b) Do sophisticated samplers such as OHEM or DPP offer any significant boost in performance? (c) Are there any rule of thumb or general good practices when it comes to sampling tasks?
To make an exhaustive study on the effect of task diversity in meta-learning, we train on four datasets: Omniglot Lake et al. (2011), miniImagenet Ravi & Larochelle (2016), tieredImageNet Ren et al. (2018), and Meta-Dataset Triantafillou et al. (2019). With this selection of datasets, we cover both simple datasets, such as Omniglot and miniImageNet, as well as the most difficult ones, such as tieredImageNet and Meta-Dataset. We train three broad classes of meta-learning models on
these datasets - Metric-based (i.e., Protonet, Matching Networks), Optimization-based (i.e., MAML, Reptile, and MetaOptNet), and Bayesian meta-learning models (i.e., CNAPs). More details about the datasets which were used in our experiments are discussed in App. A.1. More details about the models and their hyperparameters are discussed in App. A.2. We created a common pool of 1024 randomly sampled held-out tasks to test every algorithm in our experiments to make an accurate comparison. For all experiments, we assessed the statistical significance of our results based on a paired-difference t-test, with a p-value p = 0.05.

heading: 4.1 RESULTS
text: In this section, we present the results of our experiments. Figure 4 presents the performance of the six models on the Omniglot and miniImageNet under different task samplers in the 5-way 1-shot setting. Table 2 in the Appendix presents the same results with higher precision.
We also reproduce our experiments on the 20-way 1-shot setting on the Omniglot dataset to establish that these trends are shared across different settings. Figure 5 presents our performance of the models under this setting. Furthermore, the results on the 20-way 1-shot experiments are presented in Table 3 in the Appendix with higher precision. To further establish our findings, we also present our result on notoriously harder datasets such as tieredImageNet and Meta-Dataset. Figure 6 presents the performance of the models on the tieredImageNet. Again, Table 2 in the Appendix presents the same results with higher precision.
Figure 6 presents the performance of the models on the Meta-Dataset Traffic Sign and Meta-Dataset MSCOCO datasets. We only present the results on Traffic Sign and MSCOCO of the Meta-Dataset, as these two sub-datasets are exclusively used for testing and are an accurate representation of the generalization power of the models when trained with different levels of task diversity. Other results on the Meta-Dataset are presented in Table 4. We empirically show that task diversity does not lead to any significant boost in the performance of the models. In the subsequent section, we discuss some of the other key findings from our work.

heading: 5 DISCUSSION
text: In this section, we discuss few empirical results from our experiments and shed light on some of the key findings from our research.
Poor performance by NDT Sampler The lowest performance is consistently obtained by the No Diversity Task Sampler, which is reasonable since the model only sees one task throughout its training. What is fascinating is that just one task is sufficient for the model to reach a reasonably decent performance in most cases. We do notice instances where NDT Sampler performs very well on a few sub-datasets of the Meta-Dataset. This can be explained by the fact that the model has only be trained for a single sub-dataset and has relatively less class variability (noise) when compared to training on multiple sub-datasets.
Poor performance by Single Batch Uniform Sampler Consequently, the Single Batch Uniform Sampler does perform poorly on most datasets, including Omniglot, miniImageNet, and tieredImageNet. This is reasonable since the model is trained on a tiny pool of the dataset. However, we notice instances where the Sampler performs very well on a few sub-datasets of the MetaDataset. We hypothesize that training on fewer samples keeps the model unaware of the inherent
class variability generated by training on diverse datasets and aids better performance in the case of Meta-Dataset.
Disparity between Single Batch Uniform and NDTB Sampler Another exciting result is the Disparity between Single Batch Uniform Sampler and No Diversity Tasks per Batch Sampler. As mentioned earlier, the only difference between the two samplers is that tasks are repeated in the latter. However, this repetition seems to offer a great deal of information to the model and allows the model to perform on par with the Uniform Sampler. It might be possible that the Single Batch Uniform Sampler obtains the performance observed by the No Diversity Tasks per Batch Sampler if trained for enough epochs. However, it would be safe to comment that the convergence of the model is significantly faster in the latter. Thus, repeating tasks might help speed up the convergence of the model when we have a fixed and handful amount of data. However, the same is not valid for models trained on Meta-Dataset. Although both samplers are trained over a similar pool of datasets, the NDTB sampler sees more data and might lead to more inherent class variability generated by training on diverse datasets. This might explain why repeating tasks leads to lower performance in this case.
Disparity between s-DPP and d-DPP Sampler We also note that s-DPP and d-DPP samplers do not offer any boost in performance when compared to the regular Uniform Sampler. Furthermore, there seems to be a significant disparity between these two samplers. We believe that d-DPP, which computes the most diverse tasks at regular intervals, harms the model with the diverse tasks since we observe that the model’s performance degrades over epochs. For example, consider the scenario where the model is trained on tasks involving dogs and tractors. This task is relatively easy to learn and would not require the model to fine-tune a great deal. However, during test time, suppose our task involves classifying cats and dogs; this would be a problem since the model has not learned the intricacies of the two classes. Thus, diversity seems to do more harm than good in this case. The best example of this is observed by Matching Networks in Omniglot 5-way 1-shot setting as shown in Figure 10, where each instance of diverse sampling harms the model significantly.
Limitation of samplers with DPP backbone Samplers such as s-DPP and d-DPP, which use DPP to sample diverse tasks, require task embeddings of every class in the dataset. Computing these task embeddings, although intensive, might be sustainable for small datasets such as Omniglot, miniImageNet, and tieredImageNet. However, computing the task embedding for every class of a dataset as large as the Meta-Dataset is nearly impossible due to the time and memory constraints. Hence, in our experiments on the Meta-Dataset, we do not run the model using the s-DPP and d-DPP Sampler and only report the findings from the remaining samplers.
OHTM Sampler offers no significant performance boost The OHTM Sampler is quite sophisticated since it regularly samples diverse tasks, as well as selects the most challenging tasks to improve the model. It is needless to say; the model requires more computational power and time than the Uniform Sampler. However, the OHTM Sampler offers no significant boost in performance when compared to the Uniform Sampler in the case of Omniglot, miniImageNet, and tieredImageNet. However, for Meta-Dataset, we notice that the OHTM Sampler sometimes leads to improved performance. This finding is quite puzzling since the Sampler works similarly across all datasets and does not address the inherent class variability generated from training on diverse datasets such as the Meta-Dataset. The behavior of the OHTM Sampler on Meta-Datasets warrants further research.
Comparison between NDTB, NDB, and Uniform Sampler From our experiments, we also notice that the No Diversity Tasks per Batch Sampler and No Diversity Batch Sampler are pretty similar to the Uniform Sampler in terms of performance. This would suggest that the model trained on only a data fragment can perform similarly to that trained on the Uniform Sampler.
Abnormal run of matching networks d-DPP (20-way 1-shot) In our run on the matching networks with the d-DPP Sampler under the 20-way 1-shot setting, we ran across a peculiar error. The prototypes generated by the matching networks were sometimes not fit to be used by the dDPP Sampler to sample 20 unique classes. The reason is that the rank of the matrix generated using the embeddings was lower than the required number of classes per task (i.e., 20). To create a workaround for this sole experiment, we chose to sample 5 diverse classes at a time and append them
to create the task. We hypothesize that the prototypes created by matching networks are unsuitable for downstream tasks and warrant further research regarding this behavior.
Poor performance of MAML on Meta-Dataset In our experiments on Meta-Dataset, we notice that MAML performs significantly worse than other models. Some of the reasons for this disparity of performance when compared to Triantafillou et al. (2019) have been discussed in detail in Appendix A.1. Furthermore, we hypothesize that the poor performance of MAML can be attributed to the way its adaptation process works: MAML learns within episode weights of the model before adapting to a set of new tasks via meta-update or outer loop update. This new task is again sampled from the pool of tasks and is a different set of data altogether. In most cases, where the model is trained on only one dataset, this intuition would make sense and lead to high-performing models. However, in the case of Meta-Dataset, where the new set of tasks might be of entirely different datasets or domains, this approach tends to do more harm to the model rather than aid. In the work of Triantafillou et al. (2019), the authors adapted MAML such that it focuses on learning the within-episode initialization θ of the embedding network so that it can be rapidly adapted for a new task. This allows the model to learn from a variable number of ways and shots per episode.
Peculiar behavior with MetaOptNet model Compared to all other models, MetaOptNet seems to be immune to the effects of task diversity to a great extent. The convergence of the model seems to follow a general pattern and achieve similar performance across task distributions except for the Single Batch Uniform Sampler and No Diversity Task sampler. Furthermore, we do not observe the expected pattern of d-DPP Sampler, where the performance drops upon mining diverse tasks. We present the convergence graph of the MetaOptNet model on Omniglot 5-way 1-shot run in Figure 11 in the Appendix with an added smoothing factor of 1.
General Trend From our experiments, we notice that there are generally two classes of samplers: High Performing Samplers and Low Performing Samplers. The High Performing Samplers include No Diversity Batch, No Diversity Tasks per Batch, Uniform, OHTM, and s-DPP Sampler. The Low Performing Samplers include No Diversity Task, Single Batch Uniform, and d-DPP Sampler. This trend is shared across all datasets and models. There are some perturbations in ranking within the two classes, but the High Performing Samplers tend to perform better than the Low Performing Samplers.

heading: 6 CONCLUSION
text: In this paper, we have studied the effect of task diversity in meta-learning. We have empirically shown that task diversity does not lead to any significant boost in performance in meta-learning. Instead, limiting task diversity and repeating the same tasks over the training phase allows us to obtain similar performances to the Uniform Sampler without any significant adverse effects. Furthermore, We also show that sophisticated samplers such as OHEM or DPP samplers do not offer any significant boost in performance. In contradiction, we notice that increasing task diversity using the d-DPP Sampler hampers the performance of the meta-learning model. Our experiments using the NDTB and NDB empirically show that a model trained on even a tiny data fragment can perform similarly to a model trained using Uniform Sampler. This is a crucial finding since this questions the need to increase the support set pool to improve the models’ performance. We believe that the experiments we performed lay the roadwork to further research for the effect of task diversity domain in meta-learning and lay some groundwork and rules of thumb for task sampling for meta-learning.

heading: REPRODUCABILITY STATEMENT
text: In this paper, we work with four different datasets - Omniglot, miniImageNet, tieredImageNet and Meta-Dataset. Additional details about setting up these datasets is available in Appendix A.1. Furthermore, we experiment with six different models - MAML, Reptile, Protonet, Matching Networks, MetaOptNet, and CNAPs. All these models were run after reproducing from their open-source codes. Additional details about setting up these models are available in Appendix A.2.

heading: A APPENDIX

heading: A.1 DATASETS
text: Omniglot, miniImageNet and tieredImageNet were extracted using pytorch-meta. The Meta-Dataset was downloaded using the setup information from the official repository: https://github. com/google-research/meta-dataset.
Omniglot Omniglot is a benchmark dataset proposed by Lake et al. (2011) for few-shot image classification tasks. Omniglot dataset consists of 20 instances and 1623 characters from 50 different alphabets. We experiment with both 5-way 1-shot and 2-way 1-shot in this work.
miniImageNet miniImageNet is another benchmark dataset proposed by Ravi & Larochelle (2016) for few-shot image classification tasks. The miniImageNet dataset involves 64 training classes, 12 validation classes, and 24 test classes. We run under the setting 5-way 1-shot for this experiment.
tieredImageNet tieredImageNet is notorious as a difficult benchmark dataset proposed by Ren et al. (2018). The tieredImageNet dataset is a larger subset of ILSVRC-12 with 608 classes (779,165 images) grouped into 34 higher-level nodes in the ImageNet human-curated hierarchy. This set of nodes is partitioned into 20, 6, and 8 disjoint sets of training, validation, and testing nodes, and the corresponding classes form the respective meta-sets. As argued in Ren et al. (2018), this split near the root of the ImageNet hierarchy results in a more challenging yet realistic regime with test classes that are less similar to training classes. We run under the setting 5-way 1-shot for this experiment.
Meta-Dataset Meta-Dataset is notorious as a difficult benchmark dataset proposed by Triantafillou et al. (2019). The Meta-Dataset is much larger than any previous benchmark and is comprised of multiple existing datasets. This invites research into how diverse data sources can be exploited by a meta-learner and allows us to evaluate a more challenging generalization problem to new datasets altogether. Specifically, Meta-Dataset leverages data from the following 10 datasets: ILSVRC-2012 (Russakovsky et al. (2015)), Omniglot (Lake et al. (2011)), Aircraft (Maji et al. (2013)), CUB-2002011 (Wah et al. (2011)), Describable Textures (Cimpoi et al. (2014)), Quick Draw (Jongejan et al. (2016)), Fungi (Schroeder & Cui (2018)), VGG Flower (Nilsback & Zisserman (2008)), Traffic Signs (Houben et al. (2013)) and MSCOCO (Lin et al. (2014)). There exist few classes with fewer image samples than 16. This becomes an issue since we need 1 for training and 15 for testing. We repeat some of the support images for these classes to make up for the lack of examples. Since the number of such classes is minimal, we justify this use as this solution cannot significantly increase the model’s performance. Furthermore, unlike previous experiments on this dataset which use a variable number of ways and shots during training, we train with a fixed number of ways and shots. Furthermore, unlike the works in Triantafillou et al. (2019), we do not perform any pre-training to help aid the model. We believe these are the main attributions for the disparity of our performance. However, since we are focused on the relative performance of the samplers for a given model, this discrepancy would not affect our study of task diversity in any manner. We again run under the setting 5-way 1-shot for this experiment.

heading: A.2 MODELS
text: This section describes some of the models we used for our experiments and the hyperparameters used for their training.

heading: A.2.1 PROTOTYPICAL NETWORKS
text: Prototypical Networks proposed by Snell et al. (2017) constructs a prototype for each class and then classifies each query example as the class whose prototype is ’nearest’ to it under Euclidean distance. More concretely, the probability that a query example x∗ belongs to class k is defined as:
p(y∗ = k|x∗,S) = exp(−‖g(x ∗)− ck‖22)∑
k′∈{1,...,N} exp(−‖g(x∗)− ck′‖ 2 2)
(4)
Where ck is the ’prototype’ for class k: the average embeddings of class k’s support examples.
HYPERPARAMETERS In our experiments on Omniglot and miniImageNet, and tieredImageNet under a 5-way, 1-shot setting, we run the model for 100 epochs with a batch size of 32 and a metalearning rate of 0.001. We use an Adam optimizer to make gradient steps and a StepLR scheduler with step size 0.4 and gamma 0.5. The same hyperparameters are used for training our model on Omniglot under a 20-way 1-shot setting.
We use the same parameters as the miniImageNet to train our model on the Meta-Dataset under a 5-way 1-shot setting. However, we run with a batch size of 16 rather than 32 to accommodate the extensive training period and memory constraints.

heading: A.2.2 MATCHING NETWORKS
text: Matching Networks proposed by Vinyals et al. (2016) labels each query example as a cosine distance-weighted linear combination of the support labels:
p(y∗ = k|x∗,S) = |S|∑ i=1 α(x∗, xi)Φyi=k, (5)
where ΦA is the indicator function and α(x∗, xi) is the cosine similarity between g(x∗) and g(xi), softmax normalized over all support examples xi, where 1 ≤ i ≤ |S|. We had trouble reproducing the results from matching networks using cosine distance since the convergence seemed to be slow and the final performance dependent on the random initialization. This is similar to what is observed by other repositories such as https://github.com/ oscarknagg/few-shot. Since we are focused on the relative performance of the samplers for a given model, this discrepancy would not affect our study of task diversity in any manner.
HYPERPARAMETERS In our experiments on Omniglot, miniImageNet, and tieredImageNet under a 5-way, 1-shot setting, we run the model for 100 epochs with a batch size of 32 and an Adam optimizer with a meta-learning rate of 0.001 and a weight decay of 0.0001. The same hyperparameters are used for training our model on Omniglot under a 20-way 1-shot setting.
We use the same parameters as the miniImageNet to train our model on the Meta-Dataset under a 5-way 1-shot setting. However, we run with a batch size of 16 rather than 32 to accommodate the extensive training period and memory constraints.
A.2.3 MAML
MAML proposed by Finn et al. (2017) uses a linear layer parametrized by W and b on top of the embedding function g(.; θ) and classifies a query example as: p(y∗|x∗,S) = softmax (b′ + W′g(x∗; θ′)), (6) where the output layer parameters W’ and b’ and the embedding function parameters θ′ are obtained by performing a small number of within-episode training steps on the support set S, starting from initial parameter values (b,W, θ).
HYPERPARAMETERS In our experiments on Omniglot, miniImageNet, and tieredImageNet under a 5-way, 1-shot setting, we run the model for 150 epochs with a batch size of 32, with the Adam optimizer with a meta-learning rate of 0.001, number of inner adaptations as 1, and step size 0.4. For our experiments on Omniglot under the 20-way 1-shot setting, we set the step size of 0.1 and the number of inner adaptations to 5, batch size of 16, and kept all other hyperparameters constant.
We use the same parameters as the miniImageNet to train our model on the Meta-Dataset under a 5-way 1-shot setting. However, we ran with a batch size of 16 rather than 32 and only trained for 100 epochs to accommodate the extensive training period and memory constraints.

heading: A.2.4 REPTILE
text: Like MAML, Reptile proposed by Nichol et al. (2018) learns an initialization for the parameters of a neural network model, such that when we optimize these parameters at test time, learning is fast -
i.e., the model generalizes from a small number of test tasks. Reptile converges towards a solution φ that is close (in Euclidean distance) to each task τ ’s manifold of optimal solutions. Let φ denote the network initialization, and W = φ+ ∆φ denote the network weights after performing some sort of update. LetW∗τ denote the set of optimal network weights for task τ . We want to find φ such that the distance D(φ,W∗τ ) is small for all tasks:
min φ
Eτ [ 1
2 D(φ,W∗τ )2] (7)
The official repository seems to train the model with a 5-way 15-shot and test the model on a 5-way 1-shot. However, we do not consider this to be an accurate study for the effect of task diversity. In our work, we train and test the model in a 5-way 1-shot setting to ensure fair and accurate comparison with other models. We believe this to be the source of discrepancy in our performance scores. Since we are focused on the relative performance of the samplers for a given model, this discrepancy would not affect our study of task diversity in any manner.
HYPERPARAMETERS In our experiments on Omniglot and miniImageNet under a 5-way, 1-shot setting, we run the model for 150 epochs with a batch size of 32, a learning rate of 0.01, a metalearning rate of 0.001, and a number of inner adaptations as 5. We use the SGD optimizer of inner steps and Adam optimizer for the outer steps. For our experiments on tieredImageNet, we change the number of inner adaptations to 10, keeping all other hyperparameters constant. For our experiments on Omniglot under the 20-way 1-shot setting, we set the meta-learning rate to 0.0005 and the number of inner adaptations to 10 and kept all other hyperparameters constant. Furthermore, we only run the model for 50 epochs due to the very high training time.
We use the same parameters as the tieredImageNet to train our model on the Meta-Dataset under a 5-way 1-shot setting. However, we run with a batch size of 16 rather than 32 to accommodate the extensive training period and memory constraints.

heading: A.2.5 CNAPS
text: Conditional Neural Adaptive Processes proposed by Requeima et al. (2019) is able to efficiently solve new multi-class classification problems after an initial training phase. The proposed approach, based on Conditional Neural Processes (CNPs) mentioned in Garnelo et al. (2018), adapts a small number of task-specific parameters for each new task encountered at test time. These parameters are conditioned on a set of training examples for the new task. They do not require any additional tuning to adapt both the final classification layer and feature extraction process. This allows the model to handle various input distributions. The CNPs construct predictive distributions given x∗ as:
p(y∗|x∗, θ,Dτ ) = p(y∗|x∗, θ,Ψτ = Ψφ(Dτ )), (8) where θ are global classifer parameters shared across tasks, Ψτ are local task-specific parameters, produced by a function Ψφ(.) that acts of Dτ . Ψφ(.) has another set of global parameters φ called adaptation network parameters. θ and φ are the learnable parameters in the model.
HYPERPARAMETERS In all our experiments with CNAPs, we run the model for ten epochs with a batch size of 16 and a meta-learning rate of 0.01.

heading: A.2.6 METAOPTNET
text: MetaOptNet proposed by Lee et al. (2019) proposes a linear classifier as the base learner for a metalearning based approach for few-shot learning. The approach uses a linear support vector machine (SVM) to learn a classifier given a set of labeled training examples. The generalization error is computed on a novel set of examples from the same task. The objective is to learn an embedding model φ that minimizes generalization (or test) error across tasks given a base learner A. Formally, the learning objective is:
min φ
ET [Lmeta(Dtest ; θ, φ), where θ = A(Dtest ;φ)]. (9)
The choice of base learner A has a significant impact on the above equation. The base learner that computes θ = A(Dtest ;φ) has to be efficient since the expectation has to be computed over a
distribution of tasks. This work considers base learners based on multi-class linear classifiers such as SVM, where the base learner’s object is convex. Thus, the base learner can be simplified as:
θ = A(Dtest ;φ) = arg min {wk} min ξi 1 2
∑ k ‖wk‖ 2 2 + C ∑ n ξn; subject to:
wyn .fφ(xn)−wk.fφ(xn) ≥ 1− δyn,k − ξn,∀n, k (10)
where Dtrain = {(xn, yn)}, C is the regularization parameter and δ.,. is the Kronecker delta function.
Furthermore, the official repository seems to train the model with a 5-way 15-shot and test the model on a 5-way 1-shot. However, we do not consider this to be an accurate study for the effect of task diversity. In our work, we train and test the model in a 5-way 1-shot setting to ensure fair and accurate comparison with other models. We believe this to be the source of discrepancy in our performance scores. Since we are focused on the relative performance of the samplers for a given model, this discrepancy would not affect our study of task diversity in any manner.
HYPERPARAMETERS In our experiments on Omniglot, miniImageNet and tieredImageNet under a 5-way, 1-shot setting, we run the model for 60 epochs with a batch size of 32 and a meta-learning rate of 0.01. We use an SGD optimizer with a momentum of 0.9 and a weight decay of 0.0001 to make gradient steps. We also use a LambdaLR scheduler to train our model. The same hyperparameters are used for training our model on Omniglot under a 20-way 1-shot setting.
We use the same parameters as the miniImageNet to train our model on the Meta-Dataset under a 5-way 1-shot setting. However, we run with a batch size of 16 rather than 32 to accommodate the extensive training period and memory constraints.

heading: A.3 ADDITIONAL RESULTS

heading: A.3.1 RESULTS ON REGRESSION
text: We further experiment on the few-shot regression regime to maintain our findings on classification. For this problem, we trained on three few-shot regression datasets: (i) Sinusoid (Finn et al. (2017)), (ii)Sinusoid & Line (Finn et al. (2018)), and (iii) Harmonic (Lacoste et al. (2018). Table 1 presents our results using the mean square error metric on the 5-shot and 10-shot setting for all three datasets. Figure 7 and Figure 8 presents our results on the Sinusoid dataset specifically for easier comparison. For both MAML and Reptile, we use the same network, with two hidden layers, each of size 40.
HYPERPARAMETERS For both MAML and Reptile, we train the model for 150 epochs, and use the same hyperparameters as Omniglot. Only in the case of MAML, we use a step size of 0.001 for Sinusoid and Sinusoid & Line dataset, instead of the traditional 0.4.
Results We again observe that samplers that limit diversity, such as NDB, NDT/B samplers achieve performances similar to the uniform sampler with no adverse effects. Astonishingly, we also notice that the NDT sampler seems to outperform all other samplers in the regression domain. Furthermore, similar to our previous findings, samplers that increase diversity, such as the OHTM sampler, do not perform very well in this domain either, as shown in Figure 7 and Figure 8. We cannot train using DPP-oriented samplers for regression since the data is continuous, and computing features or representations for the entire range would be theoretically impossible.

heading: A.3.2 RESULTS ON CLASSIFICATION
text: In this section, we present the results with higher precision from our earlier experiments in a Table 2 and Table 3. Subsequently, we also plot convergence curves to aid better visualizations of findings mentioned earlier in Figure 10 and Figure 11. We also present the results of our models on the Meta-Dataset in Table 4.
Statistical Results We compare the performance of different models to the Uniform Sampler. All samplers are poorer than the Uniform Sampler and are statistically significant with a confidence interval of 95%. We use the symbol † to represent the instances where the results are not statistically
significant and similar to the performance achieved by uniform Sampler. We only observe a few cases where a sampler performs significantly better than the Uniform Sampler, which we represent using the symbol ‡. To assess statistical significance, we used a paired-difference t-test, with a pvalue p = 0.05.
Comparisons with SBU Sampler Previously, we compared the performance of NDTB and SBU samplers. However, due to the possibility of unfair comparisons, limited by the number of iterations of the SBU sampler, we extend the SBU Sampler to propose two new samplers: (1) SBUunbounded sampler - This sampler is similar to the SBU sampler but is run for more iterations. However, this sampler allows the model to be trained on more classes than our traditional SBU sampler. Since this remains an unfair comparison, we use this as a higher bound/ideal performance and propose a more appropriate sampler. (2) SBU-bounded sampler - This sampler is again similar to the SBU sampler but ran for more iterations with a bounded pool of tasks, similar to the NDTB
sampler or the SBU sampler. Our results from these experiments are presented in Figure 9. Table 5 shows the same results with higher precision.
We notice that training for more iterations using samplers such as SBU-unbounded and SBUbounded leads to a boost in performance. However, we also observe that the NDTB sampler remains to perform better in most cases. With this, we maintain our previous findings. Furthermore, we also notice a peculiar behavior where the SBU-bounded sampler performs better than the SBUunbounded sampler in most cases. This finding is interesting because the SBU-unbounded sampler has access to more data, and the model being trained should lead to better representations and performance. However, we notice that repeating the tasks and fixing the number of classes achieves the same performance with little to no adverse effects.
Matching Networks : Omniglot 5-way 1-shot

heading: MetaOptNet: Omniglot 5-way 1-shot

Review summaries:
In this paper, the authors investigate the effect of task diversity in the training process of meta-learning. The findings indicate that increasing task diversity during the meta-training process does not boost performance. They evaluate the performance on four few-shot image classification datasets. 

The paper studies how the diversity of tasks in the training phase affects the performance of meta-learning algorithms. The paper finds negative evidence, which is consistent with Setlur et al. (2021). Compared with the existing work, the paper performs more extensive experiments with different meta-learning algorithms, different task samplers, and different datasets.