The Effect of diversity in Meta-Learning

1 INTRODUCTION . It is widely recognized that humans can learn new concepts based on very little supervision , i.e. , with few examples ( or ” shots ” ) , and generalize these concepts to unseen data as mentioned by Lake et al . ( 2011 ) . Recent advances in deep learning , on the other hand , have primarily relied on datasets with large amounts of labeled examples , primarily due to overfitting concerns in low data regimes . Although the development of better data augmentation and regularization techniques can alleviate these concerns , many researchers now assume that future breakthroughs in low data regimes will emerge from meta-learning , or ” learning to learn. ” Here , we study the effect of task diversity in the low data regime and its effect on various models . In this meta-learning setting , a model is trained on a handful of labeled examples at a time under the assumption that it will learn how to correctly project examples of different classes and generalize this knowledge to unseen labels at test time . Although this setting is often used to illustrate the remaining gap between human capabilities and machine learning , we could argue that the domain of meta-learning is still nascent . The domain of task selection has remained virtually unexplored in this setting . Conventional wisdom is that the performance of the model will improve as we train on more diverse tasks . To test this hypothesis to its limits , we define various task samplers which either limit task diversity by selecting a subset of overall tasks or improving task diversity using approaches such as Determinantal Point Processes ( DPPs ) proposed by Macchi ( 1975 ) . Our contributions in this work are as follows : • We show that , against conventional wisdom , task diversity does not significantly boost performance in meta-learning . Instead , limiting task diversity and repeating the same tasks over the training phase allows the model to obtain performances similar to models trained on Uniform Sampler without any adverse effects . • We also show that increasing task diversity using sophisticated samplers such as DPP or Online Hard Task Mining ( OHTM ) Samplers do not significantly boost performance . Instead , the dynamic-DPP Sampler harms the model due to the increased task diversity . • We empirically show that repeating tasks over the training phase can perform similarly to a model trained on the Uniform Sampler , achieving similar performance with only a fragment of data . This key finding questions the need to increase the support set pool to improve the model ’ s performance . 2 RELATED WORKS . Meta-learning formulations typically rely on episodic training , wherein an algorithm adapts to a task , given its support set , to minimize the loss incurred on the query set . Meta-learning methods differ in terms of the algorithms they learn , and can be broadly classified under four prominent classes : Metric-based , Model-based , Optimization-based and Bayesian-based approaches . Metricbased methods such as Koch et al . ( 2015 ) ; Vinyals et al . ( 2016 ) ; Snell et al . ( 2017 ) ; Sung et al . ( 2018 ) operate on the core idea similar to nearest neighbors algorithm and kernel density estimation . These methods are also called non-parametric approaches . Model-based methods such as Santoro et al . ( 2016 ) ; Munkhdalai & Yu ( 2017 ) depend on a model designed specifically for fast learning , which updates its parameters rapidly with a few training steps , achieved by its internal architecture or controlled by another meta-learner model . Generic deep learning models learn through backpropagation of gradients , which are neither designed to cope with a small number of training samples nor converge within a few optimization steps . To address this , Optimization-based methods such as Ravi & Larochelle ( 2016 ) ; Finn et al . ( 2017 ) ; Nichol et al . ( 2018 ) were proposed , which were better suited to learn from a small number of samples . However , all the above approaches are deterministic and are not the most suited for few-shot problems that are generally ambiguous . Hence , Bayesian-based methods such as Yoon et al . ( 2018 ) ; Requeima et al . ( 2019 ) were proposed which helped address the above issue . Although research in meta-learning models has attracted much attention recently , the effect of task diversity is virtually unexplored in the domain of meta-learning . However , task sampling and task diversity have been more extensively studied in other closely related problems such as active learning . Active learning involves selecting unlabeled data items in order to improve an existing classifier . Although most of the approaches in this domain are based on heuristics , there are few approaches to sample a batch of samples for active learning . Ravi & Larochelle ( 2018 ) proposed an approach to sample a batch of samples using a protonet as the backbone architecture . The model tries to maximize the query set , given support set and unlabeled data . Other works such as Hsu et al . ( 2018 ) proposed a framework named CACTUs , which samples tasks/examples using relatively simple task construction mechanisms such as clustering embeddings . The unsupervised representations learned via these samples lead to a good performance on various downstream human-specified tasks . Although nascent , a few recent works aim to improve meta-learning by explicitly looking at the task structure and relationships . Among these , Yin et al . ( 2019 ) proposed an approach to handle the lack of mutual exclusiveness among different tasks through an information-theoretic regularized objective . In addition , several popular meta-learning methods Lee et al . ( 2019 ) ; Snell et al . ( 2017 ) improve the meta-test performance by changing the number of ways or shots of the sampled metatraining tasks , thus increasing the complexity and diversity of the tasks . Other works such as Liu et al . ( 2020a ) proposed an approach to sample classes using class-pair-based sampling and classbased sampling . The Class-pair based Sampler selects pairs of classes that confuse the model the most . The class-based Sampler samples each class independently and does not consider the task ’ s difficulty as a whole . Our OHTM sampler is similar to the Class-pair based Sampler . Other works such as Liu et al . ( 2020b ) propose to augment the set of possible tasks by augmenting the predefined set of classes that generate the tasks with varying degrees of rotated inputs as new classes . Other works such as Setlur et al . ( 2020 ) look at the structure and diversity of tasks specifically through the lens of support set diversity , and show that , surprisingly , reducing diversity ( by fixing support set ) not only maintains—but in many cases , significantly improves—the performance of meta-learning . This experiment is very similar to our No Diversity Task Sampler if the size of the support set is equal to the number of classes per task . However , in this work , we extend their work on MetaOptNet , Protonet to many other models and a myriad of samplers to better understand task diversity in meta-learning . To the best of our knowledge , we are the first to study the effect of task diversity in meta-learning to this extent . 3 BACKGROUND . Here , we review some of the fundamental ideas required to understand our few-shot learning experiments better . 3.1 EPISODIC FEW-SHOT LEARNING . In episodic few-shot learning , an episode is represented as a K-way , N-shot classification problem where N is the number of examples per class and K is the number of unique class labels . During training , the data in each episode is provided as a support set S = { ( x1,1 , y1,1 ) , ... , ( xN , K , yN , K ) } where xi , j ∈ RD is the i-th instance of the j-th class , and yj ∈ { 0 , 1 } K is its corresponding one-hot labeling vector . Each episode aims to optimize a function f that classifies new instances provided through a ” query ” set Q , containing instances of the same class as S. This task is difficult because N is typically very small ( e , g , 1 to 10 ) . The classes change every episode . The actual test set used to evaluate a model does not contain classes seen in support sets during training . In the task-distribution view , meta-learning is a general-purpose learning algorithm that can generalize across tasks and ideally enable each new task to be learned better than the last . We can evaluate the performance of ω over a distribution of tasks p ( T ) . Here we loosely define a task to be a dataset and loss function T = { D , L } . Learning how to learn thus becomes : min ω E τ∼p ( τ ) L ( D ; ω ) ( 1 ) whereL ( D ; ω ) measures the performance of a model trained using ω on datasetD and p ( τ ) indicates the task distribution . In this experiment , we extend this setting such that we vary the task diversity in the train split to study the effects on test split , which remains to use uniform or random sampling for tasks . 3.2 DETERMINANTAL POINT PROCESSES ( DPPS ) . A DPP is a probability distribution over subsets of a ground set Y , where we assume Y = { 1 , 2 , ... , N } and N = |Y| . An L-ensemble defines a DPP using a real , symmetric , and positivedefinite matrix L indexed by the elements of Y . The probability of sampling a subset Y = A ⊆ Y can be written as : P ( Y = A ) ∝ det LA , ( 2 ) where LA : = [ Li , j ] i , j∈A is the restriction of L to the entries indexed by the elements of A . As L is a positive semi-definite , there exists a d × N matrix Ψ such that L = ΨTΨ where d ≤ N . Using this principle , we define the probability of sampling as : P ( Y = A ) ∝ det LA = Vol2 ( { Ψi } i∈A ) , ( 3 ) where the RHS is the squared volume of the parallelepiped spanned by { Ψi } i∈A . In Eq . 3 , Ψi is defined as the feature vector of element i , and each element Li , j in L is the similarity measured by dot products between elements i and j . Hence , we can verify that a DPP places higher probabilities on diverse sets because the more orthogonal the feature vectors are , the larger the volume parallelepiped spanned by the feature vector is . In this work , these feature embeddings represent class embeddings , which are derived using either a pre-trained protonet model or the model being trained as discussed in Sec . 3.3 . In a DPP , the cardinality of a sampled subset , |A| , is random in general . A k-DPP is an extension of the DPP proposed in the work of Kuhn et al . ( 2003 ) , where the cardinality of subsets are fixed as k ( i.e. , |A| =k ) . In this work , we use k-DPPs as an off-the-shelf implementation to retrieve classes that represent a task used in the meta-learning step .

Summary: The paper studies how the diversity of tasks in the training phase affects the performance of meta-learning algorithms. The paper finds negative evidence, which is consistent with Setlur et al. (2021). Compared with the existing work, the paper performs more extensive experiments with different meta-learning algorithms, different task samplers, and different datasets.