Defining Benchmarks for Continual Few-Shot Learning

1 INTRODUCTION . Two capabilities vital for an intelligent agent with finite memory are few-shot learning , the ability to learn from a handful of data-points , and continual learning , the ability to sequentially learn new tasks without forgetting previous ones . Taken individually these two areas have recently seen dramatic improvements mainly due to the introduction of proper benchmark tasks and datasets used to systematically compare different methods ( Chen et al. , 2019 ; Lesort et al. , 2019a ; Parisi et al. , 2019 ) . For the set-to-set few-shot setting ( Vinyals et al. , 2016 ) such benchmarks include Omniglot ( Lake et al. , 2015 ) , CUB-200 ( Welinder et al. , 2010 ) , Mini-ImageNet ( Vinyals et al. , 2016 ) and Tiered-ImageNet ( Ren et al. , 2018b ) . For the single-incremental-task continual setting ( Maltoni & Lomonaco , 2019 ) and the multi-task continual setting ( Zenke et al. , 2017 ; Lopez-Paz & Ranzato , 2017 ) the benchmarks include permuted/rotated-MNIST ( Zenke et al. , 2017 ; Goodfellow et al. , 2013 ) , CIFAR10/100 ( Krizhevsky et al. , 2009 ) , and CORe50 ( Lomonaco & Maltoni , 2017 ) . However , none of those benchmarks is particularly well suited for evaluating the hybrid setting of low-data sequential streams . One of the main reasons behind the scarce consideration of the liaison between the two settings is that these problems have been often treated separately and handled by two distinct communities . Historically the research on continual learning has focused on the problem of avoiding the loss of previous knowledge when new tasks are presented to the learner , known as catastrophic forgetting ( McCloskey & Cohen , 1989 ) , without paying much attention to the low-data regime . On the other hand , the research on few-shot learning has mainly focused on achieving good generalization over new tasks , without caring about possible future knowledge gain or loss . Scarce attention has been given to few-shot learning in the more practical continual learning scenario . In this paper we propose to bridge the gap between the two settings by injecting the sequential component of continual learning into the framework of few-shot learning , calling this new paradigm Continual Few-Shot Learning ( CFSL ) . CFSL can be useful to the research community as a frame- work for studying continual learning under memory constraints , and for testing meta-learning systems that are capable of continual learning . While we formally define the problem in Section 3 , a high-level diagram is shown in Figure 1 . Our main contributions can be summarized as follows : 1 . We formalize a highly general and flexible continual few-shot learning setting , taking into account recent considerations expressed in the literature . 2 . We propose a novel benchmark and a compact dataset ( SlimageNet64 ) , releasing them under an open source license . 3 . We compare recent state-of-the-art methods on our benchmark , showing how CFSL is effective in highlighting the strengths and weaknesses of those methods . 1.1 MOTIVATION AND APPLICATIONS . Consider a user in a fast changing environment who must learn from the many scenarios that are encountered . There is the significant challenge of integrating common information from very few data points in each scenario in an online fashion . The small number of data points makes this distinct from normal continual learning setting : the very high uncertainty in each scenario due to the low data volume makes adaptation more challenging and makes the commonality between scenarios even more critical . The online nature of learning makes this distinct from few-shot learning , where integration from different scenarios must be learnt without access to earlier data . These two requirements for online learning without forgetting and efficient integration under uncertainty are competing : both require memory capacity of the learner . Because of these competing requirements , it is valuable to consider continual learning and few-shot learning together rather than in isolation . For a concrete example , consider typical user interfaces such as those used in online stores . The size of data points collected from each user is rather small ( few-shot ) and is generally stored in a sequential buffer or priority queue ( continual ) . Suppose an underlying learning model has been deployed to enhance the user experience by suggesting new products that are likely to be of interest . This model should be able to rapidly adapt to each user ( task ) by accessing the sequential buffer while learning on the fly . There are multiple variants to take into account . For instance , if the user is unknown or previous data is not accessible ( e.g . under privacy policies ) the model has to rapidly infer the user preferences from a single task . On the other hand , if the user profile is known the model should retain knowledge about previous interactions without the need of being retrained from scratch . Another example arise from human-robot interaction , where most of the applications require a robot to learn online by interacting with human teachers . For instance , in a manipulation task the human can provide a few trajectories representing the first task , then the second , the third , etc . The amount of trajectories for each task is usually rather limited and the tasks are learned sequentially . The robot should retain the knowledge of all tasks encountered so far , possibly by avoiding expensive training procedures that would overload the on-board hardware . Note that neither the few-shot nor the continual setting are appropriate to deal with the aforementioned examples , since the former does not consider the sequential component and the latter does not account for the limited data size . Our CFSL formulation instead , can handle all these examples and other collateral variations , as discussed more thoroughly in Section 3 . 2 RELATED WORK . 2.1 FEW-SHOT LEARNING . Progress in few-shot learning ( FSL ) was greatly accelerated after the introduction of the episodic few-shot training ( Vinyals et al. , 2016 ) . This setting , for the first time , formalized few-shot learning as a well defined problem paving the way to the use of end-to-end differentiable algorithms that could be trained , tested , and compared . Among the first algorithms to be proposed there were metalearned solutions , which here we group into three categories : metric-learning , optimization-based , hallucination ( Chen et al. , 2019 ) . Metric-learning techniques are based on the idea of parameterizing embeddings via neural networks and then use distance metrics to match target points to support points in latent space ( Vinyals et al. , 2016 ; Edwards & Storkey , 2017 ; Snell et al. , 2017 ) . Optimization-based or gradient-based techniques are trained to perform a controlled optimization or parameter initialization to learn efficiently from a support set and generalize to a target set ( Ravi & Larochelle , 2016 ; Li et al. , 2017 ; Finn et al. , 2017 ; Antoniou et al. , 2019 ; Rusu et al. , 2019 ; Antoniou & Storkey , 2019 ) . Hallucination techniques utilize one or both the aforementioned methods in combination with a generative process to produce additional samples as a complement to the support set ( Antoniou et al. , 2017 ) . There have been a number of methods that do not clearly fall in one of the previous categories ( Santoro et al. , 2017 ; Santurkar et al. , 2018 ; Chen et al. , 2019 ) , including Bayesian approaches ( Grant et al. , 2018 ; Gordon et al. , 2019 ; Patacchiola et al. , 2019 ) . For more detail , we refer the reader to the original work as well as a survey on few-shot learning ( Chen et al. , 2019 ) . 2.2 CONTINUAL LEARNING . The problem of continual learning ( CL ) , also called life-long learning , has been considered since the beginnings of artificial intelligence and it remains an open challenge in machine learning ( Parisi et al. , 2019 ) . In standard offline supervised learning , algorithms can usually access any data point as many times as necessary during the training phase . In contrast , in CL data arrives sequentially and might only be ever seen once during the training process . Following the taxonomy of ( Maltoni & Lomonaco , 2019 ) , we group the continual learning methods into three classes : architectural , rehearsal , and regularization methods . Each category brings with it a different set of advantages and disadvantages under various resource constraints . Architectural approaches can be constrained on the amount of available RAM ( Rusu et al. , 2016 ; Mallya et al. , 2018 ; Mallya & Lazebnik , 2018 ; Lesort et al. , 2019a ) . Whereas , rehearsal strategies can become quickly bounded by the amount of available storage ( Rebuffi et al. , 2017 ; Lesort et al. , 2018 ; 2019b ) . Regularization approaches can be free from resource constraints but incur in severe issues in the way they adapt model parameters ( Kirkpatrick et al. , 2017 ; Zenke et al. , 2017 ; Lee , 2017 ; He & Jaeger , 2018 ; Mitchell et al. , 2018 ) . The mentioned strategies can often be intersected and combined to form even more powerful models ( Rebuffi et al. , 2017 ; Kemker et al. , 2018 ; Maltoni & Lomonaco , 2019 ) . Due to space constraints , we refer the reader to recent surveys on continual learning ( Lesort et al. , 2019c ; Parisi et al. , 2019 ) . 2.3 JOINING META-LEARNING AND CONTINUAL LEARNING . Attempts to combine continual-learning with meta-learning produced a set of new research areas ( Caccia et al. , 2020 ) . Continual Few-Shot Learning falls into meta continual-learning which can also be thought of as ‘ learning to continually learn ’ ( Finn et al. , 2019 ; He et al. , 2019 ; Harrison et al. , 2019 ) . In contrast , continual meta-learning refers to ‘ continually learning to learn ’ which attempts to make the process of meta-learning continuous as opposed to the standard meta-learning which is typically performed offline . Very recently Caccia et al . ( 2020 ) proposed a hybrid task , called Online faSt Adaptation and Knowledge Accumulation ( OSAKA ) , linking continual-meta learning and meta continual-learning to study continual learning in the context of non-stationarity on shifting task distributions and unknown identities . Related to continual few-shot learning is the field of incremental few-shot learning ( IFSL , ( Gidaris & Komodakis , 2018 ; Ren et al. , 2018a ) ) . In contrast to standard few-shot learning and our work , in IFSL target set is composed of ‘ novel ’ classes ( drawn from a never-seen-before dataset ) as well as classes seen during the meta-learning phase ( called ‘ base classes ’ ) , and it does not consider continual updates to the novel classes . These methods are significantly different in terms of training and testing procedures . For this reason , we will not analyze thie line of research any further . Inconsistencies in the evaluation protocol In the literature , there are no established benchmarks that integrates few-shot and continual learning . Related tasks were introduced to prove the efficacy of a given system , making such tasks very restricted in terms of what methods they are applicable on , and how many aspects they could investigate . We found that tasks and datasets vary from paper to paper , making it challenging to know the actual performance of a given algorithm in comparison to others . For instance , the method proposed by Vuorio et al . ( 2018 ) has been tested exclusively on variants of MNIST . The method of Javed & White ( 2019 ) has been tested on Omniglot and incremental sine-waves . Spigler ( 2019 ) evaluated on MNIST , Le et al . ( 2019 ) on CIFAR100 and permuted MNIST , and Beaulieu et al . ( 2020 ) on Omniglot . It is evident how the problem of continual few-shot learning is not well defined , making it challenging to benchmark and compare the performance of algorithms .

Summary: This paper proposes a benchmark for a new task called continual few-shot learning. The benchmark is based on the ImageNet dataset. Basically, the model looks at a part of the support set one after another sequentially, and it is then evaluated on the query set that contains balanced samples from each part of the support set. Under the benchmark, there are four types of challenges, which differs in how they sub-partition the support set. A suite of models have been run and evaluated, including MAML, ProtoNet and SCA. SCA is found to have the best performance, whereas ProtoNet is found to be the most resource efficient.