Defining Benchmarks for Continual Few-Shot Learning
Authors: Antreas Antoniou
Massimiliano Patacchiola
Mateusz Ochal
Amos Storkey
Venue:  (2021)
Keywords: few-shot learning
continual learning
benchmark

Abstract: In recent years there has been substantial progress in few-shot learning, where a model is trained on a small labeled dataset related to a specific task, and in continual learning, where a model has to retain knowledge acquired on a sequence of datasets. Both of these fields are different abstractions of the same real world scenario, where a learner has to adapt to limited information from different changing sources and be able to generalize in and from each of them. Combining these two paradigms, where a model is trained on several sequential few-shot tasks, and then tested on a validation set stemming from all those tasks, helps by explicitly defining the competing requirements for both efficient integration and continuity. In this paper we propose such a setting, naming it Continual Few-Shot Learning (CFSL). We first define a theoretical framework for CFSL, then we propose a range of flexible benchmarks to unify the evaluation criteria. As part of the benchmark, we introduce a compact variant of ImageNet, called SlimageNet64, which retains all original 1000 classes but only contains 200 instances of each one (a total of 200K data-points) downscaled to 64 by 64 pixels. We provide baselines for the proposed benchmarks using a number of popular few-shot and continual learning methods, exposing previously unknown strengths and weaknesses of those algorithms. The dataloader and dataset will be released with an open-source license.

Under review as a conference paper at ICLR 2021
DEFINING BENCHMARKS FOR CONTINUAL FEW-SHOT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
In recent years there has been substantial progress in few-shot learning, where a model is trained on a small labeled dataset related specific task, and in con- tinual learning, where model has to retain knowledge acquired on sequence of datasets. Both of these fields are different abstractions of the same real world scenario, where learner has to adapt to limited information from different chang- ing sources and able to generalize and from each of them. Combining these paradigms, where model is trained several sequential few-shot tasks, and then tested on a validation set stemming from all those tasks, helps by explicitly defining the competing requirements for both efficient integration and continuity. In this paper we propose such a setting, naming it Continual Few-Shot Learn- ing (CFSL.) We first define theoretical framework for CFSL, then we propose a range of flexible benchmarks to unify the evaluation criteria. As part of the bench- mark, we introduce a compact variant of ImageNet, called Slimage/Vet64,, which retains all original 1000 classes but only contains 200 instances of each one (a total of 200K data-points) downscaled to 64 64 pixels. We provide baselines for the proposed benchmarks using number of popular few-shot and continual learning methods, exposing previously unknown strengths and weaknesses of those algo- rithms. The dataloader and dataset will be released with an open-source license.
INTRODUCTION
Two capabilities vital for intelligent agent with finite memory are few-shot learning, the ability to learn from a handful of data-points, and continual learning, the ability to sequentially learn new tasks without forgetting previous ones. Taken individually these two areas have recently seen dra- matic improvements mainly due to the introduction of proper benchmark tasks and datasets used to systematically compare different methods (Chen et al. 2019 Lesort et al., 2019a Parisi et al.) 2019). For the set-to-set few-shot setting (Vinyals et al. 2016) such benchmarks include Om- niglot (Lake al.) 20155), CUB-200 /Welinder et al. 2010). Mini-ImageNet (Vinyals et al., 2016)
and Tiered-ImageNeet (Ren et al.) 2018b). For the single-incremental-aass continual setting (Mal- toni & [omonaco 2019) and the multi-task continual setting (Zenke et al., 2017 Lopez-Paz &
|anzato| (017)) the benchmarks include permuted/rotated-Miit
(Zenke et |l.| Goodfellow
al. 2013). CIFARIO/100 (Krizhevsky et al. 2009), and CORe50 (Lomonaco & Maltoni 2017).
However, none of those benchmarks is particularly well suited for evaluating the hybrid setting of
low-data sequential streams.
One of the main reasons behind the scarce consideration of the liaison between the two settings is that these problems have been often treated separately and handled by two distinct communities. Historically the research on continual learning has focused on the problem of avoiding the loss of previous knowledge when new tasks are presented to the learner, known as catastrophic forgetting (cCCloskey & Cohen 1989), without paying much attention to the low-data regime. On the other hand, the research on few-shot learning has mainly focused on achieving good generalization over new tasks, without caring about possible future knowledge gain or loss. Scarce attention has been given to few-shot learning in the more practical continual learning scenario.
In this paper we propose to bridge the gap between the two settings by injecting the sequential com- ponent of continual learning into the framework of few-shot learning, calling this new paradigm Continual Few-Shot Learning (CFSL).. CFSL can be useful the research community as a frame-

---

Under review as a conference paper at ICLR 2021
Available Model
Model Leariing
Learne
Model
Learning Phase
Evaluation Phase
Memory M,
Target
Support Set So {#3,, Ms,
Support Set S1 (as, Ms.
Support Set N
T 2t,, BY
Sy
{#5s M5
Leamer L
Model {(Mo)
Model f(Mx)
Support Set So (1rs, Us.
Support Set S1 (#s,, Ms.
Support Set (#s, Mse
Sy
Lif(TT, )(ur))
Learner f(M
proposed benchmark. Left block: from left. the learner acquires task-specific information from each set, one-by-one, without being allowed to view previous sets (memory constraint). The learner can store knowledge in a shared memory bank and use in classification model. On the rightmost side, future tasks are inaccessible the leamnrr On the bottom, the same process is repeated on the second support Note that the first support set is now inaccessible. Right block: viewed all support sets, is given an evaluation set (target set) containing new examples classes contained the support-sets, and tasked with producing predictions for those samples. The evaluation procedure has access to the target set labels, and can establish generalization measure for the model.
Figure High level overview
the learner has
work for studying continual learning under memory constraints, and for testing meta-learning sys- tems that capable of continual learning. While we formally define the problem in Section 3 high-level diagram shown in Figure ! Our main contributions can be summarized as follows:
1. We formalize a highly general and flexible continual few-shot learning setting, taking into account recent considerations expressed the literature.
2. We propose novel benchmark and a compact dataset (SlimageNet64), releasing them under open source license.
3. We compare recent state-of-the-art methods our benchmark, showing how CFSL is ef- fective highlighting the strengths and weaknesses of those methods.
1.1 MOTIVATION AND APPLICATIONS
Consider a user in a fast changing environment who must learn from the many scenarios that are encountered. There is the significant challenge of integrating common information from very few data points in each scenario in an online fashion. The small number data points makes this dis- tinct from normal continual learning setting: the very high uncertainty in each scenario due the low data volume makes adaptation more challenging and makes the commonality between scenar- ios even more critical. The online nature of learning makes this distinct from few-shot learning, where integration from different scenarios must be learnt without access to earlier data. These two requirements for online learning without forgetting and efficient integration under uncertainty competing: both require memory capacity of the leamner Because of these competing requirements, is valuable to consider continual learning and few-shot learning together rather than in isolation.
For concrete example, consider typical user interfaces such as those used in online stores. The size of data points collected from each user is rather small (few-shot) and generally stored in sequential buffer or priority queue (continual). Suppose an underlying learning model has been deployed to enhance the user experience by suggesting new products that are likely to of interest. This model should be able to rapidly adapt to each user (lask) by accessing the sequential buffer while learning on the fly. There are multiple variants to take into account. For instance, the user is unknown previous data is not accessible (e.g. under privacy policies) the model has to rapidly infer the user preferences from a single task. On the other hand, if the user profile is known the model should retain knowledge about previous interactions without the need of being retrained
2

---

Under review as a conference paper at ICLR 2021
from scratch. Another example arise from human-robod interaction, where most of the applications require a robot to learn online by interacting with human teachers. For instance, in a manipulation task the human can provide few trajectories representing the first task, then the second, the third, etc. The amount of trajectories for each task is usually rather limited and the tasks are learned sequentially. The robot should retain the knowledge of all tasks encountered far, possibly by avoiding expensive training procedures that would overload the on-board hardware.
Note that neither the few-shot nor the continual setting are appropriate to deal with the aforemen- tioned examples, since the former does not consider the sequential component and the latter does not account for the limited data size. Our CFSL formulation instead, can handle all these examples and other collateral variations, as discussed more thoroughly in Section
RELATED WORK
2.1 PEW-SHOT LEARNING
Progress in few-shot learning (FSL) was greatly accelerated after the introduction of the episodic few-shot training (Vinyals et al. 2016). This setting, for the first time, formalized few-shot learning as a well defined problem paving the way to the use of end-to-end differentiable algorithms that could be trained, tested, and compared. Among the first algorithms to be proposed there were meta- learned solutions, which here we group into three categories: metric-learning, optimization- hallucination (Chen et al., 2019). Metric-learning techniques are based the idea of parame- terizing embeddings via neural networks and then use distance metrics to match target points to support points in latent space (Vinyals et al. 2016 Edwards & Storkey 2017 Snell et al. 2017). Optimization- -based or gradient-based techniques are trained to perform controlled optimization or parameter initialization to learn efficiently from a support set and generalize a target set (Ravi & Larochelle| [2016] [i et al., (2017] Finn et al.] 2017: Antoniou et al.] (219;) Rusu et al.) 2019 Anto niou & Storkey 2019). Hallucination techniques utilize one or both the aforementioned methods in combination with generative process to produce additional samples as a complement to the support set (Antoniou et al., 2017). There have been a number of methods that do not clearly fall in one of the previous categories (Santoro et al., 2017 Santurkar et al. 2018 Chen et al.) 2019). including Bayesian approaches (Grant et al. 2018 Gordon et al., 2019; Patacchiola et al., 2019). For more detail, we refer the reader to the original work as well survey on few-shot learning (Chen al. 2019).
2.2 CONTINUAL LEARNING
The problem of continual learning (CL), also called life-long learning, has been considered since the beginnings of artificial intelligence and it remains an open challenge in machine learning (Parisi let al.) 2019). In standard offline supervised learning, algorithms can usually access any data point as many times as necessary during the training phase. In contrast, in CL. data arrives sequentially and might only be ever seen once during the training process. Following the taxonomy of (Maltoni & |Looonaco 2019), we group the continual learning methods into three classes: architectural, rehearsal, and regularization methods. Each category brings with a different set of advantages and disadvantages under various resource constraints. Architectural approaches can be constrained on the amount of available RAM (Rusu et al., 2016 Mallya et al. 2018 Mallya & Lazebnik 2018] Lesort et al. (019a)). Whereas, reheursal strategies can become quickly bounded by the amount of available storage (Rebuffi et al. 2017 Lesort et al. 20188) 2019b). Regularization approaches can be free from resource constraints but incur in severe issues in the way they adapt model parameters (Kirkpatrick et al. 2017: Zenke et al.) 2017 Lee 2017 He & Jaeger 20188] Mitchell et al. 2018).
The mentioned strategies often be intersected and combined to form even more powerful models (Rebuffi et al., 2017] Kemker et al. 2018 Maltoni & Lomonaco 2019). Due to space constraints,
refer the reader recent surveys on continual learning (Lesort al. 2019c Parisi et al. 2019).
2.3 JOINING LEARNING AND CONTINUAL LEARNING
Attempts to combine continual-leaming with meta-learning produced a set of new research areas (Caccia al. 2020). Continual Few-Shot Learning falls into meta continual-learning which can
3

---

Under review as a conference paper at ICLR 2021
also be thought of as 'learning continually learn' (Finn al. 2019 He et al. 2019 Harrison let al. 2019). In contrast, continual meta-learning refers to 'continually learning to learn' which attempts to make the process of meta-learning continuous as opposed to the standard meta-learning which is typically performed offline. Very recently Caccia et al. (020) proposed a hybrid task, called Online faSt Adaptation and Knowledge Accumulation (OSAKA), linking continual-meta learning and meta continull-carning to study continual learning in the context of non-stationarity on shifting task distributions and unknown identities. Related continual few-shot learning is the field of incremental few-shot learning (IFSL, (Gidaris & Komodakis, 2018 Ren et al., 201 Sa)). In contrast to standard few-shot learning and our work, in IFSL target set is composed of "novel" classes (drawn from sever--een-before dataset) as well as classes seen during the meta-learning phase (called 'base classes"), ) and does not consider continual updates to the novel classes. These methods are significantly different in terms of training and testing procedures. For this reason, we will not analyze thie line of research any further.
Inconsistencies in the evaluation protocol In the literature, there are no established benchmarks that integrates few-shot and continual learning. Related tasks were introduced to prove the efficacy of given system, making such tasks very restricted in terms of what methods they applicable on, and how many aspects they could investigate. We found that tasks and datasets vary from paper paper, making it challenging know the actual performance of given algorithm in comparison to others. For instance, the method proposed by |Vuorio et al. (2018) has been tested exclusively on variants of MNIST. The method of (Javed & White (2019) has been tested on Omniglot and incremental sine-waves. (Spigler (2019) evaluated on MNIST. Le et al. (019)) on CIFAR and permuted MNIST, and [eaulliu al. (2020) on Omniglot. It evident how the problem of con- tinual few-shot learning is not well defined, making it challenging to benchmark and compare the performance of algorithms.
3 CONTINUAL FEW-SHOT LEARNING
3.1 DEFINITION OF THE PROBLEM
A continual few-shot learning (CFSL) task is composed of a sequence G of small training sets No (support sets) G = (h 1. and a small evaluation set (target set) T. Each support set S Ng {(x., Un == is a set of input-label pairs just like in the standard few-shot learning setup Chen
et al. 2019h. A target set T = {(x,: yn) In-1
Nr is set of input-label pairs containing previously unseen instances of classes stemming from G. The objective of the learner is to perform well on the validation set T having only temporal access to the labeled data contained in the support S.
)
The size of the support set Ns is defined by the number of classes No (way) and by the num- ber of samples per class K (shot), such that if we have a 5-way/1-shot setup we end up with Ns = Ne X K = 5 x = 5. The Number of Support Sets Per Task (NSS) parameter deter- mines No. the cardinality of G. A Class-Change Interval (CCI) parameter dictates how often the classes within the sequence G should change, expressed in numbers of support sets. This corre- sponds to assigning the elements in the support sets to a series of disjoint class sets n C, {0). where I = N NSS/CCT) and C, is set of unique classes of size Nc. For example, CCI=2 then we will draw support sets whose classes change every 2 samples. As result, support sets S₁ and Sy will contain different instances of the same class set C1, whereas S3 and S4 will contain different in- stances from the class set Cg. The process of generating CFSL tasks is also described in Algorithm (Appendix
A learner is process which extracts task-specific information and distills it into a classification model. The model can be generically defined as a function f(x 0) parameterized by a vector of weights 0. At evaluation time the learner is tested through loss function
L = (f(xx, ().x).0).TT.
(1)
where x- and yr are the input-output pairs belonging the target set. Note that intentionally provided a definition that is generic enough to fit into different methodologies and not restricted to the use of neural networks.
4

---

Under review as a conference paper at ICLR 2021
To remove the possibility of converting continual learning task to non-continual one, intro- duce a restriction, which dictates that a support set is sampled from & without replacement, and deleted once it has been used by the learner. The learner should never have access to more than one support set at a time, and should not be able to review a support set once it has moved to the next one. This restriction induces strict sequentiality in the access of G.
The setup we have described so far is very flexible, and it allows us to define a variety of different tasks and therefore target different problems. In the following section we provide a description of those tasks and show that they are consistent with the continual learning literature.
3.2
TASK TYPES
In this section we define an empirical procedure under the form of specific task types. Our CFSL tasks fully cover the standard ingle-incremental task scenario (Lomonaco & Maltoni] (017)
while introducing an additional, super-class NI setting consistent with Domain-Incrementt. learning
vvan de Ven & Tolias 2018). Specifically, task A
B and D are equivalent to New Instances (NI),
New Classes (NC), and New Instances and Classes (NIC) (Maltoni & Lomonaco 2019), respec-
tively. Task C captures the super-set NI setting where instances are sampled across super-classes instead of being sampled from previously defined class categories. This task is most similar to
Incremental learning (van de Ven & Tolias 2018). A detailed comparison between tasks and existing work is reported in Appendix Figure showcases a high-level visual represen- tation of the proposed tasks.
A New Samples: In this task type, each support set within given task are sampled from the same set of preselected classes. As a result, each support set will share the same classes and labels but will contain previously unseen samples of those classes. To achieve this, can set CCI to be equal to (or higher than) the number of support sets in a given task (CCI 2 NSS). For every support set sample new instances from the same classes as seen in previous support sets the same task. The standard supervised regime can be considered as particular case of this task.
B New Classes: In this task type, each support set has different set of classes from the other support sets within a given task (CCI = 1). Each class within each support set has a corresponding unique output unit in the model. Class-incremental learning is particular case of this task.
C New Classes with Overwrites. This task is similar task B in that each support set has different set of unique classes (CCI= = 1). The difference is that each class in the overall task is grouped and assigned a new label by overwriting the true label. As a result the number of output units in the model is equivalent to the number of unique classes within a single support set C. Intuitively, C could be the hierarchical categories of classes in GV - however, in our experiments we assign the hierarchical categories arbitrarily. This setting emulates situations where agent is tasked with learning data-streams while being limited in storing knowledge in a preselected number of output labels.
D New Classes with New Samples: Combines tasks A and B where each support set con- tains different instances of the same set of classes for some predefined class change interval (1 <CCI<NSS). The CCI defines when the classes (and labels) change to a different and disjoint set. This setting emulates situations where agent is tasked with both learning new class descriptors and updating such descriptors by observing new class instances. sheds light on how agents can perform setting that mixes all previous settings into one.
3.3 METRICS
Test Generalization Performance. A proposed model should be evaluated on at least the test sets of Omniglot and SlimageNet, on all the tasks of interest. This is done by presenting the model with number of previously unseen continual tasks sampled from these test sets, and then using the target set metrics as the task-level generalization metrics. To obtain a measure of generalization across the whole test set the model should be evaluated number of previously unseen and unique tasks. The mean and standard deviation of both accuracy and performance should be used as generalization measures compare models.
5

---

Under review as a conference paper at ICLR 2021
Sd
[i New Sumpiee
Now ussis
Ovetanll CDI I
4.5.6
Overwittz -
0,1
New assss of Pace Sumplee
2,2
Figure 2: Visual representation of the four continual few-shot task types. Each row corresponds to task with Number of Support Sets, NSS=1, and a defined Class-Change Cnterval (CCT). Given sequence of support sets, Su, the aim is to correctly classify samples in the target set, T. Colored frames correspond to the associated support set labels.
Hultiply-Additian operations (MACs) Mcasures the computational expense of both the learner and the model operations during learning and inference time. In other words, it measures the mem- ory footprint that the model itself needs to execute during cycle of inference, or a meta-learning cycle.
Across Task Memory (ATM). Even though we have imposed a restriction on the access to G, the learner is still authorized to store in a local memory bank M some representations of the inputs and/or output vectors (often implemented as embedding vectors or inner loop parameters)
M {(x ()s)) (xx sxxa)
where & and are representations of x and obtained after a given learner has processed x and and stored some of their useful components. Most learners will be compressing a given support set, but this is not strictly the case.
The potential compression rate not directly correlated to the complexity of the model (e.g. number of parameters, FL.OPs, etc). For instance, compression can be achieved by removing some of the dimensions of the input, or by using a lossless data compression algorithm, which may not require additional parameters may have minimal impact the exccution time. In this regard, the concept of memory bank M helps to disambiguate model complexity from any additional memory allocated for compressed representations of inputs. We quantify the learner efficiency. Given two learners with their corresponding models f(x, 01) and f(x 02), and assuming that the size of 0, is equal to the size of O2 with Li L2, then the learner with smaller cardinality IM| must be preferred.
the cardinality of M, indicated as [.M). to
In order compare performances across different tasks and datasets, relate the size of the stored task-specific representations (in bytes) M (c.g. embedding vectors in ProtoNets. and inner loop parameters for MAML) during task-specific information extraction the size of vectors (in bytes) contained in the episode G UN S Recall that X is compressed version of and therefore F < H, where H and F are the vectors of dimensionality of x and x, respectively. To reduce the notation burden we have only considered the inputs x and not the targets y, since significantly larger than y. Based on these considerations we define Across-Tass -ask Memory (ATM)
L
ATM
(3)
|
where M is the stored representation of a series of support sets and G is the size of the support sets. For each utilized floating point arithmetic unit we include a computation that takes into account the floating point precision level, as stored in tensor form. For example, if both M and g" use the same floating point standard then it is divided out, but the representational form uses a lower precision than the actual data-points then it becomes compressive. From practical standpoint (image classification), the ATM can be estimated relating the total number bytes stored in the memory bank (ATM numerator) with the total number of bytes over all the images in the episode (ATM denominator). Given the definition above:ATM < 1 for learners with efficient memory, ATM 0 for learners with memory, and ATM > 1 for learners with inefficient memory. Note that the ATM is undefined for empty episodes g = {6). ATM task/dataset agnostic and can be used compare various models (or the same model) across different settings.

---

Under review as a conference paper at ICLR 2021
3.4 CHOICE OF DATASETS
The proposed CFSL benchmark has to be supplemented with appropriate datasets. The choice of these datasets should be considered decoupled from the benchmark itself. However, building some of the tasks requires datasets that meet specific desiderata: very high degree of diversity in terms of classes, high number of categories, fair (but not overabundant) number of samples per class, contained in size and memory footprint. The popular Omniglot (Lake et al., 2015) is a good first choice for a lower-difficllyy dataset. Finding a higher complexity dataset (e.g. with RGB images) is arduous since existing datasets do not satisfy all the desiderata, being insufficient for robust and extensive benchmarking of CFSL algorithms (see detailed comparison in Appendix B).
SlimageNet64. To solve this issue, we propose a new variant of ImageNet64 64 (Chrabaszez al. 2017]. named SlimageNe164 (derived from Slim and ImageNet) consisting of 200 instances from each of the 1000 object categories of the ILSVRC-2012 dataset (Krizhevsky et al.] 2012 Russakovsky et al. 2015). for a total of 200K RGB images with a resolution of 64 x 64 pixels. In Appendix B Table we report detailed comparison of all the datasets available, showing how SlimageNet64 is an optimal choice over multiple criteria.
SlimageNet664 vs Tiered-ImageNNet The closest alternative SlimageNet64 is Tiered- ImageNet (Ren et al. 2018b), a subset of ILSVRC-12 with a total of 608 classes. We have found three main issues with Tiered-ImageNet: (i) it is not provided stand-alone archive but aggregated via gen- erating script (different scripts have been released leading to an incompatibility between versions); (ii) it has just 608 classes, meaning a reduced diversity in the distribution space; (iii) has large memory footprint (29 GB size) that makes it impractical for training large number of models in a reasonable time. On the other hand, SlimageNet64 contains more classes (1000) and it has lower memory footprint (9 GB size) due to the smaller resolution of the images. This makes SlimageNet64 more compact since both dataset and model can be easily loaded on a single GPU.
EXPERIMENTS
For the purposes of establishing baselines in the CFSL tasks outlined in this paper we chose to use eight existing methods: (1) randomly initializing a convolutional neural network, and fine tuning on incoming tasks; (2) pretraining a convolutional neural network on all training set classes, then fine- tune on sequential tasks (Chen et al.) 2019); (3) Elastic Weight Consolidation (EWC, KKrkpatrick al. 2017) on a network which has been initialized and then finetuned (EWC-Init) as a baseline for CL methods; (4) EWC on network which has been pretrained and finetuned (EWC-Pretrain);
(5) Prototypical Networks (Snell et al. 2017) as baseline for metric-based FSL methods; (6) the Improved Model Agnostic Meta- Mleta-Le or MAML++ L. (Low-End model (Antoniou et al., 2019) as baseline for optimization based FSL methods; (7) MAML++ H (High-End) model (Antoniou & Storkey 2019) (dense-net backbone, squeeze excite attention, mid-tier baseline); and (8) the Self- Critique and Adapt model (SCA) (Antoniou & Storkey 2019), as a state-of-the-an algorithm for FSL (high-tier baseline). Additional details are reported in Appendix A Table I and Figure 3 show the results obtained in the experimental section.
Results: accuracy. (i) Omniglot. The results on Omniglot in the New Classes without Overwrite Setting MAML++ Low-End is inferior to ProtoNets, whilst in the New Classes with Overwrite Settings CCp this result is reversed. We infer that embedding-
methods are better at retaining information from previously seen classes, assuming that each new class remains distinct. However, when overwriting enabled this trend is overturned because ProtoNet prototypes are shared by number of super-classes containing categories that are harder to semantically disentangle. Gradi- ent based methods such as MAML++ dominate in this case, since they can update their weights towards new tasks, and therefore achieve better disentanglement of super-classes. SCA and High- End MAML++ (which utilize both embeddings and gradient- based optimization) produce the best performance across all settings. In the New Samples Setting (A), gradient based methods tend to outperform embedding- in the New Classes and Samples Setting (D), embedding methods, whilst hybrid methods continue to produce the best performing models. Regarding CL methods, we notice that the performance of EWC is close to the baseline conditions (Init & Tune, Pretrain & Tune), showing that EWC is not particularly effective in the CFSL setting. (ii) Slim- ageNet. Results SlimageNet show that ProtoNets consistently outperform Low-End MAML++
methods while bybrid methods produce the best results. Furthermore, methods outperform gradient-based

---

Under review as a conference paper at ICLR 2021
Table 1: Accuracy and standard deviation (percentage) the test set for the proposed benchmarks. Best in bold.
Book 1ype
H
P
NSS
OCT
2
5
10
Overwntte
Fulse
True
Thue
Paise
Talse
True
Thee
Talse
Palse
True
Time
Inr & Tume
21 26:5
24:02
51124on
22.10 20
Ommggot
Pro. Tune
20. 17
32 00.0
24.5
11.89
526
S:1
22 22.30133:0 09
35.1 17.00
RWWC-Ioi
10.700000
27. 70
12.55
6.77.c 0
6:25
21.19e3
12.1 12.180000
5034000
22. 22.22223:44
41 82:00
RWCC-Po
20 20.2348:0 69
$02770000
7.73 0
5:9
21.20
30 30.100
5.700000
22. 10:3003::0
301 1o000
Procoiiess
00.30
45.44
08 75
48 05
61.52
35.10
08. 75
48.44
72
27 27.38
60
MAML++l
18.18.m
66. 32.12
00.38.um
25.87.444
22 22.
30. 30.764934
90 41
M.
27 63.0
50:72180
40.44
02 712.0o
50000000
50.3813.34
027 Aagoon
SCA
97.38 am
49.27 71 (1:1
08.81 1e:0
OS
99 8.
53 m0
07 19:4)
99.88
Int. & Tune
5.4
21-32836
5:1.90
20.52000
24733.
201-1144
Pre:
8.73385
31.9.6:11
21:2
6.4.00
4:
31.32606
24.5,333
33
3.7
20.7::11
21:0
EWC-hait
8.51
21.09:800
2481
6.11:c48
4.60.0000
20.0649:0
24.870000
2:9.0000
2.38-0000
20.30::3:0
24.07.000
EWC--Pre
8:83,000
22.21
25.47
6.02
5.18.9M1
21.17g886
23.46
3.20
2.06:001
20.74:3.4
21. 40.0.a
ProooNess
20.1
20.01853
43.1
15.1,50
18.2
22.71806
43,8
10.439:2
12,
21.0:546
43. 41.7.
MAML
27
10:2
10:3001
30.4:033
05.001
12:3002
2573031
14.8.am0
26
77.4
Omniglot (CC1-1: Overarite-rrue)
Omniglot (CCI-1: OverwritsFaatee)
SimageWet (CC1-1: Ovenwite-TTuul BimageeNet/CCI-11 OvermritoFaalee
SEA
Picurecy [S]
Accunaty 143
Protiam
Protchiet
(m)
(m)
(#)
(#)
Figure 3: Accuracy (percentage) on the Omniglot and SlimageNNt test set for different values Number of
Support Sets Per Task (NSS), Class Change Interval (CCT) equal 1, and with/without overwrite.
even in New Classes with Overwrite (C) where it was previously inferior. This might indicate that in SlimageNet retaining information about previously seen tasks is more important than disentan- gling complicated super-classes. Overall models that use both embedding- based and gradient- methods, seem to outperform methods that do just one of the two. In New Classes and Samples (D), embedding-basee based methods outperform gradient-based ones by significant margin, while hybrid approaches consistently generate the best performing models. Interestingly, in New Samples (A) the embedding-basee and gradient-based methods produce very similar results, whereas in Omniglot gradient-based methods dominated.
Results: Memory (ATM) and Computational Budgets (MAC) Figure in Appendix C shows the ATM and MAC costs for NSS E [1, 640). ProtoNets are the most efficient by two orders magnitude. Other methods (e.g. Low-End MAML++) start off cheaper but increase as the number of support sets incraeses, due to accumulation of features for each new learned class. In terms of ATM it is worth noting that methods such MAML++ H and SCA tend to become incremenaally cheaper than MAML++ L as the number of support sets increases, since they optimize a smaller network in their inner loop. Whereas in terms of MACs MAML++ and SCA are the most expensive by an order of magnitude or more compared to MAML++ L and ProtoNets.
CONCLUSION
In this paper, we have introduced a novel benchmark for Continual Few-shot Learning and a new variant of ImageNet, called SlimageNet64, that contains all of 1000 ImageNet classes, but only 200 samples from each class, downscaled to 64 Furthermore, we have run experiments on the pro- posed benchmark, utilizing a number of popular few-shot learning models and baselines. We have found that embedding- based models tend to perform better when incoming tasks contain different classes from one another, whereas gradient-based methods tend to perform better when the task- classes form super-classes of randomly combined categories. Methods utilizing both embedding- based and gradient- basde methods (i.e. High-End MAML++ and SCA) outperform methods that either of the two. In conclusion, help increasing the rate of progress and the understanding of the behavior of systems trained in a continual and data-limited setting.
are confident that the proposed benchmark and dataset, will
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Antreas Antoniou and Amos Storkey. Learning to Learn by Self-Critique. Neural Information Processing Systems, NeurIPS, 2019.
Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data Augmentation Generative Adversar- ial Networks. arXiv preprint arXiv: 2017.
Antreas Antoniou, Harrison Edwards, and Amos Storkey. How train your MAML. Interna- tional Conference on Learning Representations, 2019.
Shawn Beaulieu, Lapo Frati, Thomas Miconi, Joel Lehman, Kenneth O. Stanley, Jeff Clune, and Nick Cheney. Learning to Continually Learn. arXiv preprint arXiv:2002.09577. 2020.
Massimo Caccia, Pau Rodriguez, Oleksiy Ostapenko, Fabrice Normandin, Min Lin, Lucas Caccia, Issam Laradji, Irina Rish, Alexande Lacoste, David Vazquez, and Laurent Charlin. Online fast adaptation and knowledge accumulation: new approach to continual learning. arXiv preprint arXiv:2003.05856, 2020.
Wei-Ye Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Wang, and Jia-Bin Huang. A Closer Look at Few-Shot Classification. In International Conference on Learning Representations, 2019.
Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. Downsampled Variant of Imagenet as an Alternative to the CIFAR datasets. Computing Research Repository, 2017.
H. Edwards and A.J. Storkey. Towards neural statistician. In International Conference Learning Representations, 2017.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-Agnostic Meta-Learning for Fast Adapta- tion of Deep Networks. arXiv preprint arXiv: 2017.
Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online Meta-Learning. arXiv preprint arXiv:/902..000 08438, 2019.
Spyros Gidaris and Nikos Komodakis. Dynamic Few-Shot Visual Learning without Forgetting. In Computer Vison and Putlern Recognition, 2018.
Ian Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empiri- cal investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211, 2013.
Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard Turner. Meta- learning probabilistic inference for prediction. In International Conference Learning Repre- sentations, 2019.
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient- based meta-learning as hierarchical bayes. arXiv preprint arXiv: 08930, 2018.
James Harrison, Apoorva Sharma, Chelsea Finn, and Marco Pavone. Continuous Meta-Learning without Tasks. arXiv preprint arXiv: 19Xiv:1912.08866 pp. 1-21,2011
Xu He and Herbert Jaeger. Overcoming Catastrophic Interference using Conceptor-Aided Back- propagation. International Conference on Learning Representations, 2018.
Xu He, Jakub Sygnowski, Alexandre Galashov, Andrei A. Rusu, Yee Whye Teh, and Razvan Pas- canu. Task Agnostic Continual Learning via Meta Learning. arXiv preprint 2019.
Yen-Cham Ven-Chang Hsu, Yen- Ven-Cheng Liu, Anita Ramasamy, and Zsolt Kira. Re-evaluating Continual Learn- ing Scenarios: A Categorization and Case for Strong Baselines. Continual Learning Workshop. 32nd Conference on Neural Information Processing Systems (NIPS 2018), 2018
Khurram Javed and Martha White. Meta-learning representations for continual learning. arXiv
preprint arXiv: 1905.12580 12588 2019.

---

Under review as a conference paper at ICLR 2021
Ronald Kemker, Mare McClure, Angelina Abitino, Tyler L Hayes, and Christopher Kanan. Measur- ing catastrophic forgetting in neural networks. In Association for the Advancement of Artificial Intelligence, 2018.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabssa-Barwinnkaa Barwinska, al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers Technical report, Citeseer, 2009.
of features from tiny images.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Neural Information Processing Systems, 2012.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 2015.
Canyu Lc, Xihan Wei, Biao Wang, Lei Zhang, and Zhonggui Chen. Learning Continually from Low-shot Data Stream, arXiv preprint arXiv: 10223, 2019.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/nmistf/ 1998.
Sungjin Lee. Toward continual learning for conversational agents. arXiv preprint arXi:: arXiv: 2017.
Timothée Lesort, Natalia Díaz-Rodríguez, Jean-Franois Goudou, and David Filliat. State represen- tation learning for control: An overview. Neural Networks, 2018.
Timothée Lesort, Hugo Caselles-Dupré, Michael Garcia-Ortiz, Andrei Stoian, and David Filliat. Generative models from the perspective of continual learning. International Joint Conference on Neural Networks, 2019a. 9a.
Timothée Lesort, Alexander Gepperth, Andrei Stoian, and David Filliat. Marginal replay vs condi- tional replay for continual learning. In International Conference Artificial Neural Networks. Springer, 2019b.
Timothée Lesort, Vincenzo Lomonaco, Andrei Stoian, Davide Maltoni, David Filliat, and Natalia Rodriguer Continual learning for robotics: Definition, framework, learning strategies, op- portunities and challenges. arXiv preprint 197.00190.. 2019c.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-SGD: Learning to Learn Quickly for Few Shot Learning. arXiv preprint arXiv: 177.0:1707.09835, 2017.
Vincenzo Lomonaco and Davide Maltoni. Core50: A New Dataset and Benchmark for Continuous Object Recognition. arXiv preprint arXiv:1705.03550 2017.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient Episodic Memory for Continual Learning. In Advances Neural Information Processing Systems, 2017.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. In Computer Vison and Puttern Recognition, 2018.
Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting single network to multi- ple tasks by learning to mask weights. Proceedings of the European Conference on Computer Vision, 2018.
Davide Maltoni and Vincenzo Lomonaco. Continuous learning in single-inerementallee scenarios. Neural Networks, 2019.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. Elsevier, 1989.
10

---

Under review as a conference paper at ICLR 2021
Tom Mitchell, William Cohen, Estevam Hruschka, Partha Talukdar, Bo Yang, Justin Betteridge, Andrew Carlson, B Dalvi, Matt Gardner, Bryan Kisiel, et al. Never-ending learning. Communi- cations of the ACM, 2018.
German Parisi, Ronald Kemker, Jose Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 2019.
Massimilianoo Patacchiola, Jack Turner, Elliot Crowley, and Amos Storkey. Deep kernel transfer in gaussian processes for few-shot learning. arXiv preprint arXiv:1910.05/99, 2019.
Sachin Ravi and Hugo Larochelle. Optimization as a model for Few-Shot Learning. In International Conference On Learning Representations, 2016.
Solvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. ICARL: Incremental Classifier and Representation Learning. In Computer Vison and Pattern Recognition,
2017.
Mengye Ren, Renjie Liao, Ethan Fetaya, and Richard Zemel. Incremental Few-Shot Learning with Attention Attractor Networks. arXiv preprint 1110021810.07218, 2018a.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua Tenenbaum, Hugo Larochelle, and Richard S Zemel. Meta-Learning for Semi-Supervised Few-Shot Classifi- cation. arXiv preprint 1eXXvv:8800 180.000666,.. 2018b.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet Large Scale Visual Recognition Challenge. IJCV, 2015.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive Neural Networks. arXiv preprint arXiv:1606.0467 04671, 2016.
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-Learning with Latent Embedding Optimization. International Conference On Learning Representations, 2019.
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for Relational Reasoning. In Neural Information Processing Systems, 2017.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry, How Does Batch Nor- malization Help Optimization? Neural Information Processing Systems, 2018.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical Networks for Few-Shot Learning. In Neural Information Processing Systems, 2017.
Giacomo Spigler. Meta-learnt priors slow down catastrophic forgetting in neural networks. arXiv preprint arXiv:1909.00 04070, 2019.
Gido M. van de Ven and Andreas S. Tolias. Generative replay with feedback connections general strategy for continual learning. arXiv preprint / 2018.
Oriol Vinyals, Charles Blundell, Tim Lillicrap. Daan Wierstra, al. Matching Networks for One Shot Learning. In Neural Information Processing Systems, 2016.
Risto Vuorio, Dong- Cho. Daejoong Kim, and Jiwon Kim. Meta Continual Learning, arXiv preprint arXiv: 06928, 2018.
Peter Welinder, Steve Branson. Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pictro Perona. Baltech-UCSI birds 200. 2010.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: a Novel Image Dataset for Bench- marking Machine Learning Algorithms, 2017.
11

---

Under review as a conference paper at ICLR 2021
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual Learning through Synaptic Intelli- gence. International Conference on Machine Learning, 2017.
Chen Zeno, Itay Golan, Elad Hoffer, and Daniel Soudry. Task Agnostic Continual Learning Using Online Variational Bayes, arXiv preprint arXiv:1803.10125 2018.
12

---

Under review as a conference paper at ICLR 2021
A IMPLEMENTATION DETAILS
Schedule. For each model, we used the exact configurations specified in their original papers. For each method (apart from ProtoNets) we used five inner-loop update steps. That is, Conv- 4 architecture for ProtoNets, MAML+ Low End, EWC variants as well as init + fine tune and pretrain + fine tune. In all variants except ProtoNets we adapt the full network's weights in the inner loop. In ProtoNets no inner loop optimization is carried out. For MAML++ High End and SCA we use the improved architecture detailed in [Antoniou & Storkey (2019). where there an image embedding based on a densenet that is not optimized in the inner loop, with small conv network and single linear layer which are optimized in the inner loop. For each continual learning task type, experiments each dataset. Each support set contained 1 sample from 5 classes (5-way, I-shot) while the target sets contained 5 samples from all the classes seen in a given task. We ran experiments using 3, 5 and 10 support sets for each continual task, therefore creating tasks of increasingly long number of sub-tasks. We ran each experiment 3 times, each time with different seeds for the data-provider and the model initializer. All models were trained for 250 epochs, where each epoch consisted of 500 update steps, each one done on a single continual task, using the default configuration of the Adam learning rule, and weight-decay of le-5. At the end of each training epoch we validated given model by applying it on 600 randomly sampled continual tasks, keeping those tasks consistent across all validation phases. Once all epochs have been completed, we built an ensemble of the top five models across all epochs with respect to validation accuracy, and applied that on 600 random tasks sampled from the test set, to compute the final performance metrics.
Memory Representations. We have computed the ATM for each model in section ??, and in this section we will state the exact way each model represents their ATM bank. ProtoNets represents its memory as class embeddings, called prototypes which store mean of all vectors associated with any given class, whereas init + fine tune, pretrain + fine tune, MAML+ L and EWC variants, the ATM is represented as the inner loop updatable weights which span the entirety of the Conv-4 architecture used for each model. Finally, for MAML++ H and SCA, the ATM is represented as a single convolutional layer and single linear layer, which compose the small inner loop updatable network used by the models to adapt to new continual tasks.
Datasets. For Omniglot, used the first 1200 classes for the training set, and we split the rest equally to create a validation and test set. For SlimageNet64, we used 700, 100 and 200 classes to build our training, validation and test sets respectively. The SlimageNet64 splits were chosen such that the training set had mostly living organisms, with some additional everyday tools and buildings, while the validation and test sets contained largely inanimate objects. This was done ensure sufficient domain-shift between the training and evaluation distributions. As a result this enables a more robust generalization measure to be computed.
B COMPARISON BETWEEN DATASETS
We identified four desiderata that dataset should have in order to be appropriate for CFSL. Note that, it is hard to define quantitative criteria. Here, we provide qualitative criteria that should be considered generic desiderata therefore subject a certain degree of arbitrariness.
Table 2: Dataset comparisons. Details details: number of classes in the whole dataset (#Classes), number of samples per class (#Samples), Resolution, Format, Size allocation of RAM for the whole dataset. Suitabil- ity: class diversity (Diversity), enough classes (#Classes), enough samples (#Samples), proper size (Size). Omniglot and SlimageNet64 are the best choices for the tasks on grayscale and RGB datasets, respecitively.
Dataset details
Suitabality (satisfee criteria) Diversity #Classes #Sumples Size
Dutaset
AClasses
#Samples 70XI
Resolution
Format
Size (GB)
MNIST ...in
28 28
Grayscale
-1200
Fashion KINIST
Ausneta
2017
10
7000
28
Graywale
-0.0
Omnighot (lkke d ai 20133
1622
201
28
Grayscale
00095
CUB-200 Welmskt
2010
200
20-38
~=755% +000
RGB
-13
Mini-InugäNer Vinival
600)
84384
RGB
-4.
Tloved-ImageNe [6 21
660)
84x84
RGB
-29
CIFAR-100 Knnfanyyern..
2000
100
600
32x
RGB
~0.68
/
CORc50 [Lonponzer & Malcoll 2013
50
16.55
128x 128
RGB-D
~30
ILSVRC2012 (1
2015
1000
732-1300
224x/244
RGB
-800
SllmageNettd Tours)
1000
200
64x64
RGB
-9.1
13

---

Under review as a conference paper at ICLR 2021
1. Diversity: very high degree of diversity in terms of classes. This enforces robustness in the learning procedure, since the model has to be able to deal with previously unseen class semantics. Diversity enable the training, validation, and test splits to lie within different distribution spaces, covering classes that are significantly different from one another.
2. Number of classes: high number of categories. This is to ensure that we can train models CFSL tasks ranging from 1 sub-task, all the way 100s of sub-tasks. Ideally, the length of a sub-task sequence should not be constrained by the number classes the dataset.
3. Number of samples per class: fair, but not overabundant, number of samples per class. A dataset with few samples not capture the difference in distribution within each class (poor evaluation measure), whereas having too many samples per class increases the train- ing time, producing very strong learners but neutralizing the difference among them.
4. Size: should be contained. The model should be trained in reasonable time, finances and computational resources. This requirement is crucial to allow use of the dataset by a sig- nificant portion of the research community. Here, define dataset appropriate if its size does not exceed 16 GB, which is our reasonable estimate of the average laptop RAM.
C MEMORY COST
We report the memory comparison on various datasets in terms of Multppy-Accumullan Computa- tions (MACs) and Across Task Memory (ATM). Measuring MACs is a way to quantify the inference cost of each method. Mesuring ATM is useful for two reasons. (i) We do not restrict agent to a specific amount of memory, therefore it could easily store all support sets into its memory bank. ATM distinguish models in terms of memory efficiency. (ii) Default measures of computational ca- pacity such (e.g. MACs) are not enough, since they quantify the overall computational requirements and not the actual memory shared across the learning process. This might be minuscule when com- pared to the model architecture functions which are usually orders of magnitude more expensive. ATM quantifies the efficiency of the learner compressing incoming data.
Omniglet Treu
Omniglo: talee
Omniglot Overwrite-Tree
Omniglot Overerte-ralee
VIMN Lan
MACH
Restokurt
10-0
SIA
SCA High
18
18
107
18
NSS
SlimageNat Overarita-True
Slimagehiet Overwitt-FFulse
SlimageNet Overwrita
Stimagahiet Ovarvitte-
18-11
ATM
18.5
213 400
(d)
N55
Figure 4: ATM (Across-Task Memory) and MAC (MultiplyAAcummlltc Computations) costs for a variety of NSS (Number of Support Sets Per Task). ProtoNets are the superior method across the board. In terms of ATM it is worth noting that methods such MAML+ H and SCA tend to become incrementally cheaper than MAML+ as the number of support sets increases. Whereas in terms of MAC's MAML++ the most expensive by an order magnitude or more compared to MAML++ L and ProtoNets.
SCA are
D PROPOSED TASKS AND PREVIOUS WORK
We show that our tasks are consistent with the previous work in continual learning (Lomonaco & Maltoni 2017 van de Ven & Tolias 2018 Lesort et al. 2019c). Continual learning has
14

---

Under review as a conference paper at ICLR 2021
three different scenarios (Lesort et al. 2019c): Single-Increnen Snggle--ncremenal-Tasss (SIT), Multi-Task (MT), Multi Incremental- Task (MIT). The continual learning settings for object recognition fall within scenario and can be further partitioned into three update content types (Maltoni & Lomonaco 2019): New Instances (ND), New Classes (NC), New Instances and New Classes (NIC). In parallel work, van de Ven & Tolias (2018) proposes Non-, Class-, Domain-, and Task- Incremental-Leann -eernnnng (IL) for general continual learning scenarios. This categorization also appears in later work (Hsu et al. 2018 Zeno et al. (2018). We argue that our proposed New-Samples task (A is most consistent with New Instances (Maltoni & Lomonaco 2019) but is also similar to Non-IL IL in (van de Ven & Tolias 2018) since the distribution of images sampled between support sets does not change between time steps. New-Classes without Overwrite (task B is most consistent with New-Classes (Maltoni & Lomonaco, 2019) and Class-II. (van de Ven & Tolias 2018). New- Classes with Overwrite (task|C) is not defined in (Maltoni & Lomonaco] 2019) directly but could be seen as generalization of New Instances where each temporal batch of data (in our case, support sets) is generated from super-classes with Overwrite (task CC is most similar to Domain-IL. (van de Ven & Tolias 2018) where the 'head' dimension (the of output labels) is kept the same between consecutive batches of data. Finally, our New-Classes with New-Samples task (D) is most similar to New-Classes and Samples (Maltoni & Lomonaco 2019). This task does not have equivalent task in (van de Ven & Tolias 2018 Table 3 shows a disambiguation summary of the continual learning scenarios.
the Single-Ineen -atlee-nnnneeeea-l---ae
opposed to class with static distribution. New-Classes
Table 3: Continual Learning task scenarios disambiguation.
van de Ven & Tolias 20188
Maltoni & Lomonaco (0119)
Task
Class-IL
Domain-Il.
Task-IL
STT-NI STT-NC SIT-NIC
NS
NC w/o O
B
NC w/ O
NCwNS
E
PSEUDOCODE
Algorithm 1: Sampling a Continual Few-Shot Task
Data:
Given labeled dataset DD number of support sets per task NSS, number of classes per support No. number of samples per support set class Ks, number of samples per class for target set Kr, class change interval CCI, and class overwrite parameter O
a 1,b=1:
for a S (NSS/CCT) do
Sample and remove NG classes from D; for b CCI do
n a x C
Sample Ks + Kr samples for each of Nc classes Build support Sn with Ks samples per class; Build target T, with Ky samples per class;
if TRUE then
Assign labels {1, (l,...,Nc) to the classes;
else
Assign labels (1 (a 1) x Nc..... No x a} to the classes;
end
Store sets Sn, and Ta;
end end
Combine all target sets T U:oo Tn Return (S1.. T);
15


Reviews:
[{"review_id": "u4viPbfkYTC", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The work proposes Continual Few-Shot learning -- a setting to study tasks (1) with a small labeled dataset, and (2) retain knowledge acquired on a sequence of instances.  Additionally, the authors build a compact variant of ImageNet which retains all original 1000 classes but only contains 200 instances of each one (a total of 200K data-points) downscaled to 64 × 64 pixels. \n\nBy evaluating baselines on the proposed benchmark, the authors observe that embedding-based models tend to perform better when \"incoming tasks contain different classes from one another\" and gradient-based methods tend to perform better when the task classes \"form super-classes of randomly combined categories.\" \n\nThe overall idea is interesting (few-shot + sequential observations); however, it's not clear should one take home after reading this draft. The conclusions seem intuitive and reasonable, but leave the reader with questions about the main findings of the work. \n\nI would take issue with the way the word \"continual learning\" is used. In practice, the author(s) use existing datasets where the instances arrive in a sequential manner (streaming observations). However, this is not quite what they motivate: \"Consider a user in a fast-changing environment who must learn from the many scenarios that are encountered.\" since, in the described sequential setting all the instances belong to the same underlying distribution (the original dataset), even though they're observed sequentially.  \n\nNote that a realistic temporal observations are much more challenging (e.g., the language of search queries over time because of the change in the functionality of search engines, or the changing distribution of images over time because of various social changes.)\n\nThe overall direction is promising: clearly, we need to move towards more data-efficiency and build better frameworks for measuring the generalization of our models. However, I am not convinced if the presented work is a significant step toward that goal (or, at least, I don't see it). Happy to change my mind, if I am missing anything. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "e7f96JGjpah", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n--------------------\nThe authors propose a new evaluation protocol which generalizes continual learning and few-shot learning. By controlling the 1) number of support sets per task and 2) the rate at witch tasks change, the authors can span a wide variety of settings previously explored in the literature. \nThe authors proceed to evaluate several meta learning algorithms under this new protocol and provide a detailed analysis of the results.\n\n\nPros\n-------------------\n+ The overall evaluation protocol is well explained. Figure 2 is a beautiful summary of it. \n+ The authors monitor both memory and computation. To me this is one of the key novelties of the paper : it's often overlooked in the literature, yet is very important as one can solve catastrophic forgetting given enough compute\n+ The authors release clear and concise code (and installation instructions). This is a great first step to encourage other practitioners to use this dataset\n+ The results give interesting insights into popular few-shot learning methods like MAML and ProtoNet\n\nCons\n-------------------\n- Table 1 is very hard to read. Please consider either removing some entries to make it bigger (as some baselines perform very poorly), or using another way of presenting it. \n- The evaluation protocol has very little CL baselines, I'm not sure why that is. I don't consider EWC to be a good representative of CL methods, there should at least some replay algorithm. Also please consider adding OML [1] as a baseline, as it seems appropriate to this setting. \n- The memory and MAC results are barely mentioned in the text. I understand this is hard due to space constraints, however a whole page is spent motivating it. \n\nSmall suggestion : The paper would be stronger if it had a wide range of baselines, making it also an empirical study. You constructed a nice benchmark, please show us how (more) known methods perform in it! I'm willing to increase my score if this request is met. \n\nEdit : After reading the appendix, it is mentioned that each model is trained for 250 epochs of 500 steps, where each step is done on a single continual learning task. Here a CL task is a support set (or multiple support sets) from the same distribution ? I just want to double check that a given sample is **only seen during this single continual learning task **, i.e. once this task is over the sample is discarded. Moreover I want to double check that once a task / class is seen, it is never revisited again, even when changing epochs.  I think this is the case, however my whole understanding of the paper rests on this assumption. \n\n\n[1] Javed, Khurram, and Martha White. \"Meta-learning representations for continual learning.\" Advances in Neural Information Processing Systems. 2019.", "extra_scores": {}, "extra_text": {}}, {"review_id": "dPyk9VhHK1i", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new machine learning setting called “Continual Few-Shot Learning” which fuses the up until now disparate paradigms of continual learning and few-shot learning. To evaluate methods in this new setting, a new benchmark and dataset called SlimageNet64 are defined. Various methods are evaluated on the new benchmark establishing a set of baseline results for the new setting.\n\n**Pros:**   \n- The idea of fusing the previously separate domains of few-shot learning and continual learning in the form of a new benchmark is fantastic.\n- The benchmarks include metrics for memory usage and computational complexity which are critical in practice and almost always omitted in other benchmarks.\n- The paper is well written and straightforward to understand.\n\n**Concerns:**\n\n(1) The benchmark has significant limitations:\n- The benchmark focuses on classification accuracy, memory usage, and computational complexity metrics, but has no metrics for other fundamental continual learning metrics such as positive forward transfer, positive backward transfer, forgetting over time, and intransigence (i.e. the inability of an algorithm to learn new tasks) [3,4]. These metrics would have to be measured within a task over the course of learning. The proposed benchmark only does evaluation at the end of a task, which is a limitation.\n- Section 3.1 implies that shot and way are fixed across support sets. This is a limitation as realistic few-shot and continual learning scenarios require varying shot and way, including some of the motivating scenarios in Section 1.1.\n- Certain guidance about the benchmark is not mentioned. For example, what is the guidance on pre-training the learner via meta-learning or large-scale supervised classification on large datasets outside the benchmark? For example, [5] demonstrates promising continual learning results on a few-shot variant of split MNIST and split CIFAR using a few-shot learner that has been meta-trained on meta-dataset [2].\n- The 64 x 64 pixel image size choice in the SlimageNet64 dataset is short-sighted. I realize that the intention is to keep the dataset size small, but in the era of low-end smartphones producing 5 million or more pixel images and the ready availability of pre-trained networks at 224 x 224 pixels implies that the benchmark will not have longevity. The images should be available at their natural size and the allow the learner to scale the images down if they are operating in a constrained computational environment.\n\n(2) The experiments are limited:\n- All the experiments were carried out with one fixed configuration (5-way, 1-shot). It would be more revealing if a greater number of configurations were explored, including support sets with random shot and random way.\n- The tasks consisted of a very small number of support sets (NSS between 3 and 10) in the accuracy experiments. However, the memory cost tests used NSS=640 (a more realistic number), why not report accuracy results on those runs?\n- It is peculiar to compare the accuracy of various methods that use very different network backbones in the same table without clearly delineating the differences in the table (i.e. with a horizontal line or column indicating the backbone). I realize that the purpose of these experiments is to establish baselines with various types and levels of learners but the fact that SCA is bolded as the best in the accuracy tables implies that there is a direct (and unfair) comparison being made between methods that should be made within several categories (say low, medium, and high tiers). In any case, dividing up the table into categories and adding detail on architectures used for the various methods would make the presentation of the results more transparent and insightful.\n\n(3) Much detail required for reproducibility of the experiments is missing:\n- No detail is given on how the various approaches in the experiments utilize memory within a task. Section 3.3 says: “Most learners will be compressing a given support set, but this is not strictly the case.” and “(e.g. embedding vectors in ProtoNets, and inner loop parameters for MAML)”. It would be beneficial to describe precisely how memory is used in each of the methods.\n- No details were provided on how the baseline methods were carried out. For example, how was the fine tuning done in terms of network architecture, learning rate, optimization method, number of iterations, any memory that is used, were all weights in the network optimized or just the top layer, etc. The same goes for the other methods.\n- How is $|\\mathcal{G}^x|$ measured? i.e. Is it measured in its tensor form (i.e. 4-byte per channel float) or in its native form (1 unsigned byte per channel)?\n\n**Minor Comments:**  \n- In the introduction, you should mention meta-dataset [2] along with the other few-shot learning benchmarks listed as it is arguably the most challenging/meaningful one at the moment.\n- Section 2.1 should cite [1] as a thorough few-shot learning survey.\n- Section 3.2, the acronym NI is used in the sentence before it is defined.\n- In Section 3.3, Multiply-Addition operations(MACs) paragraph, it says “…it measures the memory footprint that the model”. Should it be something like “…computational complexity of the model”?\n-  Section 3.3, 1st sentence – should it say: “all the task types of interest.” instead of “all the tasks of interest”?\n- Should the title of Figure 1 be “High level overview of a CFSL task” instead of “High level overview of the proposed benchmark”? i.e. ‘benchmark’ is a bit too general as there is no other mention of key parts of the benchmark including memory and computational complexity metrics.\n\n**References:**  \n[1] Hospedales, Timothy, et al. \"Meta-learning in neural networks: A survey.\" arXiv preprint arXiv:2004.05439 (2020).  \n[2] Triantafillou, Eleni, et al. \"Meta-dataset: A dataset of datasets for learning to learn from few examples.\" arXiv preprint arXiv:1903.03096 (2019).  \n[3] Schwarz, Jonathan, et al. \"Progress & compress: A scalable framework for continual learning.\" arXiv preprint arXiv:1805.06370 (2018).  \n[4] Chaudhry, Arslan, et al. \"Riemannian walk for incremental learning: Understanding forgetting and intransigence.\" Proceedings of the European Conference on Computer Vision (ECCV). 2018.  \n[5] Requeima, James, et al. \"Fast and flexible multi-task classification using conditional neural adaptive processes.\" Advances in Neural Information Processing Systems. 2019.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "FRzCq0rqL7v", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "----------------------------------\n**Summary**\n\nThis paper proposes a benchmark for a new task called continual few-shot learning. The benchmark is based on the ImageNet dataset. Basically, the model looks at a part of the support set one after another sequentially, and it is then evaluated on the query set that contains balanced samples from each part of the support set. Under the benchmark, there are four types of challenges, which differs in how they sub-partition the support set. A suite of models have been run and evaluated, including MAML, ProtoNet and SCA. SCA is found to have the best performance, whereas ProtoNet is found to be the most resource efficient.\n\n----------------------------------\n**Strengths**\n\n1. I agree that continual few-shot learning is a very useful way of framing few-shot learning problems, especially in the domain of general purpose robotics.\n\n2. Well defined metrics that consider both accuracy and computation resources.\n\n3. There is a significant amount of effort in terms of defining four different scenarios and coming up with another split of the ImageNet dataset and running a number of models on the benchmark.\n\n----------------------------------\n**Weaknesses**\n\n1. Limitations in the setup. \n\n    a) Although the paper motivates the application of continual few-shot learning, the proposed benchmark seems rather far away from application. The two applications mentioned in the paper are 1) online shopping for user preference and 2) human-robot interaction for task learning. Why not directly target these settings? \n\n    b) Although the paper is named for defining the benchmark for continual few-shot learning, it is mainly targeting object classification, but not learning new tasks (which is motivated in the human-robot interaction). \n\n    c) The setups in Figure 2 don't seem very natural. Why wouldn’t the model see any old classes in B? Why would the sequence completely overwrite all the class definitions in C? C doesn’t fully capture another very interesting yet unexplored area of domain adaptation, e.g. change the image style from real to synthetic images over time. I can see it is partially addressed in OSAKA between meta-training and meta-test but not in the sequence.\n\n    d) Why is the query set only happening at the very end but not in the middle of the sequence?\n\n2. A lack of new components in the benchmark. The benchmark seems very similar to previous benchmarks (e.g. class incremental learning and CORe50) with the main difference being low-data. According to Table 3, the only setting missing in CORe50 is setting C, which is a rather limited setting since it is extremely unnatural that we need to completely overwrite a label for another class (see my first comment). So what prevents people from using less number of images per class in CORe50, with a possibly pretrained representation from somewhere else, if they want to study a low-shot setting? Pretrained representation from supervised classification seems like a pretty robust approach for few-shot learning based on recent studies (Chen et al., 2019).\n\n3. From an implementation point of view, the proposed benchmark seems no different from a few newly defined mini-batch samplers for regular few-shot learning episodes. If it is no different from a few newly defined mini-batch samplers, then what prevents a model from storing all the examples in the episode? Since this is low-data, it doesn’t seem to have a huge memory cost. Let’s say for each class we store a fixed constant K exemplars, then the overall memory storage is still on the same order as ProtoNet, which stores a single mean vector per class. At the very least, the author should provide results for such a model, as an offline oracle.\n\n4. A lack of models evaluated. \n\n    a) If it can potentially include models that store some of the examples, then there seems no reason to exclude other baselines such as MatchingNet (Vinyals et al., 2016), infinite mixture prototypes (Allen et al., 2019), which can probably address the challenges that ProtoNet faces when dealing with type C sequences. \n\n    b) There have also been a bunch of incremental class learning baselines that can be covered (which doesn’t require meta-learning). For example, iCaRL (Rebuffi et al., 2017), LwF (Li & Hoiem, 2018), BiC (Wu et al., 2019), PODNet (Douillard et al., 2020). Currently I only see EWC as a pretty weak baseline in continual learning and it wasn’t entirely designed for class incremental learning.\n\n    c) I do notice that there are Pre. & Tune and EWC-Pre. baselines that use pretrained representation. This is good. However, it doesn’t explain whether their failures are due to the sequential gradient descent steps or the lack of generalizability of the representation. I suspect it is probably the first case, and have the authors considered using pretrained representation and then using ProtoNet or some other FSL methods at test time?\n\n----------------------------------\n**Minor comments**\n\n1. 64x64 seems rather small resolution. I would suggest at least 84x84 for ImageNet images.\n2. The memory utilization results should go into the main paper, along with other baselines I suggested that use some example-based storage.\n3. Section 2.3, Page 4, Line 13: thie line -> this line.\n\n----------------------------------\n**Conclusion**\n\nIn conclusion, based on the weaknesses I mentioned above, my score is 5. The paper is one step towards a more useful type of few-shot learning, and I am glad to see the field progresses towards it and I appreciate the effort that the authors have contributed. However, the way it samples new classes and the datasets that the paper study are still in a very limited sense, which prevent the paper from getting direct applications. On the other hand, it is very similar to some other incremental class learning benchmarks and it is not clear what different kinds of conclusions we can draw (e.g. different models that may work better) by using this benchmark instead of using previous ones. Therefore I am leaning towards rejection.\n\n----------------------------------\n**References**\n\n- Li, Zhizhong and Hoiem, Derek. Learning without forgetting. In ECCV 2016.\n- Vinyals, Oriol, Blundell, Charles, Lillicrap, timothy, Kavukcuoglu, Koray and Wierstra, Daan. Matching networks for one shot learning. In NIPS 2016.\n- Rebuffi, Sylvestre-Alvise, Kolesnikov, Alexander, Sperlm, Georg and Lampert, Christoph H. iCaRL: Incremental classifier and representation learning. In CVPR 2017.\n- Chen, Wei-Yu, Liu, Yen-Cheng, Kira, Zsolt, Wang, Yu-Chiang and  Huang, Jia-Bin. A closer look at few-shot classification. In ICLR 2019.\n- Wu, Yue, Chen, Yinpeng, Wang, Lijuan, Ye, Yuancheng, Liu, Zicheng, Guo, Yandong and Fu, Yun. Large scale incremental learning. In CVPR 2019.\n- Allenn, Kelsey R., Shelhamer, Evan, Shin, Hanul and Tenenbaum, Joshua B. Infinite mixture prototypes for few-shot learning. In ICML 2019.\n- Douillard, Arthur, Cord, Matthieu, Ollion, Charles, Robert, Thomas and Valle, Eduardo. PODNet: Pooled outputs distillation for small-tasks incremental learning. In ECCV 2020.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers appreciate the steps taken to combine continual learning with few-shot learning, this is an interesting intersection with many potential applications. However, the reviewers generally outlined a number of concerns with the benchmark and paper in its current form. They largely feel that this benchmark doesn’t differentiate itself well enough from other incremental learning benchmarks, nor does it experiment with a wide enough variety of settings (additional episode configurations, more FSL/CL approaches). As such, it is difficult to determine at this point what major insights can be gained from this benchmark. I understand that there is a tradeoff: computation is limited, and there is merit in keeping things simple. However, the general consensus is that more work needs to be done in order to fully realize the potential of this benchmark.

Decision: Reject