Towards Learning to Remember in Meta Learning of Sequential Domains
Authors: Zhenyi Wang
Tiehang Duan
Donglin Zhan
Changyou Chen
Venue:  (2021)
Keywords: Meta learning
Continual Learning
Sequential Domain Learning

Abstract: Meta-learning has made rapid progress in past years, with recent extensions made to avoid catastrophic forgetting in the learning process, namely continual meta learning. It is desirable to generalize the meta learner’s ability to continuously learn in sequential domains, which is largely unexplored to-date. We found through extensive empirical verification that significant improvement
is needed for current continual learning techniques to be applied in the sequential domain meta learning setting. To tackle the problem, we adapt existing dynamic learning rate adaptation techniques to meta learn both model parameters and learning rates. Adaptation on parameters ensures good generalization performance, while adaptation on learning rates is made to avoid
catastrophic forgetting of past domains. Extensive experiments on a sequence of commonly used real-domain data demonstrate the effectiveness of our proposed method, outperforming current strong baselines in continual learning. Our code is made publicly available online (anonymous)

Under review as a conference paper at ICLR 2021
TOWARDS LEARNING TO REMEMBER IN META LEARNING OF SEQUENTIAL DOMAINS
Anonymous authors Paper under double-blind review
ABSTRACT
Meta-lcarning has made rapid progress in past years, with recent extensions made to avoid catastrophic forgetting in the learning process, namely contin- ual meta learning. It is desirable to generalize the meta learner's ability to continuously learn in sequential domains, which is largely unexplored to-date. We found through extensive empirical verification that significant improvement is needed for current continual learning techniques to be applied in the se- quential domain meta learning setting. To tackle the problem, we adapt ex- isting dynamic learning rate adaptation techniques to meta learn both model parameters and learning rates. Adaptation on parameters ensures good gen- eralization performance, while adaptation on learning rates is made to avoid catastrophic forgetting of past domains. Extensive experiments on a sequence of commonly used real-domain data demonstrate the effectiveness of our pro- posed method, outperforming current strong baselines in continual learning. Our code is made publicly available online hutpoyymmoosntpppe//ggtt...o.o ICIA20210927/Secueeee ial-domallnnnnee in- wea.ttaaaaaaninn....m
https:
com/
INTRODUCTION
Humans have the ability quickly learn new skills from few examples, without erasing old skills. It is desirable for machine-learning models to adopt this capability when learning under changing contexts/domains, which are common scenarios for real-world problems. These tasks are easy for humans, yet pose challenges for current deep-learning models mainly due to the following two rea- sons: 1) Catastrophic forgetting is a well-known problem for neural networks, which are prone to drastically losing knowledge on old tasks when a domain is shifted (McCloskey & Cohen 1989;; 2) It has been long-standing challenge to make neural networks generalize quickly from a lim- ited amount of training data (Wang et al. 2020a). For example, the dialogue system can be trained on sequence of domains, (hotel booking, insurance, restaurant, car services, etc) due to the se- quential availability of dataset (Mi al.
2020 For each domain, each task is defined learning
one customer-specifie model (Lin et al. 2019). After finishing meta training, the model could be deployed the previously trained domains, the new (unseen) customers from previous domains may arrive later, they have their own (small) training data (support set) used for adapting the sequen- tially meta-learned models. After adaptation, the newly adapted model for the new customers can be deployed to make responses to the customers.
We formulate the above problem sequential domain few-shot learning, where model is required to make proper decisions based on only a few training examples while undergoing constantly chang- ing contexts/domains. It expected that adjustments to new context/domain should erase knowledge already learned from old ones. The problem consists of two key components that have been considered separately previous research: the ability to learn from limited amount of data, referred to as few-shot learning; and the ability to learn new tasks without forgetting old knowledge, known as continual learning. The two aspects have been proved to be particularly challenging for deep learning models, explored independently by extensive previous work (Finn al. 2017 Snell et al. 2017 Kirkpatrick et al.] 2017 Lopez-Paz Ranzato 2017p. However, a more challenging yet useful perspective jointly integrate the two aspects remains less explored.
Generally speaking, meta-learning targets learning from a large number of similar tasks with lim- ited number of training examples per class. Most existing works focus developing the general-

---

Under review as a conference paper at ICLR 2021
Domain
Domain
Semple
Memory
Time
Domain
backpropagation
Domain specific
Domain
Dopain shared
CNN for Domain
CNN
CNN for Domain
Paz
B12
Meta training
Domain
Eic
R.s
S/4
CNN for Domain
Shared Layerr
Figure 1: Meta-lcarning over sequential domains. Data in each domain arrive sequentially. Our model consists a domain-shared part and a domain-specific part, all consist of a few convolutional layers (and possibly fully connected layers). The comainn-harees part is shared by all domains, and each domain only owns one sub-network in the domain-specific part. Parameters (e.g., convolutional filters) in each domain-shared convolutional layer i (blue) are divided into blocks, denoted as Byo B,,..... Bin- Each block associated with one learnable learning rate when meta training each domain the network. The learning rates are updated by loss defined on the memory tasks to enforce the memorization of previous domains.
ization ability under a single context/domain (Santoro et al. 2016 Finn al. 2017
2018 Snell
et al. Ravi & Beatson 2019). Recently, it has been shown that catastrophic forgetting often occurs when transferring a meta-learning model new context (Ren et al.,
Ravi
2019 Yoon et al. 2020). Continual learning aims mitigate negative backward transfer effects on learned tasks when input distribution shift occurs during sequential context changes. Related techniques of which are currently applied mostly standard classification problems (Serrà et al.) 2018 Ebrahimi et al.) 2020b). In this paper, we generalize it to the sequential domain meta-learning setting, which seeks good generalization on unseen tasks from all domains with only limited training resources from previous domains. We term the problem sequential domain meta learning. Note this setting is different from continual few-shol learning that focuses on remembering previously learned low- resource tasks in single domain. Our setting does not aim to remember on a specific task, but rather to maintain good generalization large amount of unseen few-shot tasks from previous do- mains without catastrophic forgetting. This setting is common and fits well in dynamic real-world scenarios such as recommendation system and dialogue training system.
The domain shift arised from this setting during meta learning poses new challenges to existing continual-learnngg techniques. This is mainly due to the high variability underlying a large number of dynamically formed few-shot tasks, making it infeasible for model to explicitly remember each task. In our setting, a model is expected to remember patterns generic to domain, while neglecting noise and variance of specific few-shot lask. This ability, termed remember to generalize, allows a model to capture general patterns of a domain that repeatedly occur in batches of tasks while avoid being too sensitive specific few-shot task.
In this paper, we propose to address the aforementioned challenges by designing a dynamic learning- rate adaptation scheme for learning remember previous domains. These techniques could jointly consider gradients from multiple few-shot tasks to filter out task variance and only remember pat- terns that are generie in each domain. Our main idea is to meta learn both the model parameters and learning rates by backpropagating both a domain loss and a memory loss to adaptively update model parameters and the learning rates, respectively. Specifically, our mechanism keeps small memory of tasks from previous domains. which are then used to guide the dynamic and adaptive lcarning behaviors on different portions of the network parameters. The proposed mechanism is versatile and applicable to both the metric-basee prototypical network (Snell et al. 2017) and the gradient-base ANIL (Raghu al. 2020) meta-learning model.
Our contributions are summarized as follows:
2

---

Under review as a conference paper at ICLR 2021
We propose challenging benchmark that requires a meta learning model to sequentially learn on a sequence of domains enduring domain shift without much forgetting on previous domains.
We extend meta learning models with existing dynamic learning rate modeling techniques. This can mitigate catastrophic forgetring through meta learning both model parameters and learning rates to dynamically control the network update process. This can be seamlessly integrated into both metric-based and gradient-based meta learning approaches.
We conduct extensive experiments on multiple public datasets under different sequen- tial domain few-shot learning scenarios. We further test functionality of the dynamic learning-rate update mechanism for both metric-based and gradient-based meta-learning approaches. Comparisons are made towards a wide range of representative continual- learning techniques and models. Results demonstrate that our method outperforms strong baselines by large margin.
2 RELATED WORKS
2.1 META LEARNING
Meta learning (Schmidhuber 1993). aka, learning to learn, aims to rapidly adapt to a new task by reusing previous experience through training on large number of tasks. Meta learning can be roughly classified into the following categories: 1) Metric/E-meedding-aane -basdd approaches such as (Vinyals et al.) 2016, Snell et al.) 2017, Edwards Storkey (017). which map input data into embedding (feature) spaces with decisions made based on some distance metric in the feature space; 2) Black-box learning methods such as (Andyychowicz et al. 2016 Graves et al. [2014] Mishra et al.
2018); 3) Optimization-aaseee based methods such as (Finn et 20017] Ravi & Larochelle 2017 Li al.
2017 Antoniou & Storkey 2019). which improve gradient-based optimization algorithms or learn
to mitaalizee network parameters; and 4) Bayesian neta-learning methods such as (Ravi & Beatson
2019 Finn et al. 2018: Yoon et al.] 2018b) Grant et al.] 2018 Wang et al.) 2020b). These methods model uncertainty
used to either interprete and understand MAMI. (Grant al. 2018), 3
of meta learning models ( (Yoon et al. 20188; Finn et al. 2018 Wang et al. (0200b) 5) Memorr-- 2020). which
based meta learning (Santoro et al. 2016 Munkhdalai & Yuj 2017 Mikulik
apply additional memory component for meta learning. Online meta learning (Finn et al. 2019)
also related to us. They focus on forward transfer, i.e., achieving better performance on future task and use all the data from previous tasks to do meta learning, while our setting is significantly
different from theirs as we focus on mitigating catastrophic forgetting with only very limited access to previous domains.
Dynamically updating the learning rates for networks not new and has been explored in several contexts. Meta-SGD (Li et al.] 2017) learns the per parameter learning rates for meta learning to improve flexibility and performance. (uupa et al. 2020) use dynamic learning rates to mitigate forgetting in online continual learning. T-net (Lee & Choi 2018) learns metric in activation space, which informs the update direction and step size for task specifc learning. [Flennerhag et al. (2020) proposes warped gradient descent to meta-learns an efficiently parameterised preconditioning ma- trix to dynamically update the network. Our work extends dynamic learning rate techniques to sequential domain meta learning setting to mitigate catastrophic forgetting.
2.2 CONTINUAL LEARNING
Continual learning tackles the problem of maintaining knowledge when input distribution shift hap- pens in sequentially arriving tasks. There are different methods address this problem, including 1) retaining memory for future replay (Lopez-Paz & Ranzato 2017] Chaudhry et al. 2011aa Riemer
al. 2019, Chaudhry et al.) (019b); 2) designing tailored network architectures (Rusu et al. [016]]
Fernando et al., 1 et al., 2018a); 3) performing proper regularization during parameter
updates (Kirkpatrick et al., 2017 Zenke et al. 2017 von Oswald et al. 2019); and 4) introduc- ing Bayesian methods for model parameter inference (Nguyen et 2018 Ebrahimi et al.) 2020a).
Specifically, methods based on memory replay store representative samples from old tasks and re-
hearsal is performed during training (Lopez-Paz & Ranzato 2017 Chaudhry et al. 2019a Riemer et al. 2019). Recent research also utilizes generative models to memorize previously seen data [Lesort et al. 2019). Representatives of architecture-baae based methods include Progressive Neural Networks (Rusu et al. 2016), PathNet (Fernando et al.] (017)) Dynamically Expandable Networks
Yoon et al. 20T8a), Hard Attention Mask (HAT) (Serrà al. 2018) and PackNet (Mallya & Lazeb-
3

---

Under review as a conference paper at ICLR 2021
nik 2017). etc. These models explicitly modify network topology to preserve previous knowledge. approaches proposed in (Serrà et al. 2018) and (Mallya & Lazebnik, 2017) do not fit into this setting, as they attempt fully remember each historic task. Progressive Neural Networks (Rusu et al. 2016) guarantee zero forgetring but at the cost of growing network architectures and increasing parameters rapidly, which is unaffordable in memory-constraint cases. Eegularization-basee methods constrain the updated parameters to avoid drastic changes to previ- ously learned tasks (Kirkpatrick et al., 2017: Zenke et al., 2017 von Oswald al. (019 Ebrahimi et al. 2020c). They can restrict the capacity to meta-learning of new domains, thus it could hurt the performance on new domain. Bayesian- based methods model parameters in a probabilistic way, and then parameters are updated either based on their posterior distributions (Nguyen et al. 2018) or on their uncertainty (Ebrahimi et al., 2020a). However, in the context of meta-learning, the uncer- tainty or posterior estimation could be highly inaccurate due to the small-data setting in each task, thus hindering the performance. Recently, there are works using meta learning to improve continual learning. For example, (Javed & White 2019) proposes to learn versatile representations by explicit training towards minimizing forgetting.
The classic architecture-basse
2.3 MULTI-DOMAIN DOMAIN META LEARNING
As new research direction, multi-domain meta learning aims to achieve good generalization across multiple domains. (Triantafillou et al. 2020)) releases a dataset containing few-shot tasks from mul- tiple domains. (Tseng et al., (2000) proposes to use transformation layers to associate feature distri- butions across different domains. (Vuorio et al. 2019) proposes a tailored initialization process of model parameters based on task embedding enable proper functionality in multiple domains. It is worth noting that the aforementioned multi-domain few-shot learning methods assume data from all domains are jointly available during training. We consider more challenging setting where domain shift comes sequentially, thus a model needs to remember knowledge from previous domains.
2.4 INCREMENTAL FEW-SHOT LEARNING
Incremental few-shot learning (Gidaris & Komodakis 2018 Ren et al. 2019 Yoon al. 2020)
aims to handle new categories with limited resources while preserving knowledge on old categories. The task requires building a generalized model while preventing catastrophic forgetting, with an implicit assumption of unlimited access to the base categories. This paper, by contrast, focuses on the case where only very limited access to previous domain available. and generalization to unseen categories in previous domains is required. (Gidaris & Komodakis (2018) introduces a novel cosine similarity function to relate incremental classes with base classes. (Ren et 2019) proposes attention attractor network to regularize the learning of new categories. (Yoon et al., 2020) proposes to integrate two sets of features with one from pretrained base module and the other from a meta- trained module for a smooth adaptation increasing new classes are assumed to be in the same domain as the base classes, which different from the changing domains in our setting.
novel categories. For incremental few-shot learning, the
2.5 CONTINUAL FEW-SHOT LEARNING
Continual few-shot learning is a relatively new research topic. (Antoniou et al. 2020 proposes a general framework for various continual few-shot learning settings, and proposes benchmarks for proper performance evaluation. (Caccia et al. 2020) introduces Pontinual-MAMA for online fast adaptation to new tasks while accumulating knowledge old tasks and conducted experiments on three domains, our setting is different from theirs because they assume previous tasks can be revisited, we assume only very limited access previous domains. (Jerfel al.) 2019) considers task distribution shift within a single domain and proposes Dirichlet process mixture of hierarchical Bayesian models (DPM) to tackle the problem. Their ideas are interesting, but they assume different set of parameters for each cluster, which is not efficient in memory-limine limited setting.
3 THE PROPOSED METHOD
Problem setup Our goal is to perform meta learning on a sequence of N domains, denoted as D1, D2..... Dx. Each domain D. consists of data divided into meta-training, meta-validation and meta-testing sets, denoted as {Dir D. Diest). Only D is used for training a meta-learning model; Dr is used for validation; and Dieat is used for meta testing. Each task T D; consists of K data examples, {(= { A task is divided into T and Tlest Our goal is to sequentially meta-learn a model fo that maps inputs æ to y for each arriving domain while not forgetting too much of all previous domains. To this end, our framework consists of very limited memory of
4

---

Under review as a conference paper at ICLR 2021
Ma. which is used to store small number of training tasks from each previous domain Da- In domain pg. the parameters are adopted from that of the last domain, 0--1. Data available for training are thus D U Our goal is to update the parameter to 0g such that it achieves good performance on a domain D, by transferring useful knowledge from previous domains without degrading performance on these domains. The final performance is evaluated on the meta-test sets of current and all previous domains.
Basic idea and model architecture For effective learning without catastrophic forgetting of pre- vious domains, our basic ideas include 1) minimizing a loss on the current domain to achieve fast adaptation and good generalization; 2) adjusting the adaptation via optimizing the learning rates to force the model to remember previous domains. In our implementation, we consider a convolutional neural network (CNN) based classifier, which is supposed to evolve over sequences of domains via meta-learning adaptation. We assume all domains share the same CNN-based structure for feature extraction, while the model should also have the flexibility define domain-specific components. As a result, we divide the CNN classifier into two parts, corresponding to bottom convolutional lay- ers for feature extraction and top convolutional layers for classification. The bottom layers of the CNN are defined as domain- -hhaid layers to serve as common feature extractor for all domains. In addition, the top layers of the CNN are defined as speecfc layers, which are defined as N parallel sub-CNNs with each accessible by only one domain. The network structure is illustrated in Figure I For more flexible adaptation, we group the parameters (the convolutional filters) in each domain-shared layer into several blocks and associate each block with one learning rate. In other words, different filters will be updated with different learning rates, which are also adaptively opti- mized enforce the memorization of previous domains (specified in the following sections). The CNN structure is also reflected in Figure [] Note the learning rate in each of the Nommann-spccific layers is not optimized the layers in different domains independent with each other, making the adaptive learning-rate scheme used in the domain-shared part inapplicable.
Problem formulation Similar to (Javed & White 2019). J. we treat the Nomain-shared and domain- specific parameters separately. The meta parameters 0' (parameters of bottom convolutional layers) and its corresponding network block learning rates x are shared across all the domains. We use O denote the domain-specific parameters and AP denote the domain-specifie learning rates from domain 1. where each element O and X represent domain D2 specific parameters and its corresponding learning rates. All the learnable parameters for domain D2 are defined as 0, = (0 } and learning rates as Ag = {>*, 1A") A are dynamically updated and A.r are fixed during the meta-training process. During meta testing, A and APT are not used and updated. We use Qq.j to denote all the learnable parameters at jth iteration and X to denote all learnable learning rates the jth iteration when meta training in domain Dg. Each element of X is the learning rate for parameters inside one block, each of which consists of several filters. Note that A and 3 are the same set of learning rates, the difference is that the latter one denotes the specific
Ma denotes all the memory tasks in all the domains learning rates at one iteration. MM- U 1==1 before domain Dz--
C(T, (()) is the task-specific loss function with model parameters @ that depends on learning rates À and data from task T. JDy- (0 4.5)) (Equation 5) represents the expected loss function with respect to training task distributions in domain D, and is used for updating model parame- ters. (0 k (A,))) (Equaatin 2 is the expected loss function with respect to memory task distributions and is used for adjusting learning rates. Detailed definitions are as following.
min JM (0 (2 Ag
JmlOastx E. L(T 0 (X TEEM
(2)
e 0pstR+i(A = 010 ApjiD(i) for k = 0, 1, .... K
(3)
$(0ms4hbs Ansi Div)) = Ogith Aaa Ve, JDgr ( 1 (Anr))
(4)
JoH (0
E L(T, 0
(5)
TED
5

---

Under review as a conference paper at ICLR 2021
0g.j +. (A² is the function of X instead of Ag.5 since X is fixed during training D, for the reasons stated above. Where K is the number of iterations in domain Dg after current iteration, we use K 1 for simplicity. + denotes gradient adaptation function. Similarly, for the learning rate adaptation at domain Dy, based on the objective equation 2
an adaptation process is defined as:
A 9J++
No V A Jm (
where 17 is the hyper learning rate of the learning- update rule.
To sum up, our method is a bi-level adaptation process. The lower level optimizes on the current domain, while the upper level optimizes on previous domains for updating the block-wise learning rates. Detailed algorithms describing the meta- meeattraiiing and testing phases are given in Algorithm and 2 Appenddix A respectively.
Domain-level meta loss computation Our model contains a component to train on previ- ous tasks stored in the memory to optimize the learning rates. In practice, only a minibatch of data is used at each iteration. Note each task is in the format of training-validation pairs from one domain. If simply uses one task from one domain at each iteration, the model could overfit to this domain and may not generalize well to other domains. To alleviate this issue, we propose to compute the meta loss at two different levels, called intra-domain and inter- domain meta losses. The intra-domain meta loss is defined with tasks in the same domain to ensure good generalization in one domain. The inter-domain meta loss, by contrast, is de- fined with tasks sampled across all available do- mains in the memory encourage model adap- tation across different domains. When calculat- ing the inter-domain loss with one task from the memory, at cach iteration, we augment the data with those from tasks of other domains. In this way, the model can be trained to better gener-
Algorithm 1 Meta training.
Require: A sequence of training domain data
D D2, .2.....3 Dv; : Initial learning rates A and A to X and model parameters 0 = Oo; X is fixed during the training process.
Require: Maximum number of iterations M for each
domain.
1: M = 0
for q 0Q=11
do
3:
Randomly initialize 0
4:
for 1 to M do
5:
Sample batch of tasks T from P(D,), distribution over tasks domain D, Evaluate Try: (0a:-00.. (0.5+1
6:
E CIT, ((T,0e5t(Ag.))
TED
Perform adaptation:
0 0gJ41
Ons
Ag Ve, Jpgr (0g.s: )n,))
q then
9:
A
= Ao
q.
10:
else
11:
X
X
XS NA A3.5)
alize to other domains. Figure / illustrates our proposed method. More details can be found in Appendix
12:
end if
13:
end for
14:
M2 = {a small batch of sampled tasks fromDyf
15:
4 EXPERIMENT
16: end for
We compare method against several related strong baselines on a sequence of five domain datasets, which exhibit large domain shift, thus posing new challenges for existing methods. We evaluate the effectivenss of our method on both gradient-based and metric-based meta-learning frameworks. We also conduct ablation studies to verify the effectiveness of each component of our model. Our implementation is based on Torch- meta (Deleu al. 2019). Results are reported in terms of mean and standard deviation over three independent runs.
Implementation details For we use a four-layer CNN with 64 filters of kernel size being as shared domain feature extractor for all the domains. The last layer is defined as domain specific with 64 filters. No fully connected layers are used following existing works. For gradient-based meta- leaannen algorithms (Raghu et al. 2020), following (Antoniou et al. 2019), being as shared domain feature extractor, and one convolutional layer with 48 filters and one fully-connected layer for domain-specifie learning. Such architectures for Prototypical network and
(Protonet) (Snell et al. 2017) baselines,
three-layer CNN with 48 filters of kernel size

---

Under review as a conference paper at ICLR 2021
ANIL are widely adopted in existing works continual learning to mitigate catastrophic forgetting (Serrå et al. 2018: Ebrahimi et al. 2020b). 750 evaluation tasks from each domain are used for meta testing. The hyper learning rate 1) set to le-4 for Protonet and 5e-5 for ANIL. The gradients with respect to all the learnable learning rates are clipped to [-10, 10] for both Protonet and ANII.. Unless otherwise specified, the number of learning blocks for each CNN layer is 4, the number of memory tasks for each previous domain is 6 and the arriving order of the domain sequence is Minilmagenet, CIFARFS, Omniglot, Aircraft and CUB. More implementation details are given in Appendix [A]
Datasets The five datasets include Minimmgenett (Vinyals et al.) 2016), CIFARES (Bertinetto et al. 2019), Omniglot (Lake et al. 201 T), CUB (Welinder et al.) 2010). AIRCRAFT (Maji et a 2013). All images are resized into 84 x 84. We follow the splits in torchmet: for the Minimmaennet, CIFARFS, Omniglot and CUB datasets. For AIRCRAFT, we follow the split in (Vuorio et al. 2019). We compare all the basclines on 5-way-l-shot and 5-way-5-shot learning settings.
Evaluation metrics We adapt existing work on continual learning to use the evaluation metrics of ACC and BWT (Ebrahimi al., 2020b). ACC is defined as the average meta testing classification accuracy across all domains (the higher the better), and BWT measures the average forgetting on all the previous domains evaluated at the end of the sequential meta learning task (the lower the better).
Formaly, the ACC and BWT are defined as: ACC = * Σ AN, and BWT = - NN ON, 0i,5, where an is the meta testing accuracy on domain after meta training on domain ni aN, -ii measures the forgetting score on domain i after meta training on domain N.
Baselines For meta-learning methods, we compare our method with ANIL (Raghu et al. 2020) and Prototypical Network (Snell et al. (017). The former is a simplified version of MAML (Finn eet al. (017)) with inner loop only conducted on the final layer. For continual learning methods, we compare our method with related strong baselines, which are regularization based method such the
Elastic Weight Consolidation (Kirkpatrick et al. (017)) architecture-based methods (Hard Attention Mask (HAT) (Serrà et al., 2018) with publicly available code from the authors), Bayesian methods (Ebrabimi et al. (020b) (with the authors' implementation). Memory- based (including A-GEM Chaudhry et al. (019a), ER-Ringuuffer (Chaudhry et al. 2019b) and Meta Experience Replay (MER) (Riemer et al. (2019). We also learn all the domains jointly in a multi-domain meta-learning setting, whose performance can be considered as the upper bound for our sequential learning setting. We also provide simple baseline, which trains sequentially without using any external memory. We denote this method as "Sequential", whose performance can be considered as reference for the catastrophic forgetting of our setting. It is worth noting that the above baselines have only been used for traditional continual learning. We adapt these methods to our setting to demonstrate the ineffectiveness of these existing techniques when generalizing them to meta-learning setting.
Comparisons to baselines Table show the comparisons to the baselines we constructed for this problem setting in terms of 5-way 5-shot accuracy. Results of 5-way 1-shot classification is given in Appendix A In the table, 'N/A' means the BWT is not available since the method does not learn sequentially. Our method significantly outperforms baselines. Especially, in the Protonet-based model, the performance of our model almost matches that of the joint-training model, indicating excellent memorization of past domains. From the results of "Sequential" baseline, we see that there is significant performance drop if no external mechanism is introduced to prevent forgetting. Among the baselines, ER- EER-iinbuffeer seems to perform worse overall. We believe this is because of the repeatedly-traiie trainedd memory, which leads overfiting, making recovery of previous knowledge difficult. Memory data in our method are not directly fit into network but are only used for guiding the training of network on current domain data, thus avoiding overfitting. The inferior performance of UCB might be because the uncertainty estimation with limited data is inaccurate in the meta- learning setting. HAT is also worse because simply remembers history tasks but not domain general information. Furthermore, A-GEM restricts gradient updates on new examples in the same direction as the gradient direction in memory tasks, potentially leading to wrong update directions. In MER, the fast weight for each task in memory across different domains could vary significantly, making updates by Reptile oscillate and unstable, and thus hindering its performance.
Sensitivity to domain ordering We study the sensitivity of all methods the order of domain arrivals. We use a different domain-sequent order as: CIFARES, Minilmagenet, Aircraft, CUB and Omniglot. Results of 5-way-5-shot learning are summarized in Table 2 Appendix AA detaiis the
7

---

Under review as a conference paper at ICLR 2021
Table 1: Compare Baselines
Way 5 Shots
Way Shots
Algorithm
ACC
BWT
Algorithm
ACC
BWT
Protonet-Seee Sequental
51.74 0.11
-1.68 + 0.15
ANIL-Sequential
50.68 + 0.25
277.54
0.34
Protonet-EWC
50.38
-11.15+
ANI
-30.24
Protonet-HAT
62.58 0.21
-8.96 + 0.30
ANIL-HAT
47.02 + 0.26
0..33
Protonet-UCB
57.82 + 0.05
-13.81 + 0.09
ANIL-UCH
51.38 + 0.17
-25.96
0.23
Protonet-A-GEn
$7.38 = 0.32
22.09
0.26
ANIL-A-GEM
51.56 0.38
26.445
Protonet-ER Ringguuffe
58.25
-99.0
+ 0.36
ANIL-ER- -iiigbfeR
45.14 0.225
-5529
0.225
Protonet-MEs MER
60.79 +: 0.16
-11.96
0.15
ANIL-MER MEI
51.50 4: 0.23
-25.67
0.26
Protonet-Ours
68.72 + 0.22
3.22
+ 0.17
ANIL-Ours
56.62 + 0.32
15.28
0.40
Seint-training
Joint- tint-trainnig
N/A
results of 5-way-1-shie shot setting. can be seen that although there are some performance differences compared to those of the previous order, our method still outperforms the baselines.
Table Compare Baselines with different domain ordering
Way Shots
Way 5 Shots
Algorithm
ACC
BWT
Algorithm
ACC
BWT
Protonet-Sequeniie
51.28 + 0.31
-28.79
+ 0.39
ANIL-SSqqeetial
41.18 0.39
33.77
0.35
Protonet-EWC
59.12 1.3 0.35
-1487
0.27
ANIL-EWC
46.27 0.37
-32.24
Protonet-HAT
60.61 0.28
0.15
ANIL-HAT
0.15
-34.79
0.31
Protonet-UGG UCB
50.36 + 0.18
20.85
0.23
ANIL-UUCB
45.32 +.322++0225
-31.28
0.36
Protonct-A-6e
59.75 1
0.33
ANIL-A-GEM
4..87
32,01 0:41
Ringhuffer
HR-Ringhuffeer
-3.82
Protonet-MEE
62.71 -: 0.18
-11.23
ANIL--MER
-.219
-8..45
0.29
Protonct-Ours
67.29 0.32
-.112
0.30
ANIL--Our
55.89 -:: 0.22
12.36 +
0.31
Joint- traintrrainng
71.81 0.29
N/A
Joint training
73.52 0.20
N/A
We use another different domain-sequence order as: Omniglot, Aircraft, CUB, CIFARFS and Mini- Imagenet. Since Omniglot is used as first domain, the model cannot learn good representations for the following domains, thus poses more challenges than the other two domain sequences. Table shows the results, where the baseline Protonet-fixfirs is the method that freeze the model parameters after finishing training on the first domain. For this baseline, there small BWT, this is because of random testing task variance at different time. In this case, all the baselines and our methods performance drop significantly compared the sequence with CIFARFS or Minimmaennet as first domain. It indicates that this domain sequence is significantly more challenging than the other two domain sequences.
Effeet of domain-level meta loss domain-level meta loss in Section domain meta loss, which only considers task data one domain. The results are shown in Table 4 It is clear that our proposed domain- meta loss consistently outperforms the intra-domain meta loss by obtaining better accuracy.
This sets of experiments verify the effectiveness of our proposed We compare the domain-level meta loss with the standard intra-
Table Compare Baselines with Omniglot as first training domain
Way Shot ACC
Way 5 Shots
Algorithm
ACC
42.69 + 0.26
-22.18 + 0.32
57.92 1 0.21
-18.32 =: 0.19
Protonet-EWe
37.98 + 0.19
-19.43 4 0.22
54.02 + 0.18
-13.24 0.23
Protonet-HH
41.02 + 0.12
-18.55
+ 0.17
58.35 + 0.21
-16.14
Protonet-UCK
41.32 + 0.20
-18.01 + 0.32
58.73 0.28
-17.28
Protonet-A A-GMM
43.18 + 0.27
-21.10
59.96 + 0.16
-14.66
Protonet-FRR Protonet--R-RRngguuie F.R-Rinbuuffee
41.92 + 0.28
-22.54
0.29
59.35 + 0.23
16.96+0.20
Protonet-MER
39.60 + 0.28
-4.21
58.10 0.22
16..66
Protonet- fixfirst
44.29 0.19
-0.16
56.29 0.29
-0.22 0.12
Protonet- Ours
46.56 +
-4..44
0.25
62.68 0.15
11.12 0.25
-rat-training
53.92 + 0.36
N/A
71.81 + 0.29
N/A
8

---

Under review as a conference paper at ICLR 2021
Table 4: Effect of domain-level meta loss
5 Way Shot
5 Way Shots
Algorithm
ACC
BWT
ACC
BWT
Protonet-inte and inter meta loss
53.36 +: 0.27
-3.39 + 0.15
68.72 = 0.22
-3.22
0.17
Protonct-intra loss
52.29 0.31
-4.02 = 0.18
67.69 = 0.21
33.97
0.23
iN and meta
0.22
-10.19
56.62 0.32
-15.28
0..40
ANIL-istra loss
44.16 + 0.30
-11.38 0.20
55.10 = 0.19
-1..97 0.33
Table 5: Compare to our method with different number of learning blocks
$ Way Shot ACC
5 Way Shots ACC
Algorithm
BWT
BWT
Protonet-Oue Ours- 2block
50.12 -: 0.19
0.11
65.25 + 0.23
-6.41 0.29
53.36 + 0.27
-3.39
0.15
68.72 + 0.22
-3.22 + 0.17
Protonet-Oure Ours-8hlock Shlock
53.51 0.33
-3.21 + +.55
68.82 0.33
-3.01 + 0.21
Table 6: Effect of number of memory tasks. Top, memory with one task for each domain; Bottom, memory with six tasks for each domain
Way Shot
5 Way Shots ACC
Algorithm
ACC
BWT
BWT
Protonet-A Montonet-A-GEM A-GEM
38.50 0.09
29.37
+ 0.17
55.22 0.11
-24.01 + 0.12
Protonet- HR- Ringhuffer
39.41 + 0.16
-27.41
0.26
56.17 + 0.28
-23.46 + 01.25
Protenet-MER
39.04
-2..73
0.25
58.37 0.24
-17.52 0.31
Protonet-Ours
5.17
-5.56
0.32
06.92
-1.96 0.30
Protonet-A -EG
40.07 0.400
266.47
57.38 0.32
-22.99
Protonet-ER- Beennet-ERRRigguuue Ringbuffer
11.12 0.19
-1.83
58.25 =
19.00
0.36
Protonet-MER
46.72 0.25
-14.82
0.11
60.79 0.16
-11.96 +: 0.15
Protonet-Ours
53.36 + 0.27
33.39
= 0.15
68.72
0.22
3.22 + 0.17
Effeet of number of learning block Our method assigns a different learning rate for each block of the network parameters. A finer grained division of learning blocks allows more flexible control, but leads to higher computation cost. In this experiment, we investigate the impact of blocks numbers to model performance. Specifically, shown in Table 55 It can be seen that when the block number is relatively large (4 in our case), the accuracies relatively stable. Thus
implement each layer with 2, 4 and 8 blocks. Results are
simply set the block number to 4 in practice.
Effect of a small memory size We compare performance differences when the number of tasks stored in the memory for each domain is small. Table the results of I and 6 tasks from each domain. It is interesting to see that with a small memory to store only one task from each domain, performances of all the memory-based baselines drop significantly, while our method maintains relatively stable performance. This might be due to the fact that with a smaller memory size, all memory- baased baselines either overfit to the small memory or are impacted by unstable gradient direction updates, which would not happen in our method.
5 CONCLUSION
In this paper, we propose a challenging Benchmark that requires a meta learning model to sequen- tially meta learn a sequence of domains with domain shift but without much forgetting previ- ous domains. Then, we extend existing dynamic learning rate techniques to existing meta learning model to meta leam the learning rates meta parameters blocks, which can be seamlessly inte- grated into both metric-based and gradient-based meta learning approaches to mitigate catastrophic forgetting. The adaptation on parameters maintains generalization performance on current domain, while adaptation on learning rates is made remember knowledge of past domains. Our proposed method significantly outperforms existing peatinual-learning techniques adapted to our problem, achieving significantly better performance than strong baselines. There are several possible future directions to pursued for future work, such as the cases of imbalanced classes in each task, scaling to longer domain sequences, and fewer training tasks in each arrival domain.
REFERENCES
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W. Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando de Freitas Learning to learn by gradient descent by gradient descent. Advances in Neural Information Processing Systems, 2016.

---

Under review as a conference paper at ICLR 2021
Antreas Antoniou and Amos J. Storkey. Learning to learn via self-critique. ArXiv, abs/1905.10295, 2019.
Antreas Antoniou, Harrison Edwards, and Amos Storkey. How to train your maml. International Conference on Learning Representations, 2019.
Antreas Antoniou, Massimiliano Patacchiola, Mateusz Ochal, and Amos Storkey. Defining bench- marks for continual few-shot learning. https:/arrii..rrabb220044441 119pp// 2020.
Atilim Gunes Baydin, Robert Cornish, David Martinez Rubio, Mark Schmidt, and Frank Wood.
Online learning rate adaptation with hypergradient descent. International Conference on Learning Representations, 2018
Luca Bertinetto, João F, Henriques, Philip H. S. Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. International Conference on Learning Representations, 2019.
Massimo Caccia, P. Rodríguez, O. Ostapenko, Fabrice Normandin, Min Lin, L. Caccia, Issam H. Laradji, Rish, Alexande Lacoste, D. Vázquez, and Laurent Charlin. Online fast adaptation and knowledge accumulation: new approach to continual learning. ArXiv, 0bs/2003.05856 2020.
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. Proceedings of the International Conference on Learning Repre- sentations, 2019 a..
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K. Dokania, Philip H. S. Torr, and Mare' Aurelio Ranzato. Continual learning with tiny episodic memories. http:://artivarr/aa///90.0.00 10486, 2019b.
Tristan Deleu, Tobias Würfl, Mandana Samici, Joseph Paul Cohen, and Yoshua Bengio. Torch- meta: A Meta-Learning library for PyTorch. 2019. URL TtRLlhttss://aaxiiii :ttpps:///axxii wwt.xxiv.or//bb////// abs11999. Available at: http://iwwb..ommtttttadddee/////
06576
Michele Donini, Luca Franceschi, Orchid Majumder, Massimiliano Pontil, and Paolo Frasconi. Marthe: Scheduling the learning rate via online hypergradients. Proceedings of the 29th Interna- tional Joint Conference on Artificial Intelligence and the 17th Pacific Rim International Confer- ence on Artificial Intelligence, 2020.
Sayna Ebrahimi, Mohamed Elhoseiny, Trevor Darrell, and Mareus Rohrbach. Uncertainty-guided continual learning with bayesian neural networks. International Conference on Learning Repre- sentations, 2020a.
Sayna Ebrahimi, Mohamed Elhoseiny, Trevor Darrell, and Marcus Rohrbach. Uncertainty-guicc continual learning with bayesian neural networks. Proceedings of the International Conference on Learning Representations, 2020b.
Sayna Ebrahimi, Franziska Meier, Roberto Calandra, Trevor Darrell, and Marcus Rohrbach. Adver- sarial continual learning. https//arri..or//a//000....05 2020c.
H. Edwards and A. Storkey. Towards a neural statistician. ArXiv, abs/1 006600008.. 185, 2017.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols. David Ha, Andrei A. Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. https:/arnth.orggas///0....0777 2017.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta- leat-lea for fast adaptation of deep networks. International Conference Machine Learning, 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic aogeel-agnostcc meta-lcarning, Advances in Neural Information Processing Systems, 2018.
Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta-learning Inter- national Conference Machine Learning, 2019.
10

---

Under review as a conference paper at ICLR 2021
Sebastian Flennerhag, Andrei A. Rusu, Razvan Pascanu, Francesco Visin, Hujun Yin, and Raia Hadsell. Meta-learning with warped gradient descent. International Conference on Learning Representations, 2020.
Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. Forward and reverse gradient-based hyperparameter optimization. Proceedings of the 34th International Conference on Machine Learning, 2017.
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. https///rriivorr/aa/////4/4 (ttpa/bbb//800004555.5 2018.
E. Grant, Chelsea Finn, S. Levine, Trevor Darrell, and T. Griffiths. Recasting gradient-based meta- learning as hierarchical bayes. ArXiv,
2018.
Alex Graves, Greg Wayne, and Ivo Danihelka. hutps:/aurii.orrzaab///H00000 2014.
Neural turing machines.
Gunshi Gupta, Karmesh Yadav, and Liam Paull. La-maml: Look-ahead meta learning for continual learning. Advances in Neural Information Processing Systems, 2020.
Khurram Javed and Martha White. Meta-leaming representations for continual learn- ing. In Advances in Neural Information Processing Systems 32, pp. 1820-18800 Curran Associates, Inc., 2019. URL http::/ppaers. nips. wttpscccpppprrr 8458 meta eearringnggg.................... representations cont - pdf
Ghassen Jerfel, Erin Grant, Thomas Griffiths, and Katherine Heller. Reconciling meta-learning and continual learning with online mixtures of tasks. Advances in Neural Information Processing Systems, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 2014.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Barrinska, Demis Hass- abis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 2017.
Brenden Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua Tenenbaum. One shot learning of simple visual concepts. Conference of the Cognitive Science Sociery, 2011.
Yoonho Lee and Seungjin Choi. Gradient-based meta-learning with learned layerwise metric and subspace. International Conference on Machine Learning, 2018.
Timothée Lesort, A. Gepperth, A. Stoian, and David Filliat. Marginal replay vs conditional replay for continual learning. In ICANN, 2019.
Zhenguo Li, Fengwei Zhou, F. Chen, and H. Li. Meta-sgd: Learning learn quickly for few shot learning. ArXiv, abs/ 10s//177.000 707309835, 2017.
Zhaojiang Lin, Andrea Madotto, Chien-Sheng Wu, and Pascale Fung. Personalizing dialogue agents via meta-learning. Annual Meeting of the Association for Computational Linguistics, 2019.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in Neural Information Processing Systems, 2017.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthe Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. hutps:/arri..og//aab///./555555 2013.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. http:://ari..org//////1111.13
2017.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology learning and motivation, 1989.
11

---

Under review as a conference paper at ICLR 2021
Fei Mi, Liangwei Chen, Mengjie Zhao, Minlie Huang, and Boi Faltings. Continual learning for natural language generation in task-oriented dialog systems. The 2020 Conference on Empirical Methods in Natural Language Processing. 2020.
Vladimir Mikulik, Grégoire Delétang, Tom McGrath, Tim Genewein, Miljan Martic, Shane Legg. and Pedro A. Ortega. Meta-trained agents implement bayes-optimal agents. 34th Conference Neural Information Processing Systems, 2020.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. International Conference on Learning Representations, 2018.
Tsendsuren Munkhdalai and Hong Yu. Meta networks. Proceedings of the 34th International Con- ference Machine Leurning, 2017.
Cuong V. Nguyen, Yingzhen Li, Thang D. Bui, and Richard E. Turner. Variational continual learn- ing. Proceedings of the International Conference Learning Representations, 2018.
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning feature reuse? towards understanding the effectiveness of maml. International Conference on Learning Repre- sentations, 2020.
S. Ravi and H. Larochelle. Optimization as a model for few-shot learning. International Confer- ence on Learning Representations, 2017.
Sachin Ravi and Alex Beatson. Amortized bayesian meta-learning. International Conference on Learning Representations, 2019.
Mengye Ren, Renjie Liao, Ethan Fetaya, and Richard S. Zemel. Incremental few-shot learning with attention attractor networks. Advances in Neural Information Processing Systems, 2019.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interfer- ence. International Conference on Learning Representations, 2019
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. hutps://arxivorraabb///666.....0 1, 2016.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with memory-augmenter neural networks. Proceedings of the 34th International Con- ference Machine Learning, 2016.
Schmidhuber. neural network that embeds its own meta-levels. IEEE International Conference on Neural Networks, 1993.
Joan Serrà, Dídac Surís. Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. Proceedings of the 35th International Conference on Machine Learning 2018
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning- Advances in Neural Information Processing Systems, 2017.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evei, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoin Manzagol, and Hugo Larochelle. Meta- dataset: dataset of datasets for learning to learn from few examples. Proceedings of the Inter- national Conference on Learning Representations, 2020.
Hung- Yu Tseng, Hing Lee, Jia-Bin Huang, and Ming-Hsuan Yang. Cross-domain few-shot classification via learned feature-wise transformation. Proceedings of the International Confer- ence on Learning Representations, 2020.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. http:://arzivoor//////// http/////......00..ppp/o/0 2016.
12

---

Under review as a conference paper at ICLR 2021
Johannes von Oswald, Christian Henning, João Sacramento, and Benjamin F. Grewe. Continual learning with hypernetworks. https///anxin..rg/b////990... 00695, 2019.
Risto Vuorio, Shao- Hua Sun, Hexiang Hu, and Joseph J. Lim. Multimodal model-agnostic meta-
learning via task-aware modulation. Proceedings of the Advances Neural Information Process- ing Systems, 2019.
Yaqing Wang, Quanming Yao, James Kwok, and Lionel M. Ni. Generalizing from few examples: A survey on few-shot learning. 2020a.
Zhenyi Wang, Yang Zhao, Ping Yu, Ruiyi Zhang, and Changyou Chen. Bayesian meta sampling for fast uncertainty adaptation. International Conference on Learning Representations, 2020b.
P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Bellongie, and P. Perona. Caltech-UCSD Birds 200. 2010.
Jaehong Yoon, Eunho Yang. Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. International Conference on Learning Representations, 2018a. 8a.
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. Advances in Neural Information Processing Systems, 2018b.
Sung Whan Yoon, Do-Yeon Kim, Jun Seo, and Jaekyun Moon. Xtarnet: Learning to extract task- adaptive representation for incremental few-shot learning. International Conference Machine Learning. 2020.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. htpp://arri.orrgaab/777000000,0 2017.
A APPENDIX
Joint-training details We describe details of ANIL and Prototypical Network, the two base mod- els tested in our framework.
The meta batch size for each domain is set to 3. For 1-shot and 5-shot learning, the number of training data for each task are set to and 5, and the number of query data is 15 for both cases. The number of inner loop update steps is with learning rate of 1e-2. The outer-loop learning rate is le-3, and the number of meta training iterations is 20000 with the Adam optimizer (Kingma & Ba 2014).
For ANIL:
For Prototypical Network: The meta batch size for each domain is 3, and the number of meta training iterations is 40000. For 1 shot learning, the number of support data for each task is the number of query data for each task is 10. For 5 shot learning, the number of support data for each task is 5, the number of query data for each task 10. The meta training loss is optimized by Adam (Kingma & Ba 2014), and the learning rate is le-3.
Sequential domain training details
For ANIL and Prototypical Network, we adopt the following hyperparameter.
ANIL (Raghu et al. 2020
The meta batch size for each domain is 3. The total number of meta training iterations is 40000. The number of meta training iterations for each domain is 8000. For 1-shot and 5-shot learning, the number of training data for each task are set to and 5, and the number of query data 15 for both cases. The number of inner loop update steps is the inner loop learning rate is le-2, the outer loop learning rate is le-3, the outer loop meta training loss is optimized by Adam (Kingma & Ba 2014).
Prototypical Network (Snell et al.) 2017)
the meta batch size for each domain is 3, and the number of meta training iterations is 40000. The number of meta training iterations for each domain is 8000. For 1-shot and
13

---

Under review as a conference paper at ICLR 2021
5-shot learning, the number of training data for each task set to and 5, and the number query data is 10 for both cases. The meta training loss is optimized by Adam (Kingmal & Ba 2014), and the learning rate is 1e-3.
For continual learning method, we adopt the following hyperparameter
HAT details (Serrà et al. 2018
This baseline is adapted from authors implementation The output h; of the units in layer is element-wise multiplied by the following: h aj @ hy, where aj is the annealed version of the single layer gated domain embedding ef. defined
af o(se))
Other hyperparameter are following (Serrà et al. 2018), the stability parameter or scaling parameter is annealed according to
1
+ (Smaz
(8)
8man
Smma
B
where Smar 400, 6 is 1, ..... ,B is batch index and B is the total number of batches.
EWC details (Kirkpatrick et al. 2017)
We vary the weight penalty À for avoiding drastic change parameters with x 10 5 x 10 1 x 10 x 102, 1 x 10 x 10 and select the best as our baseline.
A-GEM details (Chaudhry al. 2019a
The implementation is based or
MER details (Riemer et al. 2019)
The implementation is based or The within batch meta-learning rate 3 is 0.03,
across batch meta-learning rate y to 1.0, and batches per example to
UCB details (Ebrahimi al. 2020a)
Following (Ebrahimi et al. 2020a), we scale the learning rate of ju and p for each parameter proportional to its importance $2, which measured by variance, to reduce changes in important parameters and adapt authors implementation" We use 10 Monte Carlo samples at each iteration as in (Ebrahimi et al. 2020a).
Ringiufru details (Chaudhry et al. 2019b)
We use fixed amount of memory of each previous domain to jointly train with current domain.
Other meta learning related settings are the same as those in sequential domain meta training as described above.
Domain-level meta loss In this part, we will elaborate on the computation of proposed domain- level meta loss. To alleviate the overfitting issue of specific domain during the learning process, propose to compute the meta loss at two different separated levels, intra-domain and inter-domain meta losses. The intra-domain meta loss is defined with tasks in the same domain ensure good generalization in one domain. The inter-domain meta loss, by contrast, is defined with tasks sampled across all available domains in the memory encourage model adaptation across different domains. When calculating the inter-domain loss with one task from the memory, at each iteration, we aug- ment the data with those from tasks of other domains. In this way, the model can be trained to better generalize to other domains.
1 wttp:/itwww..ecoo/aaaaaaaaeeee/e
$ "nttp://githubb.c https//ggihhhh.cm//syyaaaaaaaan
14

---

Under review as a conference paper at ICLR 2021
sampled from 1... K
domain m
trainn
We use St to denote the examples labeled with for domain P. and S to denote the examples labeled with class for domain q. Suppose P-way K-shot learning is performed in both domain and domain At each iteration, the two tasks from both domains combined together, and the problem be- comes 2/-way K -shho learning. That is, the network is used for 2P class classification problem for images in both domain p and domain
task
val1
train
traii22
domain
Sample look
val2
task
train2
va12
trainK
domain K
valk
Define the prototype for class in a domain as
Meta meroory losss
Mernory interaction
Σ
fo(x,)) u {0, 1, ... ,2P ,P 1} 99)
The meta memory loss mechanism. For one task in do- main g from the memory pool we randomly sample the data from some tasks of other domains and combine them with their own data to calculate the task-specific loss.
Figure 2:
S"
(E5"
This will encourage the network to differentiate the image classes across two different domains and help the network gen- eralize across different domains.
For given query data æ, the goal is to model
exp(- exp(-d(fo(x))c")
Pe u/z) =
(10)
2P-1
Σ ezppp cxp(-d(fo(x).c"))
The memory loss is defined as minimizing the log probability J(0) log u/æ), where d distance function.
For MAML (ANIL), let Diat UD! denote the validation data from both domain q and p. The loss defined as
arg min metaa (,) Deoa UD ạ ( arg min trask (0, 1,0,01000 D'r))
(11)
(12)
Both mechanisms for the prototypical network and MAML (ANIL) can avoid overfitting and gener- alize across different domains.
On learning-rate optimization Learning rate is one of the most important and effective hyper- parameter standard neural-network optimization (Baydin et al.) 2018 Franceschi et al. 2017] Donini et al. 2020) and continual learning (Ebrahimi et al., 2020b). Different from standard su- pervised learning whose goal to minimize the generalization error, continual learning aims to avoid catastrophic forgetting while maintaining good performance
current domain. Inspired by standard gradient- based hyperparameter optimization algorithms such as (Baydin et al. 2018 Franceschi al. 2017, Donini et al.] 2020), develop a novel automatic and adaptive learning- rate update scheme for newly arrival domains. Our goal is achieved by updating the learning rate for each parameter block, which is designed adaptively update parts of the network to fit future domains while remembering previous domains.
More Results
Compare to Baselines
5-way 1-shot learning Table 7 show the comparisons to the baselines constructed for this problem setting in terms of 5-way I I-shot accuracy. the table, 'N/A' means the BWT is not avail- able since the method does not learn sequentially. Our method significantly outperform baselines. Especially, similar to the performance in 5-way 5-shot learning in Table [ in the Protonet-bast based model, the performance of our model almost matches that of the joint-training model, indicating excellent memorization of past domains.
Compare to Baselines with different domain ordering
15

---

Under review as a conference paper at ICLR 2021
Table 7: Compare to Baselines (5-way 1-shot)
5 Way Shot
Way Shot
Algorithm
ACC
BWT
Algorithm
ACC
BWT
Sottonct-Sqquental
36.31 + 0.18
30.97
0.12
ANIL-Seqqeentil
37,55 = 0.21
22. 0.26
Protonet-EWC
42.21 0.21
-9906
0.36
ANC
38.86
-4.15
Protonet-HAT
19.98 + 0.19
5.22
ANIL-HAT
39.90 = 0.11
20.02
Protonet-UCB
41.26 0.17
9946 + 0.07
ANIL-UCB
38.86 + 0.23
0.29 -21.55
Protonet-A-GBB
10.07 0.400
0.44
ANIL-A-GEM
31.62
29.04
Protonet-ER Ringhuffer
-.18 0.11
ANIL-ER-RRingann
31.49
32.45
0.56
Protonet-MER
46.72 0.272:+0.233 0.25
-14122 33.39
ANIL-MER
41.40
-19.24 0.21
Protonet-Ours
53.36 0.27
ANIL--Ours
45.85 0.22
-10.19 + 0.27
Joint training
33.92 0.36
Joint training
57.18
5-way 1-shot learning We study the sensitivity of all methods on the order of domain arrivals in the setting of 5-way 1-shot learning. We different domain-sequence order CIFARFS, Minilmagenet, Aircraft, CUB and Omniglot. Results are summarized in Table X Similar to the results in Table it can be seen that although there are some performance differences compared to those of the previous order, our method consistently outperforms the baselines.
Table 8: Compare to Baselines with different domain ordering (5-way 1-shot)
5 Way ACC
Shot
Way Shot ACC
Algorithm
BWT
Algorithm
BWT
Prolonet-Sequenane
42.66 0.14
-3304
0.12
INIL-Sequentaal
39.61 :: 0.20
-20.55
+ 0.22
Protonet-B EWC
43.97 + 0.20
-17.36
0.33
ANIL-EWC
39.18 4: 0.29
-22.12
0.35
Protonet-HAT
48.92 0.41
0.26
ANIL-HAT
39.02 0.12
-22.62
0.10
Protonet- UCB
39.64 0.14
-10.83
ANIL-UCB
-19.28
0.41
Protonet-A-Gre
46.81 0.23
-18.78
ANIL-A-GEM
37.76 0.35
24.91
Protonet- ER Ringbuffer
48.60 0.35
-1485
0: 0.18
ANIL-ER ANI--ER-RRnghuuferr
35.42 0.31
-26.03
0.37
Protonet-MER
-9.06
0.32
ANIL-MER
40.86
-18.311:
0.31
Protonet-Ours
52.17 1 0.12
-0.74 0.19
ANIL-Ourr
45.25 - 0.20
-6.02 1 0.28
Joint-training
53.92 + 0.36
N/A
Joint-training
57.18 =: 0.38
N/A
Meta testing of current and all previous domains Algorithm testing the model on current and all the previous domains.
shows the algorithm of meta
Algorithm 2 Meta testing.
Require: A sequence of training domain data Dj, Da...., . DN;
Require: Learned meta parameter of the models after training domain N.
I: for q 1 do
2: Sample 1' meta testing tasks Treea from P(D,), the distribution over tasks in domain 2,
3: Evaluate the meta testing accuracy and measurements of forgetting for domain D,, using the meta learned
model F(T, ON ). Tglest
4: end for
5: Evaluate the average meta testing accuracy on all learned domains and measurements of forgetting over all previous domains
16


Reviews:
[{"review_id": "uous3Lj6VKZ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nAt the heart of this paper are two separate contributions. The first is a new online meta-learning problem setting where the meta-learner acts on a sequence of few-shot learning *domains*, as opposed to tasks within a single domain. The second is a method for meta-learning with this form of domain shift.\n\nAs far as I know, this is the first benchmark that tackles online meta-learning over few-shot domains. This is a much needed push in the direction of obtaining harder continual learning benchmark - I'm quite excited about this line of work and would encourage the authors to continue to pursue this line of work and include longer sequences of domains to truly test the continual capabilities of our current methods. \n\nThe authors also propose a meta-learner for the newly introduced problem setup. This meta-learner makes use of two well-known concepts form the literature, experience replay and learning-rate adaptation. In particular, the model's parameters are adapted to the current task (few-shot) using meta-learned learning rates. These learning rates are meta-learned over all tasks in the memory in a multi-task loss function.\n\nPros:\n- A new benchmark that is considerably more challenging than previous online meta-learning benchmarks.\n- Extensive benchmarking of a wide range of baselines from both meta-learning and continual learning.\n- Strong performance from the proposed method.\n\nCons:\n- Unclear claims to novelty.\n- Confusing writing at critical steps.\n\nRecommendation: reject\n\nMotivation: \nI believe the proposed benchmark would be of service to the community and should eventually be published at a peer-review venue, however the current manuscript contains critical issues that prevent me from recommending acceptance. I'm open to changing my score should the authors address my concerns below. \n\nMain concerns:\n\n- *Contributions:* The authors list three contributions. Of these, the first relates to both adapting parameters and learning rates and the second relates to using layer-wise meta-learned learning rates. Both are well known approaches that have been explored extensively in the literature. Meta-learning learning rates was proposed in [1] and have since been explored in a variety of contexts. Adapting both layers and learning rates have also been explored in a variety of forms in episodic meta-learning (e.g. [2]). Further, recent meta-learners [e.g. 3, 4] learn model parameters for fast task adaptation and meta-learn optimiser parameters to avoid catastrophic forgetting, precisely what is claim as a contribution here. This is not to say that there are no algorithmic contributions in this paper, but rather that it is impossible to tease out what the authors claim as their own contribution to the field.\n\n- *Technical correctness:* The writing becomes very dense on pages 4 and 5, where the method is introduced, making it hard to understand precisely what the authors propose. In particular, they describe their method as a bi-level optimisation problem, but Eqs. 1-4 do not support this description.  Both losses have the same input arguments (they only differ in the expectation over tasks) and hence represent a multi-task setup. This is not a well defined problem because the learning rates have no effect on the loss unless the gradient update is taken into account. Similarly, the authors use a single \\theta to denote model parameters, but in Figure 1 and in the text mention that they treat the lower layers of the CNN differently from the top (task-specific layers), as in [5]. This is not described in the algorithm, where it appears as if all model parameters are being tuned to the current task and all layer-learning rates are being meta-learned on replays of previous tasks.\n\nMinor concerns:\n\n- *Clarity:* the manuscript would benefit from a simpler and clearer presentation of the problem setup. At the end of the day, the paper proposes to meta-learn over a sequence of few-shot learning domains, as opposed to a sequence of tasks from the same domain. This will require a higher degree of generalisation from meta-learners, and should stress our current methods. There is no need to make the motivation more intricate than that.  Similarly for the proposed method, a simpler and more direct presentation would greatly help the reader understand what is being proposed and what is being borrowed from previous works. I'm still unclear as to what 'double adaptation' means. Is it just that both learning rates and model parameters are being tuned?\n\n- *Citations:* while the related work section contains a wealth of citations, the introduction makes sweeping claims (such as \"it has been shown that catastrophic forgetting often occurs when transferring a meta-learning model to a new context\") that should be substantiated with appropriate citations. It is also a bit unfair to invalidate all previous works as not applicable to this problem setup because of \"high variability underlying a large number of dynamically formed few-shot tasks\". This is an empirical matter and not a an absolute fact.   \n\n- *Experiments:* generally, the experimental section is sound and features a broad set of baselines and conducts a battery of ablations. While this is great, the one ablation I was hoping for was to see what happens if you change the order of the domains such that miniImagenet and CIFAR are at the end. In general, having one (or both, as in this case) as the first two tasks means the meta-learner can learn good representations for downstream tasks pretty much immediately. This seems to precisely counter-act the goal of introducing a harder benchmark. Finally, I'm a little unclear as to whether catastrophic forgetting is measured 0-shot without re-adapation, or few-shot by allowing adaptation to tasks from past domains given \\theta^j_q?\n\nTypos:\n- catastrophe forgetting -> catastrophic forgetting\n- scarifying ->degrading(?)\n\nPost Rebuttal \n\nThe authors have improved the context of their work and clarified their proposed method. While the technical novelty is somewhat limited, the proposed method does well and the benchmark introduced herein should be of interest to the community. As such I have increased my score.\n\nReferences\n[1] Li et. al. 2017. Meta-SGD: Learning to Learn Quickly for Few-Shot Learning.\n[2] Lee et. al. 2018. Gradient-Based Meta-Learning with Learned Layerwise Metric and Subspace.\n[3] Flennerhag et. al. 2019. Meta-Learning with Warped Gradient Descent.\n[4] Gupta et. al. 2020. La-MAML: Look-ahead Meta Learning for Continual Learning.\n[5] Javed et. al. 2019. Meta-Learning Representations for Continual Learning.", "extra_scores": {}, "extra_text": {}}, {"review_id": "PmeRfM3lkOx", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work focuses on sequential adaptation of a model without forgetting. Their goal is to minimize the catastrophic interference of the model while learning a new few-shot task coming from a different domain. To that end, they introduce a problem setup where the model receives a sequence of few-shot tasks from different domains. \n\nI am not sure I entirely understand that the motivation and the proposed algorithm in this work. Overall the algorithm resembles replay-based continual learning approaches. Also, the algorithm feels similar to Online Meta-Learning. How does adaptation of learning rate address catastrophic interference? Please take a look at my comments below. Maybe, it is helpful to provide a real task for this setup. Having that said, I appreciate the authors' effort put into evaluating multiple baselines.\n\nComments and questions:\n-What is the value of N (i.e how many tasks were shown to the model?)? I am curious about different runs with varied N, for each model including baselines.\n-Maybe discuss memory and accuracy trade-off? Some baselines don’t require a replay buffer nor iterative re-training.\n-Please provide references on the following statements - Most existing works focus on developing the generalization ability under a single context/domain. Recently, it has been shown that catastrophic forgetting often occurs when transferring a meta-learning model to a new context.\n-How does the proposed setup and algorithm compare to those of “Finn, C. Online Meta-Learning”?\n-Some missing references: “Schmidhuber, J. A neural network that embeds its own meta-levels.”, “Santoro, A. Meta-Learning with Memory-Augmented Neural Networks”, “Munkhdalai, T. Meta Networks,” etc.\n-Please note that miniImageNet subset was introduced in Matching Nets.\n-The first sentence in Section 2.1 seems to describe an application of meta-learning to few-shot learning rather than meta-learning itself. There is also a line of works on memory-based meta-learning (Mikulik, V. Meta-trained agents implement Bayes-optimal agents).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "GIZEyVWt38o", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "-Summary-\nThe paper proposes a method for the sequential meta-learning problem. The author meta learn not only model parameters but also learning rate vectors for parameter blocks. To this end, the meta-learn model finds appropriate model parameters and adaptive learning rate vectors that capture task-general information. Overall experiments are performed on few-shot meta-learning settings with sequential domains (datasets).\n\n-Pros-\n- Optimizing fine-grained learnable learning rate vectors for manually grouped parameter blocks is reasonable.\n- The performance of the proposed model significantly outperforms baselines which naively combined existing few-shot meta-learning and continual learning approaches.\n\n\n-Cons-\n- The approach is too simple (Adding learnable strength vector weights for gradient update of conv. blocks) and heuristic. And core hyperparameters are manually decided, like # of blocks per Conv. layer and the size of memory.\n- Lack of analysis. There is a lack of concrete insight into how does adaptive learning rate weights mitigating forgetting. \n- The method only considers multi-head continual learning problems for task-incremental learning. The majority of recent impressing CL works considers further realistic and applicable to the broader areas, called class-incremental learning problem that task oracle isn't given during training/inference of the model.\n- The method is only performed on simple CNN architectures. It needs to be validated on further modern deep neural network architectures. And, while the construction of CNN in this paper, most of the well-known CNN architectures have a different number of filters per layer. In this case, the strategy to split blocks can be important for pursuing a better model. However, there is no discussion/analysis of the problem.\n- Meta-learning with bilevel optimization might require an additional computational cost.\n\n-Comments-\n- Experiments on domain ordering are interesting. I see that recent CL works consider evaluations on multiple domains(datasets) like \"Hard Attention on Task\" (HAT) paper. It would be interesting to perform analysis of the task(domain)-order sensitivity like 'order-normalized performance disparity' (OPD) in [1], which can be beneficial for understanding backward-forward transfer during continual learning under the domain shift.\n- Citation of the XtarNet is duplicated. Please combine them. \n\n[1] Yoon, Jaehong, et al. \"Scalable and Order-robust Continual Learning with Additive Parameter Decomposition.\", ICLR 2020.", "extra_scores": {}, "extra_text": {}}, {"review_id": "0rDp_3GLzWc", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a new instantiation of Continual Few-Shot learning (CFSL) with multiple domains coined sequential domain meta-learning.\nThey also propose an extension of the meta-SGD [1] algorithm to CFSL where the learning rates are learned on tasks from past domains to alleviate forgetting.\nBoth the setting and the idea have potential, but the paper still needs some work and considerable weaknesses.\nMainly, the setting needs further motivation and clarifications, and the method needs some justification as well as stronger performance. \n\nOn the setting:\n- 1.1 The setting's motivations are self-driving car- and dialogue system-like scenarios. The authors claim that these are aligned with the newly proposed setting. In these examples, however, the hypothetical methods would need to adapt to the changing data distribution in an **online** manner and without an explicit support set. In the new setting, methods are tested at the end of training on all past tasks, i.e. long **after** the methods were in the tested domains and with a different model. When a new setting is proposed, it should be clearly motivated. I suggest the authors either improve the current motivations or adapt their settings to better align with real-life continual-learning scenarios. For the latter, the authors could, e.g., report online cumulative performance on all domains with domain revisits, similarly to [2].\n\n- 1.2 It seems like the methods have access to the task label to properly chose their output head (or domain-dependent parameters)? If this is so, the methods are operating in the task-aware setting, where forgetting can be solved by freezing and growing the network. In that case, the authors should add this as a baseline. If they cannot beat it, they should show on what metrics they can outperform it, e.g. maybe their method is more computationally efficient. Also, task-aware doesn't seem to be aligned with the proposed motivations.\n\n- 1.3 The meta-test protocol (ALgorithm 2) is unclear. Are the fast weights retrained? If so, how? Are the learned learning rates $\\lambda_q$ ever used? If not, why are they learned?\n\n\n\nOn the method:\n- 2.1 The authors rush to explain their double adaptation method but their mechanism is not justified anywhere. Specifically, why would you learn the learning rate on past tasks? and why would you backprop through $\\theta_q^j$ which is adapted to domain $q$?\n\n- 2.2 The authors should acknowledge that learning the learning rate (or double adaptation) is a well-known trick now. They should explain how they extend this method and change the tone of the text such that their proposed method seems less novel.\n\n- Going back to 1.3, in ALgo 2, are the learned learning rates ever used? Because algorithm 2 only uses $\\theta_N^M$ i.e the model adapted to domain $N$. \n\nOn the experiment:\n\n- 3.1 why didn't the authors run a hyper-parameter search?\n\n- 3.2 Table 3 ablates their proposed double adaptation on past domain scheme, which is the novel part of their method. The gains are however relatively small. Furthermore, with the large number of hyper-parameters and without a hyper-parameter search, the gains could be noise introduced by the authors iterating on their method by not on the baselines. I suggest the authors run a hyperparameter sensitivity analysis. \n\n- 3.3 The authors explain a new memory selection mechanism that is unclear and doesn't work well yet. I encourage the authors to remove that section and Table 6, or significantly improve it. Also, on \"We then divide the losses into Q clusters with kmeans. \":  if the losses are scalars, why are you using k-means?\n\n\n\nOther concerns:\n- figure 2 seems unconstructive. I suggest the authors remove it or move it to the Appendix.\n\n- put lines in the algorithms.\n\n- The introduction is hard to follow. Also, the 4th paragraph should be moved and merged into the Related Work section.\n\n- In the related work,  the authors claim that [2] is part of a group of papers that operates in a single domain. However, this is not true for [2] which operates in multiple domains within one experiment, e.g. the Omniglot / MNIST / Fashion-MNIST experiment. Also, [3] is a Meta-Continual learning method and not an incremental few-shot learning one.\n\ntypos: \n- in abstract and elsewhere: \"catastrophe\" forgetting--> \"catastrophic\"\n\n\n\n\n_____\n\n**POST REBUTTAL**\n\nThe authors have provided some clarifications. I suggest they use them to improve the paper.\nI'm increasing my score to a 5, thus still not in favor of acceptance.\n_______\n\n[1] Zhenguo Li, Fengwei Zhou, F. Chen, and H. Li.  Meta-sgd:  Learning to learn quickly for few shot learning. ArXiv, abs/1707.09835, 2017.\n\n[2] Massimo Caccia, P. Rodr ́ıguez, O. Ostapenko, Fabrice Normandin, Min Lin, L. Caccia, Issam H.Laradji, I. Rish, Alexandre Lacoste, D. V ́azquez, and Laurent Charlin.  Online fast adaptation and knowledge accumulation: a new approach to continual learning.ArXiv, abs/2003.05856, 2020.\n\n[3] Khurram   Javed   and   Martha   White.Meta-learning   representations   for   continual   learn-ing.InAdvances   in   Neural   Information   Processing   Systems   32,    pp.   1820–1830.CurranAssociates,Inc.,2019.URLhttp://papers.nips.cc/paper/8458-meta-learning-representations-for-continual-learning.pdf.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a sequential meta-learning method over few-shot sequential domains, which meta learns both model parameters and learning rate vectors to capture task-general representations.

Reviewers raised many insightful and constructive comments. The main themes are as follows:
- The problem setting needs further motivation and clarifications, to make it more realistic and applicable.
- The novelty is relatively weak, e.g. the approach is too simple, and learning the learning rate is a common trick.
- The method needs great effort for better presentation and justification. The current presentation simply lists several equations in a dense way without detailed explanation. Some main claims such as mitigating catastrophic forgetting are not elaborated extensively.

AC scanned through the paper and agreed with the reviewers' main points. Authors' rebuttal in general did not address these concerns to the satisfaction. For example, even after revision, the readability of this paper is not good enough. The authors are encouraged to perform a thorough revision.
Decision: Reject