Recasting Continual Learning as Sequence Modeling
Authors: Soochan Lee
Jaehyeon Son
Gunhee Kim
Venue: NeurIPS 2023 poster (2023)
Keywords: meta-continual learning
sequence modeling
Transformers
efficient Transformers
TL;DR: We formulate continual learning as a sequence modeling problem, allowing advanced sequence models (e.g., Transformers) to be adopted as general meta-continual learning methods.

Abstract: In this work, we aim to establish a strong connection between two significant bodies of machine learning research: continual learning and sequence modeling.
That is, we propose to formulate continual learning as a sequence modeling problem, allowing advanced sequence models to be utilized for continual learning.
Under this formulation, the continual learning process becomes the forward pass of a sequence model.
By adopting the meta-continual learning (MCL) framework, we can train the sequence model at the meta-level, on multiple continual learning episodes.
As a specific example of our new formulation, we demonstrate the application of Transformers and their efficient variants as MCL methods.
Our experiments on seven benchmarks, covering both classification and regression, show that sequence models can be an attractive solution for general MCL.

Recasting Continual Learning as Sequence Modeling
Soochan Lee Seoul National University soochan. lee@vision smu ac kr
Jaehyeon Son Scoul National University (tth9776ssu.... .c.kr
Gunhee Kim Seoul National University gunhee@snu.ad .c..kr
Abstract
In this work, we aim to establish a strong connection between two significant bodies of machine learning research: continual learning and sequence modeling. That is, we propose to formulate continual learning as sequence modeling problem, allowing advanced sequence models to be utilized for continual learning. Under this formulation, the continual learning process becomes the forward pass of a sequence model. By adopting the meta-continual learning (MCL) framework, we can train the sequence model at the meta-level, on multiple continual learning episodes. As a specific example of our new formulation, we demonstrate the application of Transformers and their efficient variants as MCL methods. Our experiments on seven benchmarks, covering both classification and regression, show that sequence models can be attractive solution for general MCL
Introduction
Continual learning (CL) refers to the ability to learn from non-stationary stream of data. A CL data stream is typically series of distinct tasks, and the model cannot access previous data while learning the current one. For example, in CL benchmarks based image classification, each task typically consists of a small subset of classes. Since the model's 's access to the past data is restricted, standard training with stochastic gradient descent (SGD) consistently overwrites previously learned knowledge, which often leads to catastrophic forgetting. As learning usually implies SGD updates in the deep learning literature, most CL approaches try combat catastrophic forgetting while relying on SGD to update model parameters. However, using SGD is generally not a necessity of CL; one may choose any update rule other than SGD.
In this regard, we point out that CL inherently a sequence modeling problem. The basic struc- ture of CL, which trains model on training stream ((x1,)1,,-- ... (Er, yr)) and evaluates it on a test set {(En yn)) can be considered as predicting the next token yn. of the sequence (21, ....... .r, yr, 2.). From this perspective, the CL process and the test-time inference a sequence model's forward pass, with no SGD update. Instead, we can train the sequence model at the meta-level with set of CL episodes. This meta-continual learning (MCL) framework is technically equivalent to conventional sequence model training, where a sequence model is trained on set of sequences. Therefore, we argue that any recent and future advances in sequence models can be directly applied to MCL research. As precaution, our approach should not be confused with continually learning sequence model 40 5), which is a conventional CL setting (not MCL) where each datapoint x is a sequence.
As particular instance of our framework, test Transformers 43 which are considered the current sfate-of-the-an sequence model. In the past several years, Transformers have revolutionized various sequence modeling tasks such as language modeling 28 29 6), translation 43 I. speech
recognitiion/genratioo (30, 44], and protein analysis (33 3
31, to name a few. When Transformer-
based language models (LMs) scaled up and trained with massive collection of text data, they
Code available at hea.lb//lllttp///taaaooooo/ooaoo
seg
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

exhibit an interesting ability called in- no-context learning 6 Within forward pass, they can learn new patterns and knowledge from the previous tokens in the current context and upply them solve new tasks. Our framework maximizes this in-context learning ability of sequence models through meta-training and applies it as a solution to CL..
An immediate question may come to one's mind: if Transformer is parallel architecture that can attend all the training data at test time, is this even valid CL setting? Borrowing the words of Katharopoulos et al. 17.. Transformers with causal attention can be considered as recurrent neural networks (RNNs; 13). We can think of a Transformer's s attention keys and values as the internal state of recurrent model. When a new token comes in, this state is updated to include new key-value pairs. At test time, the Transformer queries the key-value pairs in the internal state, not the raw training data. Therefore, with proper implementation, using Transformers does not violate the assumptions of CL.
One drawback of standard Transformers is that the cost of handling each example grows linearly along with the internal state. Therefore, the computational complexity of handling an episode of length T' becomes () Fortunately, there is a considerable amount of work on efficient Transformers 42). which have sub-quadratic complexity. Especially, the efficient Transformers with linear ((T) complexity maintain a constant computational cost to handle each token. In this work, we test Linear Transformer 17 and Performer g as such examples.
By conducting extensive experiments seven diverse benchmarks covering both classification and regression tasks, we demonstrate that Transformers, including the efficient variants, have great potential as general MCL. approach, especially in large-data regimes. Furthermore, it is important to note that our approach is not limited to Transformers. By offering a novel perspective on MCL, we open up the possibility for other sequence models, including those that may be developed in the future, to be applied to this challenging problem.
Lastly, our approach offers unique advantage in terms of biological plausibility. Despite its uni- versality in modern deep learning, backpropagation has long been criticized for being biologically implausible 33 26 21 [2]. This is due to the lack of convincing proof to suggest that biological neurons perform the complex backpropagation process of (i) storing their activations during the for- ward pass, (ii) computing the error derivatives, and (iii) transmitting them backward. Our formulation is free from such criticism since the learning process is a simple forward pass of the sequence model. Only the meta-level optimization, which would have been performed by the evolutionary process in nature, is replaced by more efficient backpropagation.
2 Background and Related Work
2.1 Cota--ontinual Learning (MCL)
Due to the additional layer of complexity from meta-level optimization, compounded by the absence of a standardized lexicon, there exists a high likelihood of confusion in discerning the settings of MCL and related domains. Due to the limited space, this section summarizes only the most relevant works. For a comprchensive comparison of MCL with other fields, such as continual meta-learning we recommend referring to (39.
Continual Learning (CL). A continual learning episode DD consists of stream of training data Dirain = ((#1. V1),.... (IT, yr)) and a test set Diest = { y1). .... , (Z,, yN where x, E X and y. E y are the input and target variables. The data in the training stream can only be accessed one at a time, and is not possible to access previously seen data. If we assume a buffer that temporarily holds each task, this formulation can also cover the offline CL settings where the stream is segmented by tasks rather than examples. The training stream is generally assumed to be a concatenation of K task streams, i.c., Derain Dirain @ @ Dirain each of which is a stationary sequence of data sampled from a distinct distribution. The test set is the union of the task-specific test sets: Diest Diest U... U Diest The objective of the CL algorithm is to continually learn good model fo : X' + ) from the stream by optimizing parameter 0. For example, in the popular CL benchmarks of image classification, fo is classifier that takes an image x X input output its class y E y.
Meta-Continual Learning (MCL). Rather than manually crafting a CL algorithm, MCL aims to learn how to continually learn. More formally, we introduce the concept of continual learner H (X' y) x (X + )/ > (X' - >)), which a functional that takes a data point (are, y1) and an old model fa.., as inputs and produces updated model fe.. For convenience, additionally
2

---

define another functional G. (X' x )1)* - (X' + ) that sequentially processes training stream as input and outputs trained model: fo. = G,(((21,)1) vi), (Iz, H, ((r,, yr), fo.-)). This sequential refinement of the model parameter 0 is commonly referred to as the inner loop, which is the training phase of standard CL., MCL. adds another layer of optimization for tuning the meta-parameter 1) of the learner, which is often called the outer loop. The contents of can vary widely depending on the method, c.g., the initial parameters of a model, a meta-learned encoder, etc. Unlike traditional CL settings where a model is trained on single training data stream and evaluated on a test set, an MCL setting consists of multiple CL episodes split into the meta-training set and the meta-test set, shown in Fig. Ia The episodes of the two meta-splits from the same distribution. Generally, they are created by first randomly splitting available tasks into two groups, where each episode is built corresponding to its meta-split. Since there are no overlapping tasks between the meta-splits, one cannot achieve high meta-test score by simply memorizing all tasks in meta-training. For each episode D, a model is produced by the learner, i.e., for = (Dirrai)) and evaluated on the test set Diest to produce the meta-loss [(fe, D.est). Then 17 updated by meta-gradient descent, i.e., SGD with V,, Vfer,, ) to reduce this meta-loss during the meta-training phase. During the meta-test phase, the learner HI evaluated on multiple CL episodes in the meta-test set. While this description represents the most basic form of MCL that our work is based on, meta-training can deviate from this scheme as long as the integrity of the meta-test is maintained, e.g., 16
assumed to be drawn
random combination of the tasks from the group
Prior Works in MCL. Online aware Meta-Learning (OML: splits a model into an encoder and a small prediction network top of it. In the inner loop, only the prediction network is updated by SGD, while the encoder remains frozen. In the outer loop, the encoder and the initial parameters of the prediction network are optimized. A Neuromodulated Meta-Learning algorithm (ANML; 4 has an additional meta-learned component named neuromodulatory network with the same architecture as the encoder. Its output is passed through the sigmoid and multiplied the encoder's output, gating some features. Meanwhile, there are several MCL approaches specialized to classification. Initially introduced as a few-shot learning method, Prototypical Network (PN: 388 has a meta-trained encoder and computes the average embedding, i.e., prototype, for each class during training. By computing the average embeddings online. PN can be directly used in MCL settings (3). Generative MCL (GeMCL: extends PN by (i) additionally modeling a diagonal covariance matrix of the embeddings for each class and (ii) adopting a Bayesian formulation.
2.2 Transformers
At the core of Transformers is the self-attention mechanism, which interchanges information among different positions within a sequence. In self-attention, each element in the input sequence of length 7' is linearly projected to produce d-dimensional queries, keys, and values, each represented by Q, K, V E Rd'xd, The attention matrix is computed as softmax QQT//V/ where softmax is applied row-wise. It is then multiplied to the value matrix to produce the output of the layer O softmax(QUa QKTT////+ If self-attention is computed in this manner, it is more precisely called bi-directional self-attention since every token can attend to all the other tokens in both directions.
While it is used for encoding a sequence in parallel, uni-directional, or causal self-attention is generally used for decoding a sequence one token a time. causal attention, each token's output depends only on the preceding tokens, allowing new input tokens to be added without recomputing the existing embeddings. For efficient parallel training, this is typically implemented using a causal attention mask that zeros out the upper triangular elements in the attention matrix. Transformers that only have causal attention layers are called decoder-only Transformers [23] which have become the backbone of many large language models (28 29 6). In addition to decoding, decoder-only Transformers are also suitable for encoding a streaming sequence where the tokens become available one at time. This property makes them a perfect fit for our use case in MCL settings.
Efficient Transformers. One drawback of Transformers is their quadratic computational cost that arises from computing the T x T attention matrix. The computational complexity of (T2) can be a severe bottleneck in handling long sequences. Therefore, there has been extensive research on more efficient variants of Transformers. In the following, we describe the kernel-based approaches that we test in our experiments. These models have (T) complexity and can be applied to decoding. Refer to 42 for more comprehensive review of efficient Transformers.
3

---

Met-ttaii
Episode
Train
Tack
Tack
Test
Task
Mota-tost
Ligicode
>>
3
>>
Train
Test
(a) The structure of an MCI. dataset (examples from the CASIA henchmark)
Meta-updatte (outer loop)
Continual learner
Loss
Loss
Loss
Veln Anos
Meta-update (outer loop)
Metaeles
Inner loop
Sequence model (leamer+mode
Virtual model
fr
SCID
SGDD
SGID
SGD
Fiyy
Inner loop
Modol
Episode
Epissode
Ax
A.H.
Xx
72
Az
Ax
Task
Task
Test
Task
Task
Test
(b) SGD-based MCL algorithms
(c) CL as sequence modeling (ours)
Figure I: Schematic illustrations of the key concepts. (a) In MCL, multiple CL episodes split into meta-training and meta-test sets. For each CL episode, a continual learner produces model from the training stream (blue), which is evaluated on the test set (green). The learner is meta-trained on multiple CL episodes in the meta-training set and evaluated on the meta-test set. (b) In many MCL approaches, the learner mainly depends on SGD to update the model in the inner loop. (c) In our framework, a recurrent sequence model plays both the roles the learner and the model.
Kernel-Based Efficient Transformers (KETs). It approximates the attention matrix as the product of Q' ((Q) and K' <(K) where Φ RRd - R is non-linear kernel function that transforms each row of Q and K into an r-dimensional feature. The kernel function is meticulously designed such that softmax (QKT) = D (Q'(K" where D diag Q' (K'T177) 17)) is the normalizing term that ensures each row in the approximated attention matrix to sum to one. Through this approximation, we can change the computation order compute KT V first: softmax (QK ")V s D D (Q'(""Q")"" K'TV V). The overall complexity becomes linear to the sequence length, i.e., O(T). Linear Transformer [17 and Performer [8 are examples of KET' and differ only in their choice of kernel function. The kernel function of Linear Transformer is defined as ((x) = clu(x) ) + 1a where clu is the exponential linear
unit (ELU: U:PT 9 applied element-wise. Performer g uses ( = exp (W x
||a||2/2 1,) where
the rows of W E Rexd are orthogonal random vectors.
Decoding with KETs, In theory, the causal attention of KET can be computed by (i) computing the approximate attention matrix D-1Q Q' k"T (ii) applying the causal mask, and (iii) multiplying V. However, this naive process requires OTT)) computation, and it necessitates all tokens once as input. Fortunately, an alternative method exists to sequentially compute the same output with ((T) complexity [7
row of the matrix as the lowercase letter, c.g., V [ui:... Dr]". At each time step t, the new t-th input embedding is linearly projected produce the query qt, key kú, and value vt. The recurrent internal state at time & holds the matrix K:TV. Ei- @ o(kz)) ) eu and the sum of the key features de =i-- Φ(ke). ). Note that both can be computed recurrently: K V. K V-1 + ((k:)off' and d, di-1 1 + ((k)). The attention output at time is computed as o, V.)
g Here we represent vector corresponding
( ((q))(()))))))) ((::).
3 Continual Learning as Sequence Modeling
Many existing MCL (and also CL) approaches rely heavily on SGD to update the model [16] as summarized in Fig. Tb and Alg. [ When each data point (It, yt) is made available, the model prediction d, fo. (x+-1) is compared with ye to produce the loss. The gradient of the loss w.r.t.
the parameters 0.-1 1 is fed into an SGD--based update rule, which sometimes accompanies additional treatments to prevent forgetting, to produce updated parameters 8,.

---

Using sequence models can also be advantageous in terms of scalability. A major drawback of the SGDD-based MCL. is computing the meta- meta--gadient V ((or Dirsst) in the outer loop. Its computation involves caleulating higher-order derivatives and large memory for the entire computation graph of the inner loop, which holds every revision of the parameters {0, J the gradients { (Vo,_, Le ) and their intermediate calculations. Thus, it is hard to scale the model size or the length of the inner In contrast, we propose to formulate CL as a sequence modeling problem and let sequence model play both roles of the continual learner and the model, as presented in Fig. Ic In our framework, the inner loop and outer loop are equivalent to the forward pass and SGD update of the sequence model, respectively. To comply with the assumptions of CI., the sequence model should be representable as recurrent learner H, that iteratively refines an internal state h2 as new data arrive, ie., h2 = H, [hx-1, (xz, 3t)). We will explain in $ $3.1 that Transformers can be seen as a recurrent model. Since the Markov property holds between the internal states, he should summarize all the data seen up to a time step t. The combination of h2 and the sequence model's parameter v thus defines virtual model fo, (X - )) with 0₂ (n,hi). After forwarding training stream Dirain of length T. we obtain a virtual model for trained in-context. Then, each test input In is considered as the next token 41+ and Un. fe, (In) is produced from the forward pass of the sequence model conditioned on hy. The meta-loss function is defined to maximize the conditional log-likelihood:
[(Jor, Dlest)
Σ
llo Dogp(ynleee (znn 0T)
Σ
llog (yn lors v1).
(B Di
This meta-loss in the MCL perspective is technically equivalent to the popular next-word prediction loss in language modeling (28 29]. That is, we update yj to minimize the meta-loss via SGD, as commonly done for language model training. Although we mainly assume all test examples come after the training stream, it is also possible to feed test examples in the middle of a training stream if the corresponding task is already learned. To avoid unnecessary complexity, however, we mainly assume all test examples to come after the training stream.
An important characteristic of our approach is that the target y is also taken as an input of the sequence model. Since conventional MCL algorithms maintain an explicit model fo : X + Y" the only way for them inject the knowledge from y into the model is the SGD update with the inner loss VoL(fe, (z). y) (Alg. 1 On the other hand, our sequence modeling approach replaces the SGD update with the forward pass, which incorporates y's information by taking it as input. The forward pass can potentially learn a far more flexible update rule compared to SGD. As presented in Fig. Ic we denote the state after seeing Ve as h differentiate it from hy, the state after seeing both Jt. and y/c.
loop. On the other hand, our framework has no such meta-gradient (i.e., our meta-loss is normal sequence model loss), which allows us to use a much larger and more powerful architecture within the same computational budget.
It is worth mentioning that there are other meta-learning approaches that frame various learning scenarios as the forward pass of a neural network. For instance, [35] 32] employ Aasebased architectures for few-shot learning, while [24] utilize a combination of temporal convolution and attention layers for both supervised and reinforcement learning.
3.1 Transformers as Meta-Continual Learning Methods
As an instantiation of our framework, we demonstrate that Transformers can be utilized as general MCL mechanism. We only consider decoder-only Transformers, which are suitable for handling the streaming data in CL settings.
Initially, it may not be apparent how the decodcr-only Transformers can be represented as recurrent model in Fig. [C] since Transformers are frequently described as a parallel computation graph over the whole sequence. As highlighted in [17], however, Transformers with causal attention can be regarded as recurrent models. A decoder-only Transformer layer consists of two sublayers: a causal attention layer and forward layer. Since the
layers work on each token independently. information transfer across time steps occurs exclusively through the causal attention sublayer, and only in the temporal direction (-+). Therefore, at any time I, all the past information represented as keys ( (k( ) and values (w% He in each attention head I, which are accessed by the query g/- We can think of the keys and values in each attention layer as the internal state of recurrent sequence model: he U, h where ht {(k,, o/. ) At every time step, this internal state is updated to
5

---

include the keys and values from the current time step, as highlighted in Alg. 2 In Alg. 2 also notice that yt regarded as the input of Transformers (L89), 9b, and the virtual model J ((n evaluated on the test set to produce the meta-loss (I 11
3.2 Kernel-Based Efficient Transformers for Efficient Meta-Continual Learning
Meanwhile, Transformers have been criticized for their consistently growing computational cost, which is more commonly referred to as the quadratic complexity in the Transformer literature. This limitation of Transformers triggered extensive research on efficient Transformers [42] which have sub-quadratic complexity. By bridging CL and sequence modeling, we allow this family of advanced Transformers to be immediately applied as CL methods.
Algorithm 1 Inner loop of conventional SGD-based MCL.
Require: learner Hy, Dirain Di
1: T + |Diriin|
2: feg initial model
The only requirement for the model is to support decoding, which is required to sequentially take new examples in CL settings.
3: for L < 1 to T
4:
It 4- Dirain [4
5:
L + [(fo, (x.).
b Loss
6:
fo. H,, 1 Vo,1 Le)
Among the large taxonomy of efficient Transformers, our work focuses on kernel-based efficient Transformers (KETs), which can be adapted to decoding while having (TT) complexity. As explained in $2.2 the decoder version of KET works very similarly to an RNN with an internal state of a constant size. As summarized Alg. the update rule corresponding to each attention head I can be concisely expressed as adding the outer product of the key feature o(k') and the value vector oy to the hid- den state of fixed size. Note that 1 is appended to of to compute the sum of the key features d' E ((k{,) ) simultaneously. We test Linear Transformer [17] and Per- former [8) and the only difference between the two is the kernel function Φ as explained in 22.2
7: end for
8: return L(fo,, Dhost
Meta-loss
Algorithm 2 Inner loop of Transformer
17 and L of
Require: Transformer Hy, Dtrain Dreet 1: T + [Dirain|
2: I + indices of attention heads in Hap
3: ho + O
D internal state
4: for t 4-
do Jr, + D' Dirminn(]
5:
6:
{(k), v))liez t HH 11)
7:
hi m..-hh-11 11-1 [(k), wieeer
8: ((K),,)))) <- Hy (1t
9: he 4- th U {(kl, vi Hezz
Connection to AmmamicStatic-AArhiteciie proaches. Since the size of the internal states linearly grows more data come in, standard Transformers can be grouped with the dynamic CL architectures. As the size and the computational cost of
CL Ap-
10:
end for
11:
return [((in Disaty Meta-loss
Algorithm 3 Inner loop of KET
CL approaches typically grow linearly with the number of tasks K [4 2 45 20 the complexity of handling CL episode of length T is OKT)) In most CL scenarios where K T. the complexity becomes O which is the same as Transformers. On the other hand, KET'' inter- nal states remain at constant size, which makes them the counterparts of static-architecture CL approaches.
Require: KET H,, Ditrai DDtest 1: T [Dirain)
2: I < indices of attention heads in H.,
Superscript implies 'Vi I
3: Nú +- O+x(d+1)
D internal state
4: for t 4 t do
5: Ft, + (irmin(t)
6:
{(ki, vi) + (he
3.3 An Example: Continual
7:
ht hi + (k])([b
8: ((ki, + H (h
Learning of Image Classification as Sequence Modeling
9: hi 4 b + (kk)) ;1]
We describe a specific example of applying our approach to image classification, which has been widely used for evaluating MCL methods. In this setting, x is an image, while y is the corresponding class label. To feed into Transformers, both x and y should be encoded into a fixed-sized vector. While we can simply use a convolutional network (which can be jointly trained end-to-end) for z, properly encoding y requires some considerations. Since class label is essentially an abstract symbol, e.g., an arbitrary index of the class, it does not have any meaning on its own. Simply hard-assigning trainable embedding to each class would fail, as novel classes would appear in the meta-test phase. Randomly sampling a fixed embedding every time class appears also turns out to be inapplicable as we experiment. The random embedding the same as an untrained token embedding, which is expected to be incompatible with the trained weights of Transformers.
10: end for
11: return [(Sin,ay)) Dirst))
Meta-loss

---

The general solution used in language modeling is to express everything as a combination of known vocabulary [37] Following the same principle, we tokenize class label using a fixed vocabulary V and train an embedding for each token in the vocabulary. Since class labels do not impose any constraint on how they should be tokenized, we randomly sample a unique class code (v1, ...: 1c)), random combination of the tokens ve V, when a new class appears in an episode. Note that if the code length C' is greater than one, each token goes into Transformer individually, i.c., an (x, y) pair is converted to the sequence (Iene; UI, (t) before going into the Transformer. At test time, the Transformer sequentially outputs the code to predict the corresponding class. To reduce complications, we keep C in the main experiments. and the vocabulary size is set to the maximum number of classes that can appear in an episode. Experiments with C' > can be found in Appendix
4 Experiments
We demonstrate the potential of our approach through experiments with seven benchmarks. Unless otherwise noted, each experiment repeated five times, and the mean and standard deviation of the five runs are reported. Due to space constraints, this section contains only the essential information to understand the main result, while details, additional experiments and analyses are deferred to Appendix A B and
4.1 Benchmarks
To show the generality of our approach, we conduct experiments with diverse types of tasks, including both classification and regression. To construct the meta-traintn training and meta-test set for dataset, first divide it into a set tasks. The tasks are then split into two disjoint sets, one for meta-training and the other for meta-testing. The CL. episodes for both meta-splits are constructed in the same manner. To build a CL episode D, we randomly sample K unique tasks. By default, we set K = 20, while additionally testing the K = 100 setting compare performances with longer episodes. For each task k the training stream Drrain and the test set Diast contain five examples each (i.e., five shots). For each experiment, meta-train for 50K steps with a batch size of 16 (i.e., 16 episodes in parallel) and meta-test with 1,024 episodes. Task identities are not provided in any case.
4.1.1 Classification
For simplicity, we define each class as task. All input images are resized to 32 x 32 resolution. We report classification errors in percentage as the evaluation metric.
CIFAR-100 (18]. CIFAR-101 contains 60K images of 100 classes, with 600 images for each class. We randomly choose 60 classes for meta-training and the other 40 for the meta-test.
benchmark for meta-learning or MCL. It has a total of 1,623 character classes gathered from 50 alphabets. The meta-train set consists of 963 classes (30 alphabets), and the meta-test set has 660 classes (20 alphabets). There 20 images for each class, thus 19K images for meta-training and 13K images for meta-test.
Omniglot [19. Omniglot is handwriting dataset widely used
CASIA Chinese Handwriting Database (CASIA; 22). This is another handwriting dataset that consists of 7,356 character classes in total. The classes comprise 7,185 Chinese characters, 52 English alphabets. 10 digits, and other frequently used symbols. The total number of images is 3.9M, which is 120 times larger than Omniglot. We randomly sample IK classes for the meta-test and use the rest for meta-training. To the best of our knowledge, this dataset has not been used in meta-learning literature. However, its substantial size allows us to evaluate different aspects of MCL algorithms, which could not be examined with smaller datasets where meta-overfitting is major factor in the evaluation. Our results suggest that the relative performance of MCL algorithms can differ in a large data regime.
MCL
MS-Celeb-IM 10). In addition to the CASIA dataset, we also propose to repurpose MS-Celeb- 1M as large-scale MCL benchmark. It is facial image dataset with roughly 10M images of 100K celebrities. Similar CASIA, we randomly select 1 classes, the celebrity IDs, for the meta-test and use the others for meta-training.
7

---

Table 1: Classification errors (%) in 20-task 5-shot MCL. Both meta- teta-ttaining and meta errors are reported to highlight the relationship between the degree of meta meta-overfitting and the scale of the dataset (@: images, classes).
CIFAR-100
Omniglot
CASIA
MS-Celeb-Ine IM
60K // /001K
32K @ 1.6K
3.9M e 7.2K
10M @/ 100K
Type
Method
Meta- train
Meta-
Meta-
Meta-
Meta-
Meta-
Meta-
Meta- test
test
train
test
train
test
train
Offline Offline
N/A 73.3 73.3
N/A
35.0
N/A
41.726,2
N/A 63.3
PN
0.0#t.0
76.6
0.0
3.8
t0.1
0.2
0.4
t0.0
32.5
+0.1
33.6
GeMCL
0.0
76.6
0.0
3.8 3.8
0.2
0.4 t0.0
32.1
+0.1
33.3
OML
0.6
89.9
0.1
24.8
2.8+0.1
3.2
41.8
42.5
SGD
ANML
0.4
88.1 +1.4
0.0 +0.0
31.0
3.7*0.5
4.3
43.8 +0.3
44.8 +0.4
Transformer 0.0#0.0
82.8
0.0
13.7
0.3
0.4
29.1
30.0#0.2
Ours
Linear TF
0.1
83.4
0.0
36.0
0.4
0.7
31.1#03
32.4#0.3
Performer
0.0
82.9
0.1 +0.1
37.1 14.6
0.5
0,7
32.5 60.5
33.7
Table 2: Classification errors (%) and regression errors of 100-task 5-shot MCL.
Type
Method
CASIA MS MS-Celeb-IM
Sine
Rotation
Completion
Offline
Offline
70.7
87.1#333
0.0180
0.699
0.1713#0.0047
PN
0.8
44.5
N/A
N/A
N/A
CI
GeMCL
0.8
11.1=0.1
N/A
N/A
N/A
SGD
OML
6.8
54.5#0.2
0.0498
0000
0.524
0.1087=0.000
Transformer
1.0=0.0
40.5=0.1
0.0031
10.0002
0.031
0.0989=0.00m
Ours
Linear TF
2.3
15.3
0.0139
t00.0003
0.047
0.000
0.1084 000000
4.1.2 Regression
Sine Wave Reconstruction (Sine). Inspired by the sine wave regression problem in 16. we design more challenging synthetic regression problem for MCL, A sine wave w(T) = A sin + v) characterized by amplitude A, frequency v, and phase v.. We define the target y as the values of the sine wave at fixed points: = (w)( ... (T50)) All y each task share the same frequency and phase, while they can vary in amplitudes. We corrupt into x by shifting phase and adding Gaussian noise, where the phase shift amount is sampled for each task, We report the mean squared error between y and the model prediction ý as the evaluation metric.
Image Rotation Prediction (Rotation). model given an image rotated by an angle vé E [0, 2 and tasked to estimate an angle v. The evaluation metric - cos(s - v): thus, a perfect model would score 0 while random guessing would score 1.0 on average. We use the CASIA images and define each class as a task using the same meta-split.
Image Completion (Completion). An image completion problem involves predicting the missing parts of an image based on the available parts. Derived from our CASIA classification benchmark, we change the input x to be the top half of the image and the target to be the bottom half. The evaluation metric the mean squared error of the predicted pixel values.
4.2 Methods
For all methods, we use a five-layer CNN to encode image inputs and a three-layer MLP to encode vector inputs (e.g., the sine regression). All components trained end-to-end from scratch. We group compared methods into four types: offline, class-incremental MCL, SGD-based MCL, and ours.
Offline. We compare the scores of offline learning since it generally perceived as the performance upper bound of non-meta-CL, Note that MCL approaches outperform offline learning by lever-
8

---

aging additional meta- traininr data. Since the model overfits to the training data we report the best test score achieved during training. We repeat each experiment ten times and report the average and standard error of mean.
MCL (CI). We test PN specialized for caasminnrremenal settings, Although they cannot be applied to other settings, such as regression, they show strong performance in the classification benchmarks.
38 and GeMCL
k
MCL. approaches that
SGD-Based MCL (SGD). We select OML T6 as the main baseline since it can perform both classification and regression tasks. It has a two-layer MLP as the prediction network, top of meta-learned encoder. For 20-task 5-shot classification benchmarks, we also test ANML [4], which is an extension of OML with an additional modulatory network that regulates the features of the encoder. However, due to its extensive memory consumption, it could not be tested on longer episodes.
Ours. We test the vanilla Transformer [33] Linear Transformer [17], and Performer [8. All the models share similar architecture: layers, heads, and 512 hidden dimensions. Although is tiny compared to modern language models, it is still larger than the FC layers of the baselines. Surprisingly, however, they require comparable or less GPU memory and time for meta-training in our experiments due to the lack of meta-gradient computation, as noted in S For more analysis on the computational cost, refer to Appendix C
4.3 Results
In Table D report the meta-training training and meta-test errors of the 20-task 5-shot classification benchmarks to show the necessity of large-scale MCL benchmarks. In CIFAR-100 and Omniglot, all methods show severe degree of meta-overfitiing the gap between the meta-training and meta-test scores. Meta-overfitting is more serious in CIFAR-100 where the task (class) diversity is lower. In such situations, the Transformers still perform competitively, but the baselines may achieve better meta-test score due to less meta-overfitting. On the other hand, if more data is available, as in the CASIA and MS-Celeb-I1 IM benchmarks, the Transformers outperform OML and ANML by large margin and perform on par or better than PN or GeMCL, which are specialized for classification. shows shows the results of longer 100-task 5-shot experiments in both classification and regression domains. In this more challenging scenario, the Transformers still perform comparably to class- incremental methods and significantly better than SGD-based approaches. The scores of offline learning mostly fall behind other MCL methods, which implies that non-meta- CL methods are not suitable for our experimental settings where the amount of data is far from abundant.
A strength of our approach, which comes with the use of Transformers, is parallel meta- meta-training. While the baselines require sequential SGD update the training stream, Transformers can process the whole training stream in parallel, which can dramatically improve the meta- meta-trainin training efficiency once combined with massively parallel processors like GPUs As a result, the meta-training time of Transformers is several times shorter than the baselines. The details can be found in Appendix C
The tradeoff between the standard Transformer and the KETs is also worth discussing. In all experi- ments, the KETs sacrifice some level of accuracy for greater efficiency. The accuracy degradation is more severe in longer episodes, which is expected given that more information has to be fit into the internal state of the same size. This tradeoff suggests that the choice between standard and efficient Transformers ultimately depends the required accuracy and the available computational resources.
we present more experiments and analyses, including different episode configurations and model sizes. Although preliminary, the experiments with Transformers of varying sizes are particularly interesting, as the performance scales with the number of parameters. This is reminiscent of the scaling law of large language models [4], suggesting that scaling might be a potential solution to more challenging MCL problems.
In Appendix B
5 Limitations and Conclusion
By formulating MCL as sequence modeling, we pave the way for incorporating advanced sequence modeling technologies into the MCL field. Given the observation that our approach demonstrates greater efficacy with increased data, seems consistent with The Bütter Lesson, an argument popular- ized by Sutton (1]. It states that, in the long run, general-purpont learning algorithms capable of

---

leveraging more computation and data are likely to be more effective than human deiiged algorithms. We believe our work is step forward in this direction, generalizing the SGD-based update rules in CL with the learnable forward pass of a sequence model.
However, it is important acknowledge that the sequence models also bring their own limitations and challenges, Efficiently handling long sequences is one such challenge. According to our experiments, even the efficient Transformers, which are designed solve the exact challenge, have to trade off a nonnegligible amount of performance for efficiency, which implies plenty of room to improve. Nonetheless, considering the current pace of development in the sequence modeling field, we are optimistic about the potential of our approach; if a new solution is developed for sequence modeling, it can be readily integrated into our formulation.
Acknowledgements
We thank Jaekyeom Kim, Dongjoo Kim, Dongyeon Woo, and Sangwoo Moon for their thoughtful feedback. This work was partly supported by Samsung Advanced Institute of Technology. the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. (223RAA2C2005773, and Institute of Information & communications Technology Planning Evaluation (ITP) grant funded by the Korea government (MSIT) (No. 2022-0-00156 Fundamental research on continual meta-learmin transformation; No. 2021-0-01344 Artificial Intelligence Graduate School Program (Seoul National University)). Gunhee Kim is the corresponding author.
for quality enhancement of casual videos and their 3D metaverse
References
[1] Roee Aharoni, Melvin Johnson, and Orhan Firat. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pages 3874-3884, Minneapolis, Minnesota, 2019. Association for Computational Linguistics, doi: 10.18653/v 1/N19-1388, URL btttps://aacanthonn
org/ N19-
[2] Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 2 21-26, 2017, pages 7120-7129. IEFE Computer Society, 2017. doi: 10. L109CCVP.2007...5
Utt.httts:///o1 org/ 10. 1109/CVPR. 2017 753
[3] Mohammadamin Banayeeanzade, Rasoul Mirzaiezadeh, Hosein Hasani, and Mahdieh Soley- mani. Generative vs. discriminative: Rethinking the meta-continual learning. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neu- ral Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 21592-21604, 2021. ht h/tp:eeep///roceiinng./ wtwpoeeeeiinnn.euriiiippeee neurips wwtp:ipps.cppp.e/.eeeepep wwtppapppe///////aa/eeee 2021/has// 04426668887775666000676663333333 www.:cccooo.octtt..........a
[4] Shawn Beaulieu, Lapo Frati, Thomas Miconi, Joel Lehman, Kenneth O. Stanley, Jeff Clune, and Nick Cheney. Learning to continually learn. In Giuseppe De Giacomo, Alejandro Catalá, Bistra Dilkina, Michela Milano, Senén Barro, Alberto Bugarín, and Jérôme Lang, editors, ECAI 2020- 24th European Conference on Artificial Intelligence, 29 August-8 September 2020, Santiago de Compostela, Spain, August 29 September 8, 2020 Including 10th Conference on Prestigious Applications of Artificial Intelligence (PAIS 2020), volume 325 of Frontiers in Artificial Intelligence and Applications, pages 902-1001, IOS Press, 2020. doi: 10.3233/ 000.3333//AAA2001199
FAIA200193. URL httpst//s///oi. /tt/////000.rr0...22//2//2221210
[5] Magdalena Biesialska, Katarzyna Bicsialska, and Marta R. Costa-jussà. Continual lifelong learning in natural language processing: A survey. In Proceedings of the 28th International Conference on Computational Linguistics, pages 25211, Barcelona, Spain (Online), 2020. International Committee Computational Linguistics. doi: 00.186533/1/2220.ool/// www.oiiinnnniii5777. main.574. www.oolooyoooogg////ooooo org///20000000 col.ng-oonnnnonooo main. 574
URL
[6] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranay Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal,
10

---

Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learn- ers. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tich Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 12, 2020, virtual, 2020, URL p/t.teed//p.o..ednggg....
www.nnuuip.... ceturpps.ccppppeeee. wwt.cc//ppppp///// 202h/hash/hh
www.rAsssttttttmmmmm html
[7] Arslan Chaudhry, Puneet Kumar Dokania, Thalaiyasingam Ajanthan, and Philip H. Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. ArXiv preprint, abs/1801 111180110011 101110 2018. URL httpt:ttsss::iiit///apa org/ www.///g//////o/ooeeeeeeee////// 10t1.1111...1 10112
[8] Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamás Sarlós, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger. Lucy J. Colwell, and Adrian Weller. Rethinking attention with performers. In 9th International Conference on Learning Representations, ICLR 2021, Virtall Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://openrreiiee /ete//eeeeeeeeee/ee/eeee f forum?id=UabbuukORRi
[9] Djork-Aré Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. http:://rxxii...o// www.worggabb..... 1511 07289
[10] Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He,, and Jianfeng Gao. Ms-celeb- Im: A dataset and benchmark for large-scale face recognition. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision ECCV 2016 14th European Conference, Amsterdam, The Netherlands, October 11-14, 20/6, Proceedings, Part III, volume 9907 of Lecture Notes in Computer Science, pages 87- Springer, 2016. doi: 10. 10000107----319-46600 4644--999999 URL
[11] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778 IEEE Computer Society, 2016. doi: 10.1 109/0/P..2221..00 URL pesttts://oot orgooog//11 10 1109/CVPPPP.. 2016.90
[12] G. Hinton. The forward-fowart algorithm: Some preliminary investigations. ArXiv preprint, 1is/2212.13345, 2022. URL htttt::t//rrtttt http:/www.rrg/b//22/2//// 13345
[13] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9: 1735-1780, 1997.
[14] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models. CoRR, abs/2203. 15556, 2022. doi: http: org/ 10. 48550/ arXiv 2203. 15556
10.485500arx1v.2203..
[15) Sergey loffe and Christian Szegedy. Batch normalization: by reducing internal covariate shift. Francis R. Bach and David M. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pages 448-456. JMLR.org, 2015. URL URL http:t//p//poceeeee http:///cccceeeng..oeee mlr ptwss/v37/ioftteeeo hhmll
Accelerating deep network training
[16] Khurram Javed and Martha White. Meta-learning representations for continual learn- ing. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché- Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Informa- tion Processing Systems 32: Annual Conference on Neural Information Process- ing Systems 2019, NeurIPS 2019, December 8-14. 2019, Vancouver, BC, Canada,
11

---

pages 1818-1828, 2019. URL ptt.Lattp///ooocccggg..... http::/:w....oo.pppooppapppppp/p https//c/aaapee//////eassa f4a01661772111835562222000..0662
[17] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mns:: Fast autoregressive transformers with linear attention. Proceedings of the 37th
International Conference on Machine Learning. ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 5156-5165, PMLR, 2020.
URL http::h/ppocceediinggge///////ee mlr, prwps/ prress/v119/e http//1111//1//e #119/katharopoulos220a htmml
[18] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
[19] Brenden M. Lake, Ruslan Salakhutdinov, and Joshua B. Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350: 1338, 2015.
[20] Soochan Lee. Junsoo Ha, Dongsu Zhang, and Gunhee Kim. A neural dirichlet process mixture model for task-free continual learning. In 8th International Conference on Learning Represen- tations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https://openrevies. hett/formmidd-SSS0000
[21] Timothy Lillicrap, Adam Santoro, Luke Marris, Colin Akerman, and Geoffrey Hinton. Backpropagation and the brain. Nature Reviews Neuroscience, 31:335-346 2020.
[22] Cheng-Lin Liu, Fei Yin, Da-Han Wang, and Qiu-Feng Wang. CASIA online and offline chinese handwriting databases. In 2011 International Conference Document Analysis and Recognition, ICDAR 2011, Beijing, China, September 18-21, 2011, pages 37-41. IEEE Computer Society, 2011. doi: 10. 10.100/CCDA..211..1 URL https:://oi....... www.aai....oggeoo 10.11011111 1109/
ICDAR. 20
[23] Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating wikipedia by summarizing long sequences. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May
3, 2018, Conference Track Proceedings. OpenReview.net, 2018. httl:httt////peereeeee
het//forwniid-tttgrrbbc-
[24] Nikhil Mishra. Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. In 6th International Conference on Learning Representations, ICLR 2018, Van- couver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, net wtt.rooormmddiiiie BImUZWWAW
2018. URL pattpps//openreviien
[25] Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In Johannes Fürnkranz and Thorsten Joachims, editors, Proceedings of the 27th International Conference on Machine Learning (ICML-10), June 21-24, 2010, Haifa, Israel, pages 807-814. Omnipress, 2010. URL https: //ccm//c.lll../ cc /ttp////eeeeeeee/////e/o/ooooeee p 432. pdf
[26] Arild Nøkland and Lars Hiller Eidnes. Training neural networks with local error signals. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICMI. 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 4839 4850. PMLR, 2019. URL. htth:::/ppocceedingggee///////ee mlr .wwes/pes///eee v97 nokland19a. html
[27] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary De Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- high-performance deep learning library.
Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' A'ché- Buc, Emily B. Fox, and Roman Gareet,, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 8035-8035, 2019. URL https: httptt/ppp/ppppppinggg..e.. beeteeedinns.neuuiiiin neur ips. falbae288feeettt2222rrrfff000172 Abstract..ttttm
cct.iisss.....eeeeeeo hatsscccppapa hcc/paper/2019//ee
12

---

[28] Alec Radford, Karthik Narasimhan. Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.
[29] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
[30] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. ArXiv preprint, abs/2212.04356, 2022. URL //ttps:///axii wwt.abs.///ooooooooooo...oo..s./ 2212. 04356
[31] Roshan Rao, Joshua Meier, Tom Sercu, Sergey Ovchinnikov, and Alexander Rives. Transformer protein language models are unsupervised structure learners. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL http::ttt/ppperrriiiiii
[32] Sachin Ravi and Hugo Larochelle. Optimization as model for few-shot learning. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24- 26, 2017, Conference Track Proceedings. OpenReview.net, 2017. https:/::/
met//forumiddxxXY-KKK1]]
[33] Alexander Rives, Siddharth Goyal, Joshua Meier, Demi Guo, Myle Ott, C. Lawrence Zitnick, Jerry Ma, and Rob Fergus. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences of the United States of America, 118, 2019.
[34] Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. ArXiv preprint, abs/ 1606046661 2016. URL https://arxii wwrggabb//6606.. 04671
[35] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy P. Lillicrap. Meta-learning with memory-augmentee neural networks. In Maria-Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 1842-1850. JMLR.org, 2016. URL http:///proceedings mlr wttppp/s//////attrrroo/. httm html
[36] Benjamin Scellier and Yoshua Bengio. Equilibrium propagation: Bridging the gap between energy-based models and backpropagation. Frontiers in Computational Neuroscience, 11, 2016.
[37] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany, 2016. Association for Computational Linguistics. doi: (0.18653/v1/PH// 1/P16-16111 1162. URL https / //aclanthology.org/ 101.1/1-11.//0111111111
[38] Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA. USA, pages 4077-4087, 2017, URL httttpp:://proceedinng nww.iis....ppp..p cc/ happ:7/000/aaaae//ce//////////e2 htt.2/9000.aacbbb22atttttrr....
[39] Jaehyeon Son, Soochan Lee, and Gunhee Kim, When meta-learning meets online and continual learning: A survey. CORR, abs/231 1.52411, 2023. doi: 10.48550/ARXIV22311 1.05241. URL
https://doi.ows org/10.
0erpg//..8555000 48550/ arXiv. 2311 05241
[40] Fan-Keng Sun, Cheng- Hao Ho, and Hung- Yi Lee. LAMOL: language modeling for life- long language learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL
attps://openrevier pta.prrrviee..nn//oormmm..
[41] Richard Sutton. The bitter lesson. Blog post, 2019. URI. http:://wwttt http://www..inomppeeeeeeeee. htte//nclldeas/BitteeLLeeeee html
13

---

[42]
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. ACM Comput. Surv. 55(6). 2022. ISSN 0360-0300. doi: 10.1145/3530811. URL https 7 7/doi. /doi.org/10... 1145/11155 3530811
[43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30; Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017. Long Beach, CA, USA. pages 5998 6008, 2017. URL httpttts:: bttpe://////r neur ps. cc/ www.e/ape//aaeeeeeeeeee 201////////////////
[44] Yuxuan Wang, R. J. Skerry-Raan, Daisy Stanton, Yonghui Wu, Ron J. Weiss, Navdeep Jaitly, Zongheng Yang. Ying Xiao, Zhifeng Chen, Samy Bengio, Quoc V. Lc. Yannis Agiomyrgian- nakis, Rob Clark, and Rif A. Saurous. Tacotron: Towards end-to-end speech synthesis. In Francisco Lacerda, editor, Interspeech 2017, 18th Annual Conference of the International Speech Communication Association, Stockholm, Sweden, August 20-24, 2017, pages 4006- 4010. ISCA, 2017. Jtthttp: / www.wissca-ppeeee speech. org/ hatpereechnttrreeeee0e0 abstracts/1452. html
[45] Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Hwang. Lifelong learning with dynamically expandable networks. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 2018, Conference Track Proceedings. OpenReview.net, 2018. URL URLhttps: http://paarrrroo.eeeeeoooorrraaa
14

---

A Experimental Details
A.1 Common Training Setups
Unless explicitly stated, all methods share the same encoder for input æ. For image inputs, we use five-layer CNN architecture. The kernel size is set to 3 x 3 in all convolutional layers, and batch normalization [5]] and ReLU [25] activation are applied after each convolution. the stride is set to 2, except for the first layer which has stride 1. The channel sizes are set to 32-64-128 104-128-226-2555 for the five layers. In the case of the vector imput in the Sine benchmark, we use a simple three-layer MLP architecture. It has three linear layers followed by batch normalization and ReLU. For all benchmarks, we do not use any data augmentation technique.
Since there are lot of different experimental settings in this work, we could not perform an extensive hyperparameter search in every setting. Therefore, we mainly tune the hyperparameters to the CASIA classification benchmark and apply the same settings to the other experiments. We find that Transformers are very robust to the choice of hyperparameters, achieving impressive scores in all benchmarks without any benchmark-specific hyperparameter tuning. In some baselines, however, we had to manually tune the hyperparameters specifie to a benchmark, in order to achieve a reasonable score. Please refer to the configuration files and commands in the included code for the exact configurations.
A.2 Benchmarks
CASIA. The CASIA dataset originally contains grayscale images of various sizes since the images are cropped tightly to the handwriting. We preprocess these images into 32 32 by (i) resizing the image such that the longer edge becomes 32 pixels long and (ii) adding paddings symmetrically to both sides of the shorter edge fill the empty space. This maintains the original aspect ratio of the images while standardizing their size.
MS-Celeb-IM. MS-Celeb-IM IM also contains images of various sizes. Since the images are roughly aligned to the face, we simply resize them to 32 x 32 resolution without keeping the aspect ratio. We also exclude the classes (celebrity IDs) with fewer than 20 examples.
Rotation. In the Rotation benchmark, a model should produce an angle &b given an image. Due to the cyclic nature of the angle, we let the model output (cos vi, sin 3) instead of v.. To necessitate learning from each training stream, additionally shift all va by the same random amount sampled from range [0, 2T). Therefore, even there is a trivial pattern in X' indicating orientation, e.g., a unique shape marking the top of image, the model cannot predict the shifted without the knowledge from the training stream.
Completion. Although the output format is an image, simply treat the output and target 512-dimensional vector (flattened from 16 x 32 image). We simply use fully connected layers for output and do not use specialized architectures, such transposed convolution.
A.3 Baselines
As SGD-based MCL baselines, we test OML [6] and ANML While we do our best for a fair comparison, there are several modifications made to the settings and architectures. In their original paper, OML and ANML do not maintain a full computation graph of the inner loop during meta-training to reduce the computation cost. Instead, they segment the training stream every five tasks and compute the -atadirr only within each segment. Since they do not backpropagate through a long chain of updates, the meta-training cost is reduced at the expense of optimizing with a suboptimal objective. However, since our Transformers can backpropagate through the whole episode, comparing it with such suboptimal configuration can be considered unfair. Therefore, in our experiments, we maintain a full computation graph of the inner loop for a fair comparison. In an robust to the choice of hyperparameters, we also set the learning
effort to make the baselines rate of the inner loop as a learnable parameter.
The model's classification output mechanism is also modified. In the original design, the weights of the output head are hard-assigned to individual classes. When learning a class, the weights corresponding to the class are randomly reinitialized, to prevent overfitting to the classes in meta-
15

---

training, Since we developed the class representation scheme in $3.3 inspired by the tokenization schemes in the language modeling domain, we use the same class representation for the baselines. Using this representation, the model can output token in the fixed vocabulary, and when a new class introduced, an unused token randomly assigned to it. The model can predict class by producing the corresponding token. Since the tokens are randomly assigned to a class, we do not need to reset the weights.
A.4 Transformers
In the early experiments, we observed that the (meta- ttraining often fails to converge, especially when the episode length becomes longer. After closer inspection of the attention pattern of the failure cases, we discovered that the attention tends to be uniformly distributed across all tokens, not transmitting any useful information to the next layer.
One possible reason is that our meta-loss (Eq, D is not easy to optimize. Given a test input, a well-trained Transformer should send some attention query from the test input to the keys from the training sequence of the corresponding task. However, since the meta-loss is simply a next-word prediction loss, it provides only high-level guidance without enforcing any specific attention pattern. Note that the final prediction layer and self-attention layers are two distinct mechanisms. When a Transformer is randomly initialized, it seems hard for it to notice that attending to specific part of the training stream is useful for predicting the target. Therefore, the attention layers become "dead," which is the best thing do if their inputs considered random noise.
As a solution, we add an auxiliary loss called attention loss to explicitly guide the attention pattern in the early stage of training. For each query of a test input In of task k, we maximize the log sum of its attention to the keys from all training examples of task k. In plain words, we guide the attention to be headed to task k's training sequence, but we do not care exactly which token is attended. We apply attention loss to half of the attention heads in each layer for the first 10K steps of (meta-) )training.
B Additional Experiments and Analyses B.1
Experiments with ResNet Encoder
In the main text, we present the experiments with a simple 5-layer CNN encoder. This is mainly because some baselines, e.g., ANML 44) require meta-training of the whole network, including the encoder. Using more sophisticated encoders, such as ResNet [].. will increase the computational cost, which is already high due to the meta- veta-gradient computation. Furthermore, it is hard to define the proper usage of the batch normalization [1), which is part of the ResNet architecture, in CL settings.
On the other hand, OML [6] and our Transformers do not update the encoder in the inner loop. Since the encoder is updated only in the outer loop, using ResNet with these methods does not violate the underlying assumptions of batch normalization. Here we provide the results with the ResNet- 18 encoder. The R8 architecture is slightly modified from the original version since it was originally designed for handling much larger images with 224 x 224 resolution. We replace the first 7 X 7 stride-2 convolution with 3 x stride-I convolution, remove the next 3 3 max-pooling, and reduce the number of features in all layers by half. For the efficient Transformers, we test Linear Transformer only since it generally achieves better scores than Performer. The results are presented in Table 3 The first three columns are the classification benchmarks, while the last two are regression benchmarks.
We find that using ResNet does not make a meaningful difference in the large benchmarks using CASIA or MS-Celeb-1M Since the images used in our experiments are small (32 x 32 resolution), the simple CNN encoder seems sufficient to deliver the necessary information to the upper components. In the case of Omniglot classification, the scores of all methods even degrade due more meta- overfitting.
B.2 Experiments with Combinatorial Class Representation
Although we introduced the combinatorial representation for classes in $.3 we simply keep the number of tokens for each class representation to in our main experiments. Table we present the results of the 20-task CASIA benchmark with different class representations. The default setting is
16

---

Table 3: Experiments with ResNet encoder in 20-task 5-shot settings.
Method
Omniglot
CASIA
MS-Celeb-IM
Completion
OML
24.8=2.2
3.2=0.1
42.5+0.2
0.1092#0.0002
OML (ResNet)
27.7#1.3
3.4
41.1=0.2
0.1092+0:0002
Transformer
13.7 #..6
0.4
30.0 #0.2
0.0999
+0,0002
Transformer (ResNet)
25.5
0.5 0.5=0.0
32.4 +0.2
0.1014 +0,0002
Linear TF
36.0#14
0.7=0.0
#0.3
0.1039 000000
Linear TF (ResNet)
63.6=2.1
0.7=0.0
34.9#0.1
0.1051 #0.0001
CASIA
MS-Celeb-IM
1.6
44
1.4
Erno (36)
Urrrr (%)
42
1.2
40
0.8
38
24M
SMM
10M
36M
2.4M
3.M
10M
36M
Model parameters
Model parmeeters
(a) Effect of scaling model size in 100-task 5-shot classification benchmarks.
Sine
Rotation
Completion
0.006
0.005
0.102
Eiror
Liror
Eiror
0.004
0.100
0.003
0.030
0.098
1.4M
5.5M
9.7M
3MM
2.5M
6.4M
HM
38M
2.6M
6.6
HM
39M
Model parameters
Model parameters
Model parameters
(b) Effect of scaling model size in 100-task 5-shot regression benchmarks.
Figure 2: Scaling behavior of Transformers.
represented as [V| 20 and C = 1, where V| is the vocabulary size, and C' is the class representation length. We additionally test two more settings with [V] 8, C = 2 and = C= 3. Note that the total number of unique class representations is [V]C which increases exponentially to C. We find that the classification errors are mostly similar regardless of the representation, although there a slight performance degradation as the C' increases. We suspected that it is caused by the increased sequence length. Since the class representations are taken as input by the Transformer, increasing C induces more computational cost. Nonetheless, we believe the idea of combinatorially representing class or a new concept is interesting and might have a useful application in the future.
Table 4: Transformers with combinatorial class representation.
Method
Representation
CASIA error (%)
Transformer
|V| 20, C =
0.4 10.0
0..0
Transformer
[V| 8, C 2
0.5
Transformer
VV =I=8 8, C = 3
0.6
17

---

B.3 Scaling Behavior of Transformers
Here we test how the model size of the Transformer related to performance. This experiment inspired by the scaling law of large language models [4], where the performance keeps increasing with the computational power. We test four different Transformer architectures whose number of parameters roughly ranges from 2M to 40M. The architecture details are summarized in Table 5 Note that we use the third architecture in other experiments.
In CL literature, which generally relies on SGD updates to incorporate new knowledge, increasing model size does not necessarily lead to better CL ability. On the other hand, as shown in Fig. 2a and Fig. 2b] the performance of Transformers consistently improves as more parameters are added. This is promising result since implies that we can simply achieve better scores by providing more computation and data. We speculate that scaling up the model size may be a necessary condition to solve more realistic and challenging scenarios.
Table 5: Transformer architecture configurations.
Method
Talayers
dgoode]
Theads
dbeads
dMLP
Transformer (S)
256
64
512
Transformer (M)
2
512
64
1024
Transformer
4
512
8
64
1024
Transformer (XL)
1024
16
64
2048
B.4 Forgetting Analysis
Fig. and Fig. 4sow the forgetting analysis 7 of 100-task CASIA classification and Sine regression benchmarks. For each task k, we measure its error increase after learning k k'(> k) tasks, relative to the error right after learning it. Each graph in Fig. 3 and Fig. shows the average of five consecutive tasks to save space. The last plot shows the average forgetting of all tasks.
Note that forgetting alone cannot be used as evaluation metric since it measures the relative change in performance throughout an episode. For example, a model scores zero accuracies in all tasks throughout the whole episode, the forgetting is measured as zero. Therefore, it should rather be used as an additional tool for analysis.
For the forgetting analysis, we meta-train the Transformer and Linear Transformer to also handle test examples in the middle the training stream. For each test example task k. we randomly sample the evaluation moment from the range [k: K], i.e., example can be evaluated at the end of any task's training as long as the corresponding task is learned. In both benchmarks, Transformers' forgetting is significantly lower than OML, demonstrating the effectiveness of our approach.
C
Computational Cost Analysis
First, we emphasize that it is hard define single unified measure to compare the computational cost. In some environments, the total memory consumption may be a hard constraint, while running time can be the top priority in others. Moreover, the actual cost vary dramatically depending the software implementation or the underlying hardware. Here we compare various aspects of the computational cost using our PyTorch (27) implementation NVIDIA A40 GPUs which have 48 GB of VRAM. We report the cost of meta-training with a batch size of 16.
The results of the Linear Transformer and Performer omitted due to the current limitation of our implementation. Theoretically, these efficient Transformers can far more efficient than the standard Transformer both in terms of memory and computation time. However, realizing such advantages while maintaining high GPU utilization requires sophisticated CUDA kernel programming and cannot be implemented with simple Py PyTorch code. Therefore, explicitly compute the attention matrix, the same as the standard Transformer, to save implementation time. This suboptimal implementation results in slightly worse memory and computation time compared the standard Transformer.
Table compares the number of parameters, GPU memory consumption (in GB), meta-training speed (in episodes per second), and the classification error the CASIA benchmark. Remarkably,
18

---

OML
Linear TF
Transformer
Task 1-5
Task 6-10
Task 11-15
Task 16-20
Task 21-25
Forgotting (l))
10.0
7.5
5.0
2.5
0.0
20
40
80
100
80 100
20
60
80
100
20
40
80
100
20
60
100
Leamed tasks
Learned tasks
Learned tasks
Learned tasks
Learned tasks
Tusk 26-30
Task 31-35
Task 36-40
Task 41-45
Task 46-50
Forgeting (5)
7.5
5.0
2.5
0.0
40
60
80
100
20 40 30 60 70 s0 90100
40 50 60 70 80 90100
40 50
60
70
80 90 100
50 60
70
80
90 100
Leamnd tasles
Leamed tasks
Learned tacks
Learned tusks
Leamed tasks
Task 51-55
Task 56-60
Task 61-65
Task 66-70
Task 71-75
Forgeting (%)
7.5
2.5
50
60
70
90
100
60
70
80
90
100
60
70
80
100
70
90
100
75
80
85
90
95 100
Learned
Task 76-80
Task 81-85
Task 86-90
Task 91-95
Task 1-100
Fongeting 1%0
7.5
5.0
2.5
0.0
75
90
95
100
95
100
90
100
95
100
40
60
100
Leamnd
Leamed
Learned
Learned
Leamed
Figure 3: Forgetting analysis of 100-task CASIA benchmark.
even the largest Transformer consumes less memory than OML, the most memory-efficient baseline. Although OML has much fewer parameters than Transformers, they consume far more GPU memory due to the meta-gradin peta-gradient computation. If we scale up OML to have similar number of parameters with the default Transformer architecture, which is referred to as OML (XL) in Table the memory consumption increases even further, requiring four A40 with data parallelism.
Transformers are also superior in terms of (meta-)training speed. Since they can process all the examples in an episode parallel, they can utilize GPUs more effectively. On the other hand, SGD-based baselines need to sequentially process each example and also sequentially backpropagate in the reverse direction. Therefore, Transformers have several times faster training speed compared to the baselines.
19

---

OML
Linear TF
Transformer
Tusk 1-5
Task 6-10
Tusk 11-15
Task 16-20
Task 21-25
Foggoting
0.02
0.00
20
40
60
80
100
20
40
60
80
100
20
40
60
80
100
20
40
60
80
100
20
40
60
80
100
Leamed tusles
Leamed tanks
Learned tasks
Learned taskss
Leamed tasks
Task 26-30
Task 31-35
Task 36-40
Task 41-45
Task 46-50
Forgeting
0.06
0.04
0.02
0.00
40
60
80
100
40 50 70 90100
40 50 60 90 100
50 60
70
80
90 100
60 70 90
100
Leamed
Leamed
Task 1-55
Task 56-60
Task 61-65
Task 66-70
Task 71-75
Forgotinng
0.06
0.04
0.02
100
Leamnd
Leamed racks
Learned tacks
Learned tasks
Leamed tasks
Task 76-80
Task 81-85
Task 86-90
Task 91-95
Task 1-100
Fonetting
0.06
0.04
0.02
100
100
100
Leamed tasks
Learned tasks
Learned tasks
Learned tasks
Learned tasks
Figure Forgetting analysis of 100-task Sine benchmark.
Table 6: Computational cost comparison.
Episode
Method
Parameters
GPU memory
Training speed
CASIA emoor
OML.
1.5M
10.7 GB
51.2 ep/s
3.2
OML (XL)
11.5M
33.7 GB
3.5 ep/s
1.9+0.2
ANML
2.0M
18.4 GB
6.6 ep/s
4.3 4.3
Transformer (S)
2.4M
4.3 GB
278.5 ep/s
0.6
20-task 5-shot
Transformer (M)
5.9M
4.7 GB
252.9 ep/s
0.5
Transformer
10.1M
5.2 GB
201.5 ep/s
Transformer (XL)
36.0M
6.2 GB
138.0 ep/s
t0.0
OML
1.5M
43.7 GB
9.40 ep/s
6.8+0.9
Transformer (S)
2.4M
22.0
49.69 ep/s
1.5
100-task 5-shot
Transformer (M)
5.9M
25.8 GB
38.78 ep/s
1.1 t0.0
Transformer
10.1M
31.5 GB
26.80 ep/s
1.0#0.0
Transformer (XL)
36.0M
41.4 GB
16.34 cp/s
0.8+0.0
20


Reviews:
[{"review_id": "sc40qcTTdb", "reviewer": "Reviewer_N8DE", "summary": "This work looks at treating the meta-continual learning problem as instead a sequence modeling problem. Instead of traditional approaches that train a model with an inner loop and then compute a meta-gradient in the outer loop, they replace the inner loop with just inference in a sequence model. The meta gradient step is replaced by a normal gradient step that is taken based on the sequence seen by the model. In order to make this approach work, the paper uses transformers that (1) use causal attention (ie the information transfer only happens forward in time) (2) make use of kernel-based transformers to address the quadratic memory usage of traditional transformers. They test their approach on 4 classification datasets and 3 regression problems.", "questions": "- Do you have results showing how much the performance drops off with efficient transformers compared to the standard one as the data size increases? \n- Do you see different performance trends when you increase data per task as opposed to number of tasks?", "limitations": "The paper discusses the performance tradeoff that occurs with using efficient transformers to handle long sequences. I think this is a sizable problem, as the method would likely start underperforming or be unscalable when given more data per sequence. They do point out, however, that since they are using standard transformer models, as progress is made with those architectures, that should map well to their approach.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "- The results show that there is potential for this approach of treating meta-continual learning as sequence modeling. The performance is generally competitive or superior to prior approaches.\n    \n- The presented method is memory efficient and fast compared to previous MCL approaches, at least when dealing with typical MCL benchmarks.\n    \n- The paper is well presented and readable, with helpful figures.", "weaknesses": "- The paper does mention that their model tends to overfit on the CIFAR-100 dataset. The paper hypothesizes that this is because task diversity is lower, but this is not verified experimentally.\n- The novelty is a bit limited, as the paper simply applies transformers to this task of MCL, but still good enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the encouraging and insightful comments.\n\n#### **Meta-Overfitting and the Task Diversity**\n\nIn the context of classification benchmarks, the term “task diversity” mostly refers to the number of classes. We apologize for not using a clearer description. At the top of Table 1, we highlighted the number of classes in each benchmark to show that the number of classes is strongly correlated with the degree of meta-overfitting.\n\n#### **Novelty**\nWe respectfully argue that the main novelty of our work is the reformulation of MCL as a conventional sequence modeling problem. The use of Transformers is just a specific example. Our work opens the possibility that any sequence model that will come in the future can be used as an MCL solver as it is.\n\n---\n### Questions\n#### **Efficient Transformers vs. Standard Transformers with More Data**\nYes, we do have results. In Appendix C, we present a lot more experiments analyzing the various aspects of our approach. Among them are the experiments with longer episodes (Appendix C.2). We test the standard Transformer, Linear Transformer (the best efficient Transformer), and OML (the best baseline). The errors due to longer episodes increase in the order of Transformer < Linear Transformer < OML.\n\n#### **Increasing Examples per Task vs. Increasing the Number of Tasks**\nThis is an interesting question. To check the trends when the number of examples per class increases, we additionally tested OML, Transformer, and Linear Transformer in 20-task 25-shot settings. The results are summarized below with 20-task 5-shot and 100-task 5-shot results for comparison. Every score represents an error (the lower, the better).\n\n[CASIA Classification (%)]\n| Method | 20-task 5-shot | 20-task 25-shot | 100-task 5-shot |\n|---|---:|---:|---:|\n| OML | $3.2^{\\pm 0.1}$ | $2.4^{\\pm 0.0}$ | $6.8^{\\pm 0.9}$ | \n| Transformer | $0.4^{\\pm 0.0}$ | $0.3^{\\pm 0.0}$ | $1.0^{\\pm 0.0}$ |\n| Linear TF | $0.7^{\\pm 0.0}$ | $0.5^{\\pm 0.0} $| $2.3^{\\pm 0.1}$ |\n\n[Rotation]\n| Method | 20-task 5-shot | 20-task 25-shot | 100-task 5-shot |\n|---|---:|---:|---:|\n| OML | $0.971^{\\pm0.046}$ | $0.994^{\\pm0.004}$ | $0.990^{\\pm0.008}$ |\n| Transformer | $0.040^{\\pm0.001}$ | $0.033^{\\pm0.001}$ | $0.031^{\\pm0.001}$ |\n| Linear TF | $0.075^{\\pm0.002}$ | $0.069^{\\pm0.005}$ | $0.047^{\\pm0.002}$ |\n\n[Completion]\n| Method | 20-task 5-shot | 20-task 25-shot | 100-task 5-shot |\n|---|---:|---:|---:|\n| OML | $0.1092^{\\pm0.0002}$ | $0.1079^{\\pm0.0002}$ | $0.1087^{\\pm0.0001}$ |\n| Transformer | $0.0999^{\\pm0.0002}$ | $0.0973^{\\pm0.0002}$ | $0.0989^{\\pm0.0001}$ |\n| Linear TF | $0.1039^{\\pm0.0003}$ | $0.1013^{\\pm0.0002}$ | $0.1084^{\\pm0.0001}$ |\n\nWe found no significant difference in trends. The order of performance is consistent in all experiments: Transformer > Linear Transformer > OML. In all cases, the 20-task 25-shot setting scores better than other configurations since it has a small number of tasks and a large number of shots."}, {"review_id": "8xFEwhGY5x", "reviewer": "Reviewer_FrpD", "summary": "This paper applies transformers and their efficient variants as sequence models to the problem of meta-continual learning. More specifically, instead of running gradient descent on a stream of training data, this paper trains transformers to do in-context continual learning over the data stream. It then compares these transformer-based approaches to three other MCL baselines on regression and image classification tasks.", "questions": "1. Why is the 100-task MCL only tested on CASIA?\n2. The authors mentioned that for OML and ANML, full backprop is used instead of truncated backprop for fair comparison. But truncated backprop was what made these methods scalable to long inner loops, so for the 100-task MCL I think it’s okay to use the truncated version.\n3. One advantage of OML that was highlighted in the original paper is that it’s complementary to other CL methods. Can transformer-based MCL methods be combined with other CL methods too?\n", "limitations": "As mentioned in the weakness, the main limitation of this method is that it can’t be used for continual learning in practice. I think this paper can be dramatically improved if the authors can run experiments where meta-test episodes are very long and use different distribution and episode length for meta-train and meta-test. It would also be more convincing to compare with more competitive CL baselines.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "Originality: There have been many works on using sequence models for meta learning, and many works on meta-continual learning. However, to my knowledge this is the first paper where sequence model based meta-learning is tested for continual learning capability.\nQuality: code included, good reproducibility; rich content in the Appendix, many details and analysis.\nClarity: well-written and easy to follow. Many schematic illustrations that make the core ideas clear.\nSignificance: in the long run, being able to leverage data and compute to learn a CL algorithm instead of human-engineering one is an important topic.", "weaknesses": "The main weakness is that the experiments have not convincingly demonstrated the proposed method is practically useful as a continual learning method. \n\nOne reason is its scalability to longer sequences. If I understand correctly, during the meta-test, almost all methods are evaluated on episodes of only 20 x 5 = 100 examples. The only experiment that’s slightly longer is Table 2 with 100 x 5 = 500 examples, and only one baseline was included. In comparison, OML showed it can scale to 1000 examples and ANML to 9000 steps. But frankly even these are too short to be useful for continual learning, which is supposed to handle much longer streams than normal deep learning settings. How would this method fare if at test time the episodes are much longer?\n\nThe second reason is its generalization ability to out-of-distribution data during meta-test. All evaluations in this work assume the same distribution and the same episode length for meta-train and meta-test. However, in real CL applications, one can’t know beforehand the distribution of future data and how long the stream would be. One advantage of SGD-based continual-meta learning approach is that the inner loop uses SGD for optimization, so even if the meta-test data is OOD, SGD can still learn. Can transformers still learn new data if they are OOD?\n\nThe third reason is that it’s not compared to any competitive conventional CL baselines. Even if the authors only intend to show competitiveness among MCL approaches, it’s still good to have other types of CL approaches for reference. In addition, there are other meta-learning methods that could be competitive baselines but not referenced, for example [1, 2]. Although these meta-learning methods are not particularly designed for continual learning, they have both demonstrated continual learning capabilities. In particular, [1] proposed an optimization-based approach that can extend to arbitrarily long inner-loops and trained a continual learning optimizer with it; [2] notably also applies a transformer as a meta learner over episodes for in-context RL.\n\n[1] Meta-Learning with Warped Gradient Descent https://arxiv.org/pdf/1909.00025.pdf \n[2] In-context Reinforcement Learning with Algorithm Distillation https://openreview.net/forum?id=hy0a5MMPUv", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful review and the acknowledgment of the various strengths of our work. We believe there are some differences in viewpoints between ourselves and the reviewer concerning the relationship between CL and MCL settings. We hope our following responses help close the gap.\n\n#### **Continual Learning vs. Meta-Continual Learning**\nWe think the relationship between CL and MCL directly corresponds to the relationship between standard learning and meta-learning. Therefore, CL and MCL are two distinct learning frameworks, each with its own set of assumptions. As meta-learning methods cannot be evaluated in the standard learning setting (there is no other episode to meta-train on), MCL methods should also be compared in the MCL settings as well. Therefore, we present our method as an effective MCL approach rather than a CL method, as prior works on MCL (e.g., OML, ANML) did.\n\n#### **Scalability of Learning Paradigms Based on Meta-Learning**\nThe reviewer’s concern about the scalability of MCL methods is a valid point, and we agree that scaling up MCL approaches is an important future work. However, we think the scalability issue does not stem from individual methods but the general meta-learning setting (not limited to MCL). The two-level optimization (inner loop + outer loop) of meta-learning is inevitably resource-intensive, and the size of the inner loop has to be kept small to fit the whole optimization process within an available computational budget.  Therefore, existing meta-learning (and MCL) research mostly assumes rather small-scale episodes.\n\nGiven this perspective, more advances in hardware may be a prerequisite to solving the scalability issue of meta-learning, just like the remarkable progress of large language models (LLMs) enabled by the development of more powerful GPUs and distributed training system. It is important to note that none of the MCL methods (including the baselines) tested in this work has a theoretical limit on the size of the problem it can handle. Given a more computational budget, all the methods can be applied to bigger problems.\n\nIn addition, we believe our sequence modeling approach has better potential in terms of scalability compared to SGD-based approaches for the following reasons. First, efficiently handling longer sequences is one of the top-priority topics in sequence modeling research, often motivated by the potential applications to LLMs. Our approach can directly benefit from the advances in this field of research. Second, both Transformers and efficient Transformers can compute the inner loop in parallel, unlike the SGD-based approaches that require sequential computation of the inner loop. Therefore, it is easier to take advantage of new hardware with massive parallelism.\n\n#### **Handling OOD Data in Meta-Test**\nAlthough it is desirable to generalize OOD data in the meta-test phase, it is not the primary goal of either meta-learning or meta-continual learning. The episodes in both meta-training and meta-test sets are generally assumed to be drawn from the same distribution, just like the examples in both training and test sets are assumed to be drawn from the same distribution in standard learning settings. This assumption is also stated in widely circulated works on meta-learning, such as MAML (Finn et al.) or \"Meta-Learning in Neural Networks: A Survey\" by Hospedales et al.\n\nIt is also debatable whether SGD-based MCL approaches are truly capable of handling OOD data. As pointed out by the review, SGD update will surely enable learning new data even if it is OOD. Our main concern is whether the SGD-based approaches can effectively prevent the forgetting of such OOD data. In the case of OML, for example, it meta-trains an encoder to produce special features that are robust to forgetting from SGD. However, there is no guarantee that the encoder is still effective for OOD data.\n\n#### **Comparison with Conventional CL Methods**\nNot only in our work but also in OML and ANML papers, CL baselines are NOT compared because CL methods cannot achieve meaningful scores in MCL settings. As we explained above about the scalability issue, the current MCL settings generally focus on few-shot settings, which are unsuitable for general CL algorithms. Meanwhile, we will thoroughly discuss [1] and [2] as the review suggested.\n\n---\n### Questions\n#### **100-Task Experiments**\nWe provided more 100-task experiments in Appendix C.2. Please understand that we had to select only a few key results for the main text due to the page limit. Appendix C also contains many other experiments.\n\n#### **Truncated Backprop for OML and ANML**\nYes, one may use truncated backprop to improve the scalability of OML or ANML. However, as truncated backprop is an approximation of the full backprop, it can cause larger errors in OML and ANML. Thus, we chose to use full backprop in all methods.\n\n#### **Combination with Other CL Methods**\nAlthough it depends on the specific formulation of individual CL methods, the model updates of our approach are based on the forward pass of a sequence model (instead of SGD) and thus would be incompatible with SGD-based CL methods."}, {"review_id": "JpocBaRoPI", "reviewer": "Reviewer_3kpM", "summary": "The paper redefines Meta-Continual Learning (MCL) as a sequence learning problem. Following this definition, the paper proposes a Transformer-based meta continual learner. The method is evaluated on several classification and regression tasks.", "questions": "- LINE 270: use pretrained model. Is this model trained from scratch on each episode?\n- LINE 303: “task identities are not provided”. However, only the data from a specific task is provided, so it’s equivalent to having task identities. \n- how robust is this algorithm to the data order?\n- it is unclear to me what is the computational cost of the method at inference time compared to static networks.", "limitations": "-", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, I think the paper is well written and does a good job justifying the approach.\nThe idea of casting MCL as a sequence learning problem is novel (at least in CL) and interesting.", "weaknesses": "baselines: the paper mentions that Prototypical Networks could be used in MCL. It would be interesting to compare against them.\nIn general, the baselines are a bit limited. It would be interesting to compare against other methods that keep all the data in memory, like Transformers do.\n\nminor comments:\n- line 40-41 claims that sequence learning exploits in-context learning (ICL). However, ICL is a property of large pretrained models, I don’t think it applies to this setting\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful and insightful comments. \n\n#### **Baselines**\nSince Prototypical Network (PN) and GeMCL cannot be applied to domains other than classification, we prioritized testing other baselines that can perform both regression and classification in our initial submission. As suggested, we have tested the Prototypical Network (PN) and also GeMCL (it is a relatively simple extension of PN) on the classification benchmarks and found an interesting trend. We will include the results in the final version. In the following, we compare the classification errors of PN and GeMCL in the 20-task classification benchmarks.\n\n| Method | CIFAR-100 | | CASIA | | MS-Celeb-1M | |\n|---|---:|---:|---:|---:|---:|---:|\n| | Meta-train | Meta-test | Meta-train | Meta-test | Meta-train | Meta-test |\n| PN | $0.0^{\\pm0.0}$ | $76.6^{\\pm0.3}$ | $0.2^{\\pm0.0}$ | $0.4^{\\pm0.0}$ | $32.5^{\\pm0.1}$ | $33.6^{\\pm0.1}$ |\n| GeMCL | $0.0^{\\pm0.0}$ | $76.6^{\\pm0.4}$ | $0.2^{\\pm0.0}$ | $0.4^{\\pm0.0}$ | $32.1^{\\pm0.1}$ | $33.3^{\\pm0.2}$ |\n| OML | $0.6^{\\pm0.1}$ | $89.9^{\\pm0.4}$ | $2.8^{\\pm0.1}$ | $3.2^{\\pm0.1}$ | $41.8^{\\pm0.3}$ | $42.5^{\\pm0.2}$ |\n| ANML | $0.4^{\\pm0.1}$ | $88.1^{\\pm1.4}$ | $3.7^{\\pm0.5}$ | $4.3^{\\pm0.5}$ | $43.8^{\\pm0.3}$ | $44.8^{\\pm0.4}$ |\n| MetaFSCIL | $34.5^{\\pm2.1}$ | $82.1^{\\pm0.3}$ | $12.0^{\\pm0.4}$ | $12.2^{\\pm0.5}$ | $57.6^{\\pm0.3}$ | $57.8^{\\pm0.2}$ |\n| Transformer | $0.0^{\\pm0.0}$ | $82.8^{\\pm0.8}$ | $0.3^{\\pm0.0}$ | $0.4^{\\pm0.0}$ | $29.1^{\\pm0.2}$ | $30.0^{\\pm0.2}$ |\n| Linear TF | $0.1^{\\pm0.1}$ | $83.4^{\\pm0.5}$ | $0.4^{\\pm0.0}$ | $0.7^{\\pm0.0}$ | $31.1^{\\pm0.3}$ | $32.4^{\\pm0.3}$ |\n| Performer | $0.0^{\\pm0.0}$ | $82.9^{\\pm0.3}$ | $0.5^{\\pm0.0}$ | $0.7^{\\pm0.0}$ | $32.5^{\\pm0.5}$ | $33.7^{\\pm0.2}$ |\n\nDue to their simplicity, PN and GeMCL are robust to meta-overfitting and significantly outperform all other methods in smaller datasets such as CIFAR-100. However, in the CASIA benchmark, where a larger number of classes reduces the effect of meta-overfitting, their performance is on par with Transformers. Finally, in the most challenging MS-Celeb-1M benchmark, they fall behind Transformers. We suspect that PN's simple algorithm of averaging the embeddings is not sufficient to integrate the information of the training stream if the task distribution becomes more complex.\n\n#### **In-Context Learning (ICL)**\nICL is often introduced as an *emergent* ability of pretrained LLMs. However, considering the original description in the GPT-3 paper, the definition of ICL does not need to be restricted to pretrained LLMs. One may explicitly train an arbitrary sequence model to perform ICL, which is exactly what we do in this work.\n\n---\n### Questions\n\n#### **Line 270**\nThank you for pointing this out. There is absolutely no SGD update in an inner loop. The parameters of both CNN and Transformer are fixed inside each episode, and only the outer loop updates the parameters. We will make this clear in an updated version.\n\n#### **Line 303**\nWe would like to clarify our use of terminology. In the statement 'task identities not being provided,' we referred to the distinction between task-aware and task-agnostic CL. Similar to prior works on MCL, such as OML and ANML, we followed task-agnostic CL settings. \n\nNote that the term \"task\" has different meanings in meta-learning and MCL literature. In the meta-learning literature, a meta-training (or meta-test) set is often said to hold multiple *tasks*, but we use the term “episode” to refer to the corresponding concept in MCL. In MCL, each episode is a CL problem, which consists of $K$ *tasks* randomly sampled from the set of possible tasks. During the test phase of an episode, the model receives an input and is tasked to infer the corresponding output without knowing which task it belongs to. Since the input can belong to any of the $K$ tasks, it is not equivalent to having task IDs. We will improve the description in an updated version.\n\n#### **Robustness to Data Order**\nThis is an intriguing question. Unlike conventional (meta-)continual learning methods, we can rephrase the question of “how robust the MCL method is to the data order” as “how robust the sequence model’s ICL capability is to the order of in-context examples.” Thus, the robustness depends on which sequence model is used and how the meta-training set (i.e., a training set from the perspective of sequence modeling) is constructed. We believe as long as Transformer is used as the sequence model and the meta-training set sufficiently covers diverse data orders, our approach should be robust to the data order.\n\n#### **Computational cost**\nSince our main idea is to use generic sequence models as MCL methods, the computational cost depends on the choice of the sequence model. In this work, we tested the standard Transformer, Linear Transformer, and Performer, whose computational costs per test example are $O(T)$, $O(1)$, and $O(1)$, respectively, where $T$ is the number of training examples in an episode. Note that these are the same as the costs of the sequence models inferring one token, given a context of length $T$."}, {"review_id": "tYYctgMlzs", "reviewer": "Reviewer_JJ3m", "summary": "The paper proposes to formulate continual learning as a sequence modeling problem. This new formulation views the model's learning process with continual session data as the forward pass of a sequence model, such as a Transformer, rather than relying on backpropagation. Specifically, keys and values within the Transformer can be interpreted as the internal state of a memory module in continual learning. To optimize the sequence model's parameters for continual learning problems, meta-continual learning is employed. This involves meta-learning the Transformer model using episodic learning. To address the computational and memory costs associated with long sequence modeling, efficient transformers, including the Linear Transformer and Performer, are utilized. The proposed approach's effectiveness is substantiated through performance evaluations across multiple benchmark datasets.", "questions": "Please see questions and suggestions in Weaknesses. ", "limitations": "The paper include the limitations of the paper but fails to mention potential negative social impacts.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Formulating continual learning as the problem of sequence modeling is very novel to me, and the author provides a very detailed explanation of why that two problems can be connected with each other under the framework of meta-continual learning. For example, the similarities between the inner loops of the two problem formulations are illustrated clearly in Algorithm 1 and 2. Furthermore, the paper conceptually outlines the connection between dynamic-architecture CL approaches and standard Transformers.\n\n* The introduction is written in a very clear manner, making it easy for readers to grasp the central ideas and the contributions of the study.", "weaknesses": "* The paper doesn't adequately distinguish between continual learning and meta-continual learning. While the text mentions that meta-continual learning aims to automatically learn how to continue learning as opposed to standard continual learning by manual design, it fails to mention the necessity of a large-scale offline dataset to create a meta-training dataset. In contrast, standard continual learning does not require such an offline dataset for training.\n\n* The explanation of meta-continual learning lacks clarity. I recommend that the authors define and explain terms such as episodes, meta-train, and meta-test more explicitly. Particularly, the concepts of a support set and query set, which are typically used under the meta-learning framework, are not mentioned in this paper.\n\n* An important baseline in meta-continual learning, \"Wandering within a world: Online contextualized few-shot learning,\" presented at ICLR 2021, is overlooked in the related work section.\n\n* While the text acknowledges the limitations of SGD-based meta-continual learning, such as the high cost of second-order gradients and scalability issues, it doesn't reference previous work on efficient meta-learning approaches. e.g. [1].  I would suggest the authors incorporate references to efficient meta-learning techniques and add a dedicated section comparing them to transformers in terms of computational and memory costs.\n\n    [1] Large-Scale Meta-Learning with Continual Trajectory Shifting. ICML 2021\n\n\n* The presentation of the experiment results could be improved. A central challenge of continual learning is the balance between catastrophic forgetting and rapid adaptation to new knowledge. However, Table 1 doesn't illustrate the proposed method's effectiveness related to these two criteria. Therefore, I suggest the authors represent the experiment results using a plot charting the number of continual sessions against average accuracy. Such a plot is typically employed in previous works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed and insightful comments. \n\n#### **Continual Learning (CL) vs. Meta-Continual Learning (MCL)**\nAs pointed out by the review, a large meta-training dataset is one of the fundamental assumptions of meta-learning and its variants, such as meta-continual learning. We will make sure to state this assumption clearly in the final draft.\n\n#### **Explanation of MCL**\nWe will improve the description of MCL and terminologies following the suggestions. We did not use the terms “support set” and “query set” because (i) the concept corresponding to the support set in MCL is not a set but a stream, and (ii) prior works in MCL (e.g., Javed et al., Beaulieu et al.) did not use them either. But, it is a good idea to draw a connection to the meta-learning terminologies for better understanding. We will update the description accordingly.\n\n#### **Wandering Within a World (Ren et al.)**\nWe thank the reviewer for introducing an important related work. However, we find this work is closer to continual meta-learning (CML) rather than meta-continual learning (MCL). These two learning frameworks sound very similar, but they are fundamentally different in terms of their underlying assumptions and objectives. Given the significant confusion they have engendered within the research community, we put a considerable amount of effort into Appendix A, where we summarize and compare various learning frameworks using visual illustrations and formal algorithms.\n\nBorrowing the expression from Ren et al., the MCL setting is *episodic*. There are multiple episodes in the meta-training set, and each episode consists of a series of tasks. For each episode, an independent model is produced, which does not need to perform well on tasks from other episodes. In CML, on the other hand, there is no meta-training / meta-test distinction, and there is only one episode. Please refer to Appendix A for more detailed illustrations. We will include Ren et al. in the prior works on CML. \n\n#### **Continual Trajectory Shifting (Shin et al.)**\nThis is also an important related work. There has been little work on applying efficient meta-learning techniques to the MCL domain, and we believe this direction would be an interesting research topic in the near future. In Appendix C.6, we discussed the first-order approximation in the MCL setting. We will expand this section to more comprehensively cover various efficient meta-learning methodologies such as Shin et al.\n\n#### **Forgetting Analysis**\nIn Appendix C.7, more specifically in Figures C.10 and C.11, we presented a detailed analysis of forgetting. Please understand that the sizes of the plots are too large to be in the main text, considering the page limit.\n\n#### **Potential Negative Social Impacts**\nSince our work is not directly related to applications that are immediately deployable in society, we believed that the potential negative societal impacts would be minimal. However, we will address potential negative impacts in the final version."}]

Author rebuttal:

Meta-review: This paper establishes a connection between continual learning and sequence modeling. It is the first work applying sequence modeling to meta continual learning; the idea is very interesting and shows practical benefit in both model simplicity and empirical performance. In addition, the paper is well written and the authors provide code for reproducibility.

After rebuttal, the authors addressed most of the concerns except the ones on scalability to longer sequences and geralizability to OOD data. Considering continual learning usually involves pretty long sequences which may outpace the capacity of mainstream Transformer-based sequence models, the concerns raised by reviewers are very crucial and keen to the weaknesses which trigger careful treatment. Currently, authors have NOT addressed the concerns sufficiently. By weighing the novel solution to meta continual learning more, this paper could be accepted. But authors must further highlight the limitations of their approach in the long or even endless continual learning settings and in real OOD scenarios. Otherwise, this will mislead future readers as if the paper had solved these two important problems.

Since the merits outweigh the flaws, I recommend the paper be accepted, contingent on the required revision.
Decision: Accept (poster)