SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Soochan Lee
soochan.lee@vision.snu.ac.kr
Jaehyeon Son
Seoul National University sjh9876@snu.ac.kr
Gunhee Kim
Seoul National University gunhee@snu.ac.kr
In this work, we aim to establish a strong connection between two significant bodies of machine learning research: continual learning and sequence modeling. That is, we propose to formulate continual learning as a sequence modeling problem, allowing advanced sequence models to be utilized for continual learning. Under this formulation, the continual learning process becomes the forward pass of a sequence model. By adopting the meta-continual learning (MCL) framework, we can train the sequence model at the meta-level, on multiple continual learning episodes. As a specific example of our new formulation, we demonstrate the application of Transformers and their efficient variants as MCL methods. Our experiments on seven benchmarks, covering both classification and regression, show that sequence models can be an attractive solution for general MCL.<sup>1</sup>
Continual learning (CL) refers to the ability to learn from a non-stationary stream of data. A CL data stream is typically a series of distinct tasks, and the model cannot access previous data while learning the current one. For example, in CL benchmarks based on image classification, each task typically consists of a small subset of classes. Since the model's access to the past data is restricted, standard training with stochastic gradient descent (SGD) consistently overwrites previously learned knowledge, which often leads to catastrophic forgetting. As learning usually implies SGD updates in the deep learning literature, most CL approaches try to combat catastrophic forgetting while relying on SGD to update model parameters. However, using SGD is generally not a necessity of CL; one may choose any update rule other than SGD.
In this regard, we point out that CL is inherently a sequence modeling problem. The basic structure of CL, which trains a model on a training stream  $((x_{1},y_{1}),\dots ,(x_{T},y_{T}))$  and evaluates it on a test set  $\{(x_n,y_n)\}_{n}$ , can be considered as predicting the next token  $y_{n}$  of the sequence  $(x_{1},y_{1},\dots ,x_{T},y_{T},x_{n})$ . From this perspective, the CL process and the test-time inference are a sequence model's forward pass, with no SGD update. Instead, we can train the sequence model at the meta-level with a set of CL episodes. This meta-continual learning (MCL) framework is technically equivalent to conventional sequence model training, where a sequence model is trained on a set of sequences. Therefore, we argue that any recent and future advances in sequence models can be directly applied to MCL research. As a precaution, our approach should not be confused with continually learning a sequence model [40, 5], which is a conventional CL setting (not MCL) where each datapoint  $x$  is a sequence.
As a particular instance of our new framework, we test Transformers [43], which are considered as the current state-of-the-art sequence model. In the past several years, Transformers have revolutionized various sequence modeling tasks such as language modeling [28, 29, 6], translation [43, 1], speech recognition/generation [30, 44], and protein analysis [33, 31], to name a few. When Transformer-based language models (LMs) are scaled up and trained with a massive collection of text data, they
1Code is available at https://github.com/soochan-lee/cl-as-seq
exhibit an interesting ability called in-context learning [6]. Within a forward pass, they can learn new patterns and knowledge from the previous tokens in the current context and apply them to solve new tasks. Our framework maximizes this in-context learning ability of sequence models through meta-training and applies it as a solution to CL.
An immediate question may come to one's mind: if Transformer is a parallel architecture that can attend to all the training data at test time, is this even a valid CL setting? Borrowing the words of Katharopoulos et al. [17], Transformers with causal attention can be considered as recurrent neural networks (RNNs; 13). We can think of a Transformer's attention keys and values as the internal state of a recurrent model. When a new token comes in, this state is updated to include new key-value pairs. At test time, the Transformer queries the key-value pairs in the internal state, not the raw training data. Therefore, with proper implementation, using Transformers does not violate the assumptions of CL.
One drawback of standard Transformers is that the cost of handling each example grows linearly along with the internal state. Therefore, the computational complexity of handling an episode of length  $T$  becomes  $\mathcal{O}(T^2)$ . Fortunately, there is a considerable amount of work on efficient Transformers [42], which have sub-quadratic complexity. Especially, the efficient Transformers with linear  $\mathcal{O}(T)$  complexity maintain a constant computational cost to handle each token. In this work, we test Linear Transformer [17] and Performer [8] as such examples.
By conducting extensive experiments on seven diverse benchmarks covering both classification and regression tasks, we demonstrate that Transformers, including the efficient variants, have great potential as a general MCL approach, especially in large-data regimes. Furthermore, it is important to note that our approach is not limited to Transformers. By offering a novel perspective on MCL, we open up the possibility for other sequence models, including those that may be developed in the future, to be applied to this challenging problem.
Lastly, our approach offers a unique advantage in terms of biological plausibility. Despite its universality in modern deep learning, backpropagation has long been criticized for being biologically implausible [36, 26, 21, 12]. This is due to the lack of convincing proof to suggest that biological neurons perform the complex backpropagation process of (i) storing their activations during the forward pass, (ii) computing the error derivatives, and (iii) transmitting them backward. Our formulation is free from such criticism since the learning process is a simple forward pass of the sequence model. Only the meta-level optimization, which would have been performed by the evolutionary process in nature, is replaced by more efficient backpropagation.
2.1 Meta-Continual Learning (MCL)
Due to the additional layer of complexity from meta-level optimization, compounded by the absence of a standardized lexicon, there exists a high likelihood of confusion in discerning the settings of MCL and related domains. Due to the limited space, this section summarizes only the most relevant works. For a comprehensive comparison of MCL with other fields, such as continual meta-learning, we recommend referring to [39].
Continual Learning (CL). A continual learning episode  $\mathcal{D}$  consists of a stream of training data  $\mathcal{D}^{\mathrm{train}} = ((x_1, y_1), \dots, (x_T, y_T))$  and a test set  $\mathcal{D}^{\mathrm{test}} = \{(x_1, y_1), \dots, (x_N, y_N)\}$  where  $x_* \in \mathcal{X}$  and  $y_* \in \mathcal{Y}$  are the input and target variables. The data in the training stream can only be accessed one at a time, and it is not possible to access previously seen data. If we assume a buffer that temporarily holds each task, this formulation can also cover the offline CL settings where the stream is segmented by tasks rather than examples. The training stream is generally assumed to be a concatenation of  $K$  task streams, i.e.,  $\mathcal{D}^{\mathrm{train}} = \mathcal{D}_1^{\mathrm{train}} \oplus \dots \oplus \mathcal{D}_K^{\mathrm{train}}$ , each of which is a stationary sequence of data sampled from a distinct distribution. The test set is the union of the task-specific test sets:  $\mathcal{D}^{\mathrm{test}} = \mathcal{D}_1^{\mathrm{test}} \cup \dots \cup \mathcal{D}_K^{\mathrm{test}}$ . The objective of the CL algorithm is to continually learn a good model  $f_\theta : \mathcal{X} \to \mathcal{Y}$  from the stream by optimizing parameter  $\theta$ . For example, in the popular CL benchmarks of image classification,  $f_\theta$  is a classifier that takes an image  $x \in \mathcal{X}$  as input to output its class  $y \in \mathcal{Y}$ .
Meta-Continual Learning (MCL). Rather than manually crafting a CL algorithm, MCL aims to learn how to continually learn. More formally, we introduce the concept of continual learner  $H_{\eta}:(\mathcal{X}\times \mathcal{Y})\times (\mathcal{X}\to \mathcal{Y})\to (\mathcal{X}\to \mathcal{Y})$ , which is a functional that takes a data point  $(x_{t},y_{t})$  and an old model  $f_{\theta_{t - 1}}$  as inputs and produces an updated model  $f_{\theta_t}$ . For convenience, we additionally
define another functional  $G_{\eta}:(\mathcal{X}\times \mathcal{Y})^{*}\to (\mathcal{X}\to \mathcal{Y})$  that sequentially processes a training stream as input and outputs a trained model:  $f_{\theta_t} = G_\eta ((x_1,y_1),\dots ,(x_t,y_t)) = H_\eta ((x_t,y_t),f_{\theta_{t - 1}})$ . This sequential refinement of the model parameter  $\theta$  is commonly referred to as the inner loop, which is the training phase of standard CL. MCL adds another layer of optimization for tuning the meta-parameter  $\eta$  of the learner, which is often called the outer loop. The contents of  $\eta$  can vary widely depending on the method, e.g., the initial parameters of a model, a meta-learned encoder, etc. Unlike traditional CL settings where a model is trained on a single training data stream and evaluated on a test set, an MCL setting consists of multiple CL episodes split into the meta-training set and the meta-test set, as shown in Fig. 1a. The episodes of the two meta-splits are assumed to be drawn from the same distribution. Generally, they are created by first randomly splitting available tasks into two groups, where each episode is built as a random combination of the tasks from the group corresponding to its meta-split. Since there are no overlapping tasks between the meta-splits, one cannot achieve a high meta-test score by simply memorizing all tasks in meta-training. For each episode  $\mathcal{D}$ , a model is produced by the learner, i.e.,  $f_{\theta_T} = G_\eta (\mathcal{D}^{\mathrm{train}})$  and evaluated on the test set  $\mathcal{D}^{\mathrm{test}}$  to produce the meta-loss  $\mathcal{L}(f_{\theta_T},\mathcal{D}^{\mathrm{test}})$ . Then  $\eta$  is updated by meta-gradient descent, i.e., SGD with  $\nabla_{\eta}\mathcal{L}(f_{\theta_T},\mathcal{D}^{\mathrm{test}})$ , to reduce this meta-loss during the meta-training phase. During the meta-test phase, the learner  $H_{\eta}$  is evaluated on multiple CL episodes in the meta-test set. While this description represents the most basic form of MCL that our work is based on, meta-training can deviate from this scheme as long as the integrity of the meta-test is maintained, e.g., [16, 4].
Prior Works in MCL. Online aware Meta-Learning (OML; 16) splits a model into an encoder and a small prediction network on top of it. In the inner loop, only the prediction network is updated by SGD, while the encoder remains frozen. In the outer loop, the encoder and the initial parameters of the prediction network are optimized. A Neuromodulated Meta-Learning algorithm (ANML; 4) has an additional meta-learned component named neuromodulatory network with the same architecture as the encoder. Its output is passed through the sigmoid and multiplied to the encoder's output, gating some features. Meanwhile, there are several MCL approaches specialized to classification. Initially introduced as a few-shot learning method, Prototypical Network (PN; 38) has a meta-trained encoder and computes the average embedding, i.e., prototype, for each class during training. By computing the average embeddings online, PN can be directly used in MCL settings [3]. Generative MCL (GeMCL; 3) extends PN by (i) additionally modeling a diagonal covariance matrix of the embeddings for each class and (ii) adopting a Bayesian formulation.
2.2 Transformers
At the core of Transformers is the self-attention mechanism, which interchanges information among different positions within a sequence. In self-attention, each element in the input sequence of length  $T$  is linearly projected to produce  $d$ -dimensional queries, keys, and values, each represented by  $Q, K, V \in \mathbb{R}^{T \times d}$ . The attention matrix is computed as  $\text{softmax}(QK^{\mathrm{T}} / \sqrt{d})$  where softmax is applied row-wise. It is then multiplied to the value matrix to produce the output of the layer  $O = \text{softmax}(QK^{\mathrm{T}} / \sqrt{d})V$ . If self-attention is computed in this manner, it is more precisely called bi-directional self-attention since every token can attend to all the other tokens in both directions.
While it is used for encoding a sequence in parallel, uni-directional, or causal self-attention is generally used for decoding a sequence one token at a time. In causal attention, each token's output depends only on the preceding tokens, allowing new input tokens to be added without recomputing the existing embeddings. For efficient parallel training, this is typically implemented using a causal attention mask that zeros out the upper triangular elements in the attention matrix. Transformers that only have causal attention layers are called decoder-only Transformers [23] which have become the backbone of many large language models [28, 29, 6]. In addition to decoding, decoder-only Transformers are also suitable for encoding a streaming sequence where the tokens become available one at a time. This property makes them a perfect fit for our use case in MCL settings.
Efficient Transformers. One drawback of Transformers is their quadratic computational cost that arises from computing the  $T \times T$  attention matrix. The computational complexity of  $\mathcal{O}(T^2)$  can be a severe bottleneck in handling long sequences. Therefore, there has been extensive research on more efficient variants of Transformers. In the following, we describe the kernel-based approaches that we test in our experiments. These models have  $\mathcal{O}(T)$  complexity and can be applied to decoding. Refer to [42] for a more comprehensive review of efficient Transformers.
(a) The structure of an MCL dataset (examples from the CASIA benchmark)
(b) SGD-based MCL algorithms
(c) CL as sequence modeling (ours)
Figure 1: Schematic illustrations of the key concepts. (a) In MCL, multiple CL episodes are split into meta-training and meta-test sets. For each CL episode, a continual learner produces a model from the training stream (blue), which is evaluated on the test set (green). The learner is meta-trained on multiple CL episodes in the meta-training set and evaluated on the meta-test set. (b) In many MCL approaches, the learner mainly depends on SGD to update the model in the inner loop. (c) In our framework, a recurrent sequence model plays both the roles of the learner and the model.
Kernel-Based Efficient Transformers (KETs). It approximates the attention matrix as the product of  $Q' = \phi(Q)$  and  $K' = \phi(K)$  where  $\phi: \mathbb{R}^d \to \mathbb{R}^r$  is a non-linear kernel function that transforms each row of  $Q$  and  $K$  into an  $r$ -dimensional feature. The kernel function is meticulously designed such that  $\mathrm{softmax}(QK^{\mathrm{T}}) \approx D^{-1}(Q'K'^{\mathrm{T}})$  where  $D = \mathrm{diag}(Q'(K'^{\mathrm{T}}\mathbf{1}_T))$  is the normalizing term that ensures each row in the approximated attention matrix to sum to one. Through this approximation, we can change the computation order to compute  $K'^{\mathrm{T}}V$  first:  $\mathrm{softmax}(QK^{\mathrm{T}})V \approx D^{-1}Q'(K'^{\mathrm{T}}V)$ . The overall complexity becomes linear to the sequence length, i.e.,  $\mathcal{O}(T)$ . Linear Transformer [17] and Performer [8] are examples of KETs and differ only in their choice of kernel function. The kernel function of Linear Transformer is defined as  $\phi(\boldsymbol{x}) = \mathrm{elu}(\boldsymbol{x}) + \mathbf{1}_d$  where elu is the exponential linear unit (ELU; 9) applied element-wise. Performer [8] uses  $\phi(\boldsymbol{x}) = \exp(W\boldsymbol{x} - \| \boldsymbol{x} \|^2 / 2 \cdot \mathbf{1}_r)$  where the rows of  $W \in \mathbb{R}^{r \times d}$  are orthogonal random vectors.
Decoding with KETs. In theory, the causal attention of KET can be computed by (i) computing the approximate attention matrix  $D^{-1}Q^{\prime}K^{\prime \mathrm{T}}$ , (ii) applying the causal mask, and (iii) multiplying  $V$ . However, this naive process requires  $\mathcal{O}(T^2)$  computation, and it necessitates all tokens at once as input. Fortunately, an alternative method exists to sequentially compute the same output with  $\mathcal{O}(T)$  complexity [17, 8]. Here we represent a vector corresponding to a row of the matrix as the lowercase letter, e.g.,  $V = [v_{1};\dots ;v_{T}]^{\mathrm{T}}$ . At each time step  $t$ , the new  $t$ -th input embedding is linearly projected to produce the query  $q_{t}$ , key  $k_{t}$ , and value  $v_{t}$ . The recurrent internal state at time  $t$  holds the matrix  $K_{t}^{\prime \mathrm{T}}V_{t} = \sum_{t^{\prime} = 1}^{t}\phi (k_{t^{\prime}})v_{t^{\prime}}^{\mathrm{T}}$  and the sum of the key features  $d_{t} = \sum_{t^{\prime} = 1}^{t}\phi (k_{t^{\prime}})$ . Note that both can be computed recurrently:  $K_{t}^{\prime \mathrm{T}}V_{t} = K_{t - 1}^{\prime \mathrm{T}}V_{t - 1} + \phi (k_{i})v_{i}^{\mathrm{T}}$  and  $d_{t} = d_{t - 1} + \phi (k_{t})$ . The attention output at time  $t$  is computed as  $o_{t} = (K_{t}^{\prime \mathrm{T}}V_{t})^{\mathrm{T}}\phi (q_{t}) / d_{t}^{\mathrm{T}}\phi (q_{t})$ .
3 Continual Learning as Sequence Modeling
Many existing MCL (and also CL) approaches rely heavily on SGD to update the model [16, 4] as summarized in Fig. 1b and Alg. 1. When each data point  $(x_{t},y_{t})$  is made available, the model prediction  $\hat{y}_t = f_{\theta_{t - 1}}(x_{t - 1})$  is compared with  $y_{t}$  to produce the loss. The gradient of the loss w.r.t. the parameters  $\theta_{t - 1}$  is fed into an SGD-based update rule, which sometimes accompanies additional treatments to prevent forgetting, to produce updated parameters  $\theta_t$ .
In contrast, we propose to formulate CL as a sequence modeling problem and let a sequence model play both roles of the continual learner and the model, as presented in Fig. 1c. In our framework, the inner loop and outer loop are equivalent to the forward pass and SGD update of the sequence model, respectively. To comply with the assumptions of CL, the sequence model should be representable as a recurrent learner  $H_{\eta}$  that iteratively refines an internal state  $h_t$  as new data arrive, i.e.,  $h_t = H_{\eta}(h_{t-1}, (x_t, y_t))$ . We will explain in §3.1 that Transformers can be seen as a recurrent model. Since the Markov property holds between the internal states,  $h_t$  should summarize all the data seen up to a time step  $t$ . The combination of  $h_t$  and the sequence model's parameter  $\eta$  thus defines a virtual model  $f_{\theta_t} : (\mathcal{X} \to \mathcal{Y})$  with  $\theta_t = (\eta, h_t)$ . After forwarding a training stream  $\mathcal{D}^{\mathrm{train}}$  of length  $T$ , we obtain a virtual model  $f_{\theta_T}$  trained in-context. Then, each test input  $x_n$  is considered as the next token  $x_{T+1}$ , and  $\hat{y}_n = f_{\theta_t}(x_n)$  is produced from the forward pass of the sequence model conditioned on  $h_T$ . The meta-loss function is defined to maximize the conditional log-likelihood:
$$
\mathcal {L} \left(f _ {\theta_ {T}}, \mathcal {D} ^ {\text {t e s t}}\right) = \sum_ {\left(x _ {n}, y _ {n}\right) \in \mathcal {D} ^ {\text {t e s t}}} - \log p \left(y _ {n} \mid x _ {n}; \theta_ {T}\right) = \sum_ {\left(x _ {n}, y _ {n}\right) \in \mathcal {D} ^ {\text {t e s t}}} - \log p \left(y _ {n} \mid x _ {n}, h _ {T}; \eta\right). \tag {1}
$$
This meta-loss in the MCL perspective is technically equivalent to the popular next-word prediction loss in language modeling [28, 29]. That is, we update  $\eta$  to minimize the meta-loss via SGD, as commonly done for language model training. Although we mainly assume all test examples come after the training stream, it is also possible to feed test examples in the middle of a training stream if the corresponding task is already learned. To avoid unnecessary complexity, however, we mainly assume all test examples to come after the training stream.
An important characteristic of our approach is that the target  $y$  is also taken as an input of the sequence model. Since conventional MCL algorithms maintain an explicit model  $f_{\theta}:\mathcal{X}\rightarrow \mathcal{Y}$ , the only way for them to inject the knowledge from  $y$  into the model is the SGD update with the inner loss  $\nabla_{\theta}\mathcal{L}(f_{\theta_t}(x),y)$  (Alg. 1:L6). On the other hand, our sequence modeling approach replaces the SGD update with the forward pass, which incorporates  $y$ 's information by taking it as input. The forward pass can potentially learn a far more flexible update rule compared to SGD. As presented in Fig. 1c, we denote the state after seeing  $x_{t}$  as  $h_{t - 1}^{+}$  to differentiate it from  $h_t$ , the state after seeing both  $x_{t}$  and  $y_{t}$ .
Using sequence models can also be advantageous in terms of scalability. A major drawback of the SGD-based MCL is computing the meta-gradient  $\nabla_{\eta}\mathcal{L}(f_{\theta_T},\mathcal{D}^{\mathrm{test}})$  in the outer loop. Its computation involves calculating higher-order derivatives and large memory for the entire computation graph of the inner loop, which holds every revision of the parameters  $\{\theta_t\}_{t = 1}^T$ , the gradients  $\{\nabla_{\theta_{t - 1}}L_t\}_{t = 1}^T$  and their intermediate calculations. Thus, it is hard to scale the model size or the length of the inner loop. On the other hand, our framework has no such meta-gradient (i.e., our meta-loss is a normal sequence model loss), which allows us to use a much larger and more powerful architecture within the same computational budget.
It is worth mentioning that there are other meta-learning approaches that frame various learning scenarios as the forward pass of a neural network. For instance, [35, 32] employ RNN-based architectures for few-shot learning, while [24] utilize a combination of temporal convolution and attention layers for both supervised and reinforcement learning.
3.1 Transformers as Meta-Continual Learning Methods
As an instantiation of our framework, we demonstrate that Transformers can be utilized as a general MCL mechanism. We only consider decoder-only Transformers, which are suitable for handling the streaming data in CL settings.
Initially, it may not be apparent how the decoder-only Transformers can be represented as a recurrent model in Fig. 1c since Transformers are frequently described as a parallel computation graph over the whole sequence. As highlighted in [17], however, Transformers with causal attention can be regarded as recurrent models. A decoder-only Transformer layer consists of two sublayers: a causal attention layer and a feed-forward layer. Since the feed-forward layers work on each token independently, information transfer across time steps occurs exclusively through the causal attention sublayer, and only in the temporal direction  $(\rightarrow)$ . Therefore, at any time  $t$ , all the past information is represented as keys  $\{\pmb{k}_{t'}^l\}_{t' = 1}^t$  and values  $\{\pmb{v}_{t'}^l\}_{t' = 1}^t$  in each attention head  $l$ , which are accessed by the query  $\pmb{q}_t^l$ . We can think of the keys and values in each attention layer as the internal state of a recurrent sequence model:  $h_t = \bigcup_l h_t^l$  where  $h_t^l = \{(k_{t',}^l,v_{t'}^l)\}_{t'=1}^t$ . At every time step, this internal state is updated to
include the keys and values from the current time step, as highlighted in Alg. 2. In Alg. 2, also notice that  $y_{t}$  is regarded as the input of Transformers (L8-9), and the virtual model  $f_{(\eta ,h_T)}$  is evaluated on the test set to produce the meta-loss (L11).
3.2 Kernel-Based Efficient Transformers for Efficient Meta-Continual Learning