SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Meanwhile, Transformers have been criticized for their consistently growing computational cost, which is more commonly referred to as the quadratic complexity in the Transformer literature. This limitation of Transformers triggered extensive research on efficient Transformers [42] which have sub-quadratic complexity. By bridging CL and sequence modeling, we allow this family of advanced Transformers to be immediately applied as CL methods. The only requirement for the model is to support decoding, which is required to sequentially take new examples in CL settings.
Among the large taxonomy of efficient Transformers, our work focuses on kernel-based efficient Transformers (KETs), which can be adapted to decoding while having  $\mathcal{O}(T)$  complexity. As explained in §2.2 the decoder version of KET works very similarly to an RNN with an internal state of a constant size. As summarized in L7 and L9 of Alg. 3, the update rule corresponding to each attention head  $l$  can be concisely expressed as adding the outer product of the key feature  $\phi(\pmb{k}_t^l)$  and the value vector  $\pmb{v}_t^l$  to the hidden state of a fixed size. Note that 1 is appended to  $\pmb{v}_t^l$  to compute the sum of the key features  $\pmb{d}_t^l = \sum_{t'=1}^{t} \phi(\pmb{k}_{t'}^l)$  simultaneously. We test Linear Transformer [17] and Performer [8], and the only difference between the two is the kernel function  $\phi$ , as explained in §2.2.
Connection to Dynamic/Static-Architecture CL Approaches. Since the size of the internal states linearly grows as more data come in, standard Transformers can be grouped with the dynamic CL architectures. As the size and the computational cost of dynamic-architecture CL approaches typically grow linearly with the number of tasks  $K$  [34, 2, 45, 20], the complexity of handling a CL episode of length  $T$  is  $\mathcal{O}(KT)$ . In most CL scenarios where  $K \propto T$ , the complexity becomes  $\mathcal{O}(T^2)$ , which is the same as Transformers. On the other hand, KETs' internal states remain at a constant size, which makes them the counterparts of static-architecture CL approaches.
3.3 An Example: Continual Learning of Image Classification as Sequence Modeling
We describe a specific example of applying our approach to image classification, which has been widely used for evaluating MCL methods. In this setting,  $x$  is an image,
while  $y$  is the corresponding class label. To feed into Transformers, both  $x$  and  $y$  should be encoded into a fixed-sized vector. While we can simply use a convolutional network (which can be jointly trained end-to-end) for  $x$ , properly encoding  $y$  requires some considerations. Since a class label is essentially an abstract symbol, e.g., an arbitrary index of the class, it does not have any meaning on its own. Simply hard-assigning a trainable embedding to each class would fail, as novel classes would appear in the meta-test phase. Randomly sampling a fixed embedding every time a class appears also turns out to be inapplicable as we experiment. The random embedding is the same as an untrained token embedding, which is expected to be incompatible with the trained weights of Transformers.
Algorithm 1 Inner loop of conventional SGD-based MCL
Require: learner Hη, Dtrain, Dtest
1: T ← |Dtrain|
2: fθ0 ← initial model
3: for t ← 1 to T do
4: x_t, y_t ← Dtrain[t]
5: L_t ← L(fθt-1(x_t), y_t) ▷ Loss
6: fθt ← Hη(fθt-1, ∇θt-1Lt)
7: end for
8: return L(fθT, Dtest) ▷ Meta-loss
Algorithm 2 Inner loop of Transformer
Require: Transformer Hη, Dtrain, Dtest
1: T ← |Dtrain|
2: I ← indices of attention heads in Hη
3: h0 ← {} ▷ internal state
4: for t ← 1 to T do
5: xt,yt ← Dtrain[t]
6: {(kl,t, vt)}l∈I ← Hη(ht-1, xt)
7: ht-1← ht-1 ∪ {(kl,t, vt)}l∈I
8: {(kl,t, vt)}l∈I ← Hη(ht-1, yt)
9: ht← ht+1 ∪ {(kl,t, vt)}l∈I
10: end for
11: return L(f(η,hT), Dtest) ▷ Meta-loss
Algorithm 3 Inner loop of KET
Require: KET Hη, Dtain, Dtest
1: T ← |Dtrain|
2: I ← indices of attention heads in Hη
▷ Superscript l implies ∀l ∈ I
3: h0l← Or×(d+1)	▷ internal state
4: for t ← 1 to T do
5: xt,yt ← Dtain[t]
6: {(kti,vi)}l∈I ← Hη(ht-1, xt)
7: ht-1← ht-l-1 + φ(kl)[vtT; 1]
8: {(kti,vi)}l∈I ← Hη(ht+1, yt)
9: ht← ht+,l+1 + φ(kl)[vtT; 1]
10: end for
11: return L(f(η,hT), Dttest) ▷ Meta-loss
The general solution used in language modeling is to express everything as a combination of known vocabulary [37]. Following the same principle, we tokenize a class label using a fixed vocabulary  $V$  and train an embedding for each token in the vocabulary. Since class labels do not impose any constraint on how they should be tokenized, we randomly sample a unique class code  $(v_{1},\dots,v_{C})$ , a random combination of the tokens  $v_{c}\in V$ , when a new class appears in an episode. Note that if the code length  $C$  is greater than one, each token goes into a Transformer individually, i.e., an  $(x,y)$  pair is converted to the sequence  $(x_{\mathrm{enc}},v_1,\dots,v_C)$  before going into the Transformer. At test time, the Transformer sequentially outputs the code to predict the corresponding class. To reduce complications, we keep  $C = 1$  in the main experiments, and the vocabulary size is set to the maximum number of classes that can appear in an episode. Experiments with  $C > 1$  can be found in Appendix B.
We demonstrate the potential of our approach through experiments with seven benchmarks. Unless otherwise noted, each experiment is repeated five times, and the mean and standard deviation of the five runs are reported. Due to space constraints, this section contains only the essential information to understand the main result, while details, additional experiments and analyses are deferred to Appendix A, B, and C.
To show the generality of our approach, we conduct experiments with diverse types of tasks, including both classification and regression. To construct the meta-training and meta-test set for a dataset, we first divide it into a set of tasks. The tasks are then split into two disjoint sets, one for meta-training and the other for meta-testing. The CL episodes for both meta-splits are constructed in the same manner. To build a CL episode  $\mathcal{D}$ , we randomly sample  $K$  unique tasks. By default, we set  $K = 20$ , while additionally testing the  $K = 100$  setting to compare performances with longer episodes. For each task  $k$ , the training stream  $\mathcal{D}_k^{\mathrm{train}}$  and the test set  $\mathcal{D}_k^{\mathrm{test}}$  contain five examples each (i.e., five shots). For each experiment, we meta-train for 50K steps with a batch size of 16 (i.e., 16 episodes in parallel) and meta-test with 1,024 episodes. Task identities are not provided in any case.
4.1.1 Classification
For simplicity, we define each class as a task. All input images are resized to  $32 \times 32$  resolution. We report classification errors in percentage as the evaluation metric.
CIFAR-100 [18]. CIFAR-100 contains 60K images of 100 classes, with 600 images for each class. We randomly choose 60 classes for meta-training and the other 40 for the meta-test.
Omniglot [19]. Omniglot is a handwriting dataset widely used as a benchmark for meta-learning or MCL. It has a total of 1,623 character classes gathered from 50 alphabets. The meta-train set consists of 963 classes (30 alphabets), and the meta-test set has 660 classes (20 alphabets). There are 20 images for each class, thus 19K images for meta-training and 13K images for meta-test.
CASIA Chinese Handwriting Database (CASIA; 22). This is another handwriting dataset that consists of 7,356 character classes in total. The classes comprise 7,185 Chinese characters, 52 English alphabets, 10 digits, and other frequently used symbols. The total number of images is  $3.9\mathrm{M}$ , which is 120 times larger than Omniglot. We randomly sample 1K classes for the meta-test and use the rest for meta-training. To the best of our knowledge, this dataset has not been used in meta-learning or MCL literature. However, its substantial size allows us to evaluate different aspects of MCL algorithms, which could not be examined with smaller datasets where meta-overfitting is a major factor in the evaluation. Our results suggest that the relative performance of MCL algorithms can differ in a large data regime.
MS-Celeb-1M [10]. In addition to the CASIA dataset, we also propose to repurpose MS-Celeb-1M as a large-scale MCL benchmark. It is a facial image dataset with roughly 10M images of 100K celebrities. Similar to CASIA, we randomly select 1K classes, the celebrity IDs, for the meta-test and use the others for meta-training.
Table 1: Classification errors (%) in 20-task 5-shot MCL. Both meta-training and meta-test errors are reported to highlight the relationship between the degree of meta-overfitting and the scale of the dataset ( images, classes).
Type	Method	CIFAR-100	Omniglot	CASIA	MS-Celeb-1M
60K ▲/ 0.1K ▲	32K ▲/ 1.6K ▲	3.9M ▲/ 7.2K ▲	10M ▲/ 100K ▲
Meta-train	Meta-test	Meta-train	Meta-test	Meta-train	Meta-test	Meta-train	Meta-test
Offline	Offline	N/A	73.3±7.2	N/A	35.0±6.8	N/A	41.7±6.2	N/A	63.3±7.7
CI	PN	0.0±0.0	76.6±0.3	0.0±0.0	3.8±0.1	0.2±0.0	0.4±0.0	32.5±0.1	33.6±0.1
GeMCL	0.0±0.0	76.6±0.4	0.0±0.0	3.8±0.1	0.2±0.0	0.4±0.0	32.1±0.1	33.3±0.2
SGD	OML	0.6±0.1	89.9±0.4	0.1±0.0	24.8±2.2	2.8±0.1	3.2±0.1	41.8±0.3	42.5±0.2
ANML	0.4±0.1	88.1±1.4	0.0±0.0	31.0±6.2	3.7±0.5	4.3±0.5	43.8±0.3	44.8±0.4
Ours	Transformer	0.0±0.0	82.8±0.8	0.0±0.0	13.7±0.6	0.3±0.0	0.4±0.0	29.1±0.2	30.0±0.2
Linear TF	0.1±0.1	83.4±0.5	0.0±0.0	36.0±1.4	0.4±0.0	0.7±0.0	31.1±0.3	32.4±0.3
Performer	0.0±0.0	82.9±0.3	0.1±0.1	37.1±4.6	0.5±0.0	0.7±0.0	32.5±0.5	33.7±0.2
Table 2: Classification errors (%) and regression errors of 100-task 5-shot MCL.
Type	Method	CASIA	MS-Celeb-1M	Sine	Rotation	Completion
Offline	Offline	70.7±3.7	87.1±3.3	0.0180±0.0038	0.699±0.049	0.1713±0.0047
CI	PN	0.8±0.0	44.5±0.1	N/A	N/A	N/A
GeMCL	0.8±0.0	44.4±0.1	N/A	N/A	N/A
SGD	OML	6.8±0.9	54.5±0.2	0.0498±0.0004	0.524±0.087	0.1087±0.0001
Ours	Transformer	1.0±0.0	40.5±0.1	0.0031±0.0002	0.031±0.001	0.0989±0.0001
Linear TF	2.3±0.1	45.3±0.1	0.0139±0.0003	0.047±0.002	0.1084±0.0001
4.1.2 Regression
Sine Wave Reconstruction (Sine). Inspired by the sine wave regression problem in [16], we design a more challenging synthetic regression problem for MCL. A sine wave  $\omega(\tau) = A\sin(2\pi \nu \tau + \psi)$  is characterized by amplitude  $A$ , frequency  $\nu$ , and phase  $\psi$ . We define the target  $y$  as the values of the sine wave at 50 fixed points:  $y = [\omega(\tau_1), \dots, \omega(\tau_{50})]$ . All  $y$ 's in each task share the same frequency and phase, while they can vary in amplitudes. We corrupt  $y$  into  $x$  by shifting phase and adding Gaussian noise, where the phase shift amount is sampled for each task. We report the mean squared error between  $y$  and the model prediction  $\hat{y}$  as the evaluation metric.
Image Rotation Prediction (Rotation). A model is given an image rotated by an angle  $\psi \in [0,2\pi)$  and tasked to estimate an angle  $\hat{\psi}$ . The evaluation metric is  $1 - \cos (\hat{\psi} -\psi)$ ; thus, a perfect model would score 0 while random guessing would score 1.0 on average. We use the CASIA images and define each class as a task using the same meta-split.
Image Completion (Completion). An image completion problem involves predicting the missing parts of an image based on the available parts. Derived from our CASIA classification benchmark, we change the input  $x$  to be the top half of the image and the target  $y$  to be the bottom half. The evaluation metric is the mean squared error of the predicted pixel values.
For all methods, we use a five-layer CNN to encode image inputs and a three-layer MLP to encode vector inputs (e.g., the sine regression). All components are trained end-to-end from scratch. We group compared methods into four types: offline, class-incremental MCL, SGD-based MCL, and ours.
Offline. We compare the scores of offline learning since it is generally perceived as the performance upper bound of non-meta-CL. Note that MCL approaches can outperform offline learning by lever
aging additional meta-training data. Since the model overfits to the training data we report the best test score achieved during training. We repeat each experiment ten times and report the average and standard error of mean.
Class-Incremental MCL (CI). We test PN [38] and GeMCL [3] as MCL approaches that are specialized for class-incremental settings. Although they cannot be applied to other settings, such as regression, they show strong performance in the classification benchmarks.
SGD-Based MCL (SGD). We select OML [16] as the main baseline since it can perform both classification and regression tasks. It has a two-layer MLP as the prediction network, on top of a meta-learned encoder. For 20-task 5-shot classification benchmarks, we also test ANML [4], which is an extension of OML with an additional modulatory network that regulates the features of the encoder. However, due to its extensive memory consumption, it could not be tested on longer episodes.
Ours. We test the vanilla Transformer [43], Linear Transformer [17], and Performer [8]. All the models share a similar architecture: 4 layers, 8 heads, and 512 hidden dimensions. Although it is tiny compared to modern language models, it is still larger than the FC layers of the baselines. Surprisingly, however, they require comparable or less GPU memory and time for meta-training in our experiments due to the lack of meta-gradient computation, as noted in §3. For more analysis on the computational cost, refer to Appendix C.
In Table 1, we report the meta-training and meta-test errors of the 20-task 5-shot classification benchmarks to show the necessity of large-scale MCL benchmarks. In CIFAR-100 and Omniglot, all methods show a severe degree of meta-overfitting, the gap between the meta-training and meta-test scores. Meta-overfitting is more serious in CIFAR-100 where the task (class) diversity is lower. In such situations, the Transformers still perform competitively, but the baselines may achieve a better meta-test score due to less meta-overfitting. On the other hand, if more data is available, as in the CASIA and MS-Celeb-1M benchmarks, the Transformers outperform OML and ANML by a large margin and perform on par or better than PN or GeMCL, which are specialized for classification. Table 2 shows the results of longer 100-task 5-shot experiments in both classification and regression domains. In this more challenging scenario, the Transformers still perform comparably to class-incremental methods and significantly better than SGD-based approaches. The scores of offline learning mostly fall behind other MCL methods, which implies that non-meta-CL methods are not suitable for our experimental settings where the amount of data is far from abundant.
A strength of our approach, which comes with the use of Transformers, is parallel meta-training. While the baselines require a sequential SGD update on the training stream, Transformers can process the whole training stream in parallel, which can dramatically improve the meta-training efficiency once combined with massively parallel processors like GPUs. As a result, the meta-training time of Transformers is several times shorter than the baselines. The details can be found in Appendix C.
The tradeoff between the standard Transformer and the KETs is also worth discussing. In all experiments, the KETs sacrifice some level of accuracy for greater efficiency. The accuracy degradation is more severe in longer episodes, which is expected given that more information has to be fit into the internal state of the same size. This tradeoff suggests that the choice between standard and efficient Transformers ultimately depends on the required accuracy and the available computational resources.
In Appendix B, we present more experiments and analyses, including different episode configurations and model sizes. Although preliminary, the experiments with Transformers of varying sizes are particularly interesting, as the performance scales with the number of parameters. This is reminiscent of the scaling law of large language models [14], suggesting that scaling might be a potential solution to more challenging MCL problems.
By formulating MCL as sequence modeling, we pave the way for incorporating advanced sequence modeling technologies into the MCL field. Given the observation that our approach demonstrates greater efficacy with increased data, it seems consistent with The Bitter Lesson, an argument popularized by Sutton [41]. It states that, in the long run, general-purpose learning algorithms capable of
leveraging more computation and data are likely to be more effective than human-designed algorithms. We believe our work is a step forward in this direction, generalizing the SGD-based update rules in CL with the learnable forward pass of a sequence model.
However, it is important to acknowledge that the sequence models also bring their own limitations and challenges. Efficiently handling long sequences is one such challenge. According to our experiments, even the efficient Transformers, which are designed to solve the exact challenge, have to trade off a nonnegligible amount of performance for efficiency, which implies plenty of room to improve. Nonetheless, considering the current pace of development in the sequence modeling field, we are optimistic about the potential of our approach; if a new solution is developed for sequence modeling, it can be readily integrated into our formulation.
We thank Jaekyeom Kim, Dongjoo Kim, Dongyeon Woo, and Sangwoo Moon for their thoughtful feedback. This work was partly supported by Samsung Advanced Institute of Technology, the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 2023R1A2C2005573), and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2022-0-00156, Fundamental research on continual meta-learning for quality enhancement of casual videos and their 3D metaverse transformation; No. 2021-0-01343, Artificial Intelligence Graduate School Program (Seoul National University)). Gunhee Kim is the corresponding author.
[1] Roee Aharoni, Melvin Johnson, and Orhan First. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3874-3884, Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1388. URL https://aclanthology.org/N19-1388.
[2] Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 7120-7129. IEEE Computer Society, 2017. doi: 10.1109/CVPR.2017.753. URL https://doi.org/10.1109/CVPR.2017.753.
[3] Mohammadamin Banayeeanzade, Rasoul Mirzaiezadeh, Hosein Hasani, and Mahdieh Soleymani. Generative vs. discriminative: Rethinking the meta-continual learning. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 21592-21604, 2021. URL https://proceedings.neurips.cc/paper/2021/bit/ b4e267d84075f66ebd967d95331fcc03-Abstract.html.
[4] Shawn Beaulieu, Lapo Frati, Thomas Miconi, Joel Lehman, Kenneth O. Stanley, Jeff Clune, and Nick Cheney. Learning to continually learn. In Giuseppe De Giacomo, Alejandro Catalá, Bistra Dilkina, Michela Milano, Senén Barro, Alberto Bugarín, and Jérôme Lang, editors, ECAI 2020 - 24th European Conference on Artificial Intelligence, 29 August-8 September 2020, Santiago de Compostela, Spain, August 29 - September 8, 2020 - Including 10th Conference on Prestigious Applications of Artificial Intelligence (PAIS 2020), volume 325 of Frontiers in Artificial Intelligence and Applications, pages 992-1001. IOS Press, 2020. doi: 10.3233/FAIA200193. URL https://doi.org/10.3233/FAIA200193.
[5] Magdalena Biesialska, Katarzyna Biesialska, and Marta R. Costa-jussa. Continual lifelong learning in natural language processing: A survey. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6523-6541, Barcelona, Spain (Online), 2020. International Committee on Computational Linguistics. doi: 10.18653/v1/2020.coling-main.574. URL https://aclanthology.org/2020.coling-main.574.
[6] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal,
[42] Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. ACM Comput. Surv., 55(6), 2022. ISSN 0360-0300. doi: 10.1145/3530811. URL https://doi.org/10.1145/3530811.
[43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008, 2017. URL https://proceedings.neurips.cc/paper/2017/bit/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html.
[44] Yuxuan Wang, R. J. Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J. Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, Quoc V. Le, Yannis Agiomyrgian-nakis, Rob Clark, and Rif A. Saurous. Tacotron: Towards end-to-end speech synthesis. In Francisco Lacerda, editor, Interspeech 2017, 18th Annual Conference of the International Speech Communication Association, Stockholm, Sweden, August 20-24, 2017, pages 4006-4010. ISCA, 2017. URL http://www.isca-speech.org/archive/Interspeech_2017/abstracts/1452.html.
[45] Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https://openreview.net/forum?id=Sk7KsfW0-.
A.1 Common Training Setups
Unless explicitly stated, all methods share the same encoder for input  $x$ . For image inputs, we use a five-layer CNN architecture. The kernel size is set to  $3 \times 3$  in all convolutional layers, and batch normalization [15] and ReLU [25] activation are applied after each convolution. The stride is set to 2, except for the first layer which has stride 1. The channel sizes are set to 32-64-128-256-256 for the five layers. In the case of the vector input in the Sine benchmark, we use a simple three-layer MLP architecture. It has three linear layers followed by batch normalization and ReLU. For all benchmarks, we do not use any data augmentation technique.
Since there are a lot of different experimental settings in this work, we could not perform an extensive hyperparameter search in every setting. Therefore, we mainly tune the hyperparameters to the CASIA classification benchmark and apply the same settings to the other experiments. We find that Transformers are very robust to the choice of hyperparameters, achieving impressive scores in all benchmarks without any benchmark-specific hyperparameter tuning. In some baselines, however, we had to manually tune the hyperparameters specific to a benchmark, in order to achieve a reasonable score. Please refer to the configuration files and commands in the included code for the exact configurations.
A.2 Benchmarks
CASIA. The CASIA dataset originally contains grayscale images of various sizes since the images are cropped tightly to the handwriting. We preprocess these images into  $32 \times 32$  by (i) resizing the image such that the longer edge becomes 32 pixels long and (ii) adding paddings symmetrically to both sides of the shorter edge to fill the empty space. This maintains the original aspect ratio of the images while standardizing their size.