SlowGuess/NeurIPS_2023_OCR huggingface dataset:
MS-Celeb-1M. MS-Celeb-1M also contains images of various sizes. Since the images are roughly aligned to the face, we simply resize them to  $32 \times 32$  resolution without keeping the aspect ratio. We also exclude the classes (celebrity IDs) with fewer than 20 examples.
Rotation. In the Rotation benchmark, a model should produce an angle  $\psi$  given an image. Due to the cyclic nature of the angle, we let the model output  $(\cos \hat{\psi}, \sin \hat{\psi})$  instead of  $\hat{\psi}$ . To necessitate learning from each training stream, we additionally shift all  $\psi$  by the same random amount sampled from range  $[0, 2\pi)$ . Therefore, even if there is a trivial pattern in  $\mathcal{X}$  indicating orientation, e.g., a unique shape marking the top of an image, the model cannot predict the shifted  $\psi$  without the knowledge from the training stream.
Completion. Although the output format is an image, we simply treat the output and target as a 512-dimensional vector (flattened from  $16 \times 32$  image). We simply use fully connected layers for output and do not use specialized architectures, such as transposed convolution.
As SGD-based MCL baselines, we test OML [16] and ANML [4]. While we do our best for a fair comparison, there are several modifications made to the settings and architectures. In their original paper, OML and ANML do not maintain a full computation graph of the inner loop during meta-training to reduce the computation cost. Instead, they segment the training stream every five tasks and compute the meta-gradient only within each segment. Since they do not backpropagate through a long chain of updates, the meta-training cost is reduced at the expense of optimizing with a suboptimal objective. However, since our Transformers can backpropagate through the whole episode, comparing it with such a suboptimal configuration can be considered unfair. Therefore, in our experiments, we maintain a full computation graph of the inner loop for a fair comparison. In an effort to make the baselines more robust to the choice of hyperparameters, we also set the learning rate of the inner loop as a learnable parameter.
The model's classification output mechanism is also modified. In the original design, the weights of the output head are hard-assigned to individual classes. When learning a class, the weights corresponding to the class are randomly reinitialized, to prevent overfitting to the classes in meta-
training. Since we developed the class representation scheme in §3.3 inspired by the tokenization schemes in the language modeling domain, we use the same class representation for the baselines. Using this representation, the model can output a token in the fixed vocabulary, and when a new class is introduced, an unused token is randomly assigned to it. The model can predict a class by producing the corresponding token. Since the tokens are randomly assigned to a class, we do not need to reset the weights.
A.4 Transformers
In the early experiments, we observed that the (meta-)training often fails to converge, especially when the episode length becomes longer. After a closer inspection of the attention pattern of the failure cases, we discovered that the attention tends to be uniformly distributed across all tokens, not transmitting any useful information to the next layer.
One possible reason is that our meta-loss (Eq. 1) is not easy to optimize. Given a test input, a well-trained Transformer should send some attention query from the test input to the keys from the training sequence of the corresponding task. However, since the meta-loss is simply a next-word prediction loss, it provides only high-level guidance without enforcing any specific attention pattern. Note that the final prediction layer and self-attention layers are two distinct mechanisms. When a Transformer is randomly initialized, it seems hard for it to notice that attending to a specific part of the training stream is useful for predicting the target. Therefore, the attention layers become "dead," which is the best thing to do if their inputs are considered random noise.
As a solution, we add an auxiliary loss called attention loss to explicitly guide the attention pattern in the early stage of training. For each query of a test input  $x_{n}$  of task  $k$ , we maximize the log sum of its attention to the keys from all training examples of task  $k$ . In plain words, we guide the attention to be headed to task  $k$ 's training sequence, but we do not care exactly which token is attended. We apply attention loss to half of the attention heads in each layer for the first 10K steps of (meta-)training.
B Additional Experiments and Analyses
B.1 Experiments with ResNet Encoder
In the main text, we present the experiments with a simple 5-layer CNN encoder. This is mainly because some baselines, e.g., ANML [4], require meta-training of the whole network, including the encoder. Using more sophisticated encoders, such as ResNet [11], will increase the computational cost, which is already high due to the meta-gradient computation. Furthermore, it is hard to define the proper usage of the batch normalization [15], which is part of the ResNet architecture, in CL settings.
On the other hand, OML [16] and our Transformers do not update the encoder in the inner loop. Since the encoder is updated only in the outer loop, using ResNet with these methods does not violate the underlying assumptions of batch normalization. Here we provide the results with the ResNet-18 encoder. The ResNet-18 architecture is slightly modified from the original version since it was originally designed for handling much larger images with  $224 \times 224$  resolution. We replace the first  $7 \times 7$  stride-2 convolution with  $3 \times 3$  stride-1 convolution, remove the next  $3 \times 3$  max-pooling, and reduce the number of features in all layers by half. For the efficient Transformers, we test Linear Transformer only since it generally achieves better scores than Performer. The results are presented in Table 3. The first three columns are the classification benchmarks, while the last two are regression benchmarks.
We find that using ResNet does not make a meaningful difference in the large benchmarks using CASIA or MS-Celeb-1M. Since the images used in our experiments are small ( $32 \times 32$  resolution), the simple CNN encoder seems sufficient to deliver the necessary information to the upper components. In the case of Omniglot classification, the scores of all methods even degrade due to more meta-overfitting.
B.2 Experiments with Combinatorial Class Representation
Although we introduced the combinatorial representation for classes in §3.3, we simply keep the number of tokens for each class representation to 1 in our main experiments. In Table 4, we present the results of the 20-task CASIA benchmark with different class representations. The default setting is
Table 3: Experiments with ResNet encoder in 20-task 5-shot settings.
Method	Omniglot	CASIA	MS-Celeb-1M	Completion
OML	24.8±2.2	3.2±0.1	42.5±0.2	0.1092±0.0002
OML (ResNet)	27.7±1.3	3.4±0.1	44.1±0.2	0.1092±0.0002
Transformer	13.7±0.6	0.4±0.0	30.0±0.2	0.0999±0.0002
Transformer (ResNet)	25.5±1.2	0.5±0.0	32.4±0.2	0.1014±0.0002
Linear TF	36.0±1.4	0.7±0.0	32.4±0.3	0.1039±0.0003
Linear TF (ResNet)	63.6±2.1	0.7±0.0	34.9±0.1	0.1051±0.0001
CASIA
Model parameters
MS-Celeb-1M
Model parameters
Sine
Model parameters
(b) Effect of scaling model size in 100-task 5-shot regression benchmarks.
Figure 2: Scaling behavior of Transformers.
(a) Effect of scaling model size in 100-task 5-shot classification benchmarks.
Model parameters
Completion
Model parameters
represented as  $|V| = 20$  and  $C = 1$ , where  $|V|$  is the vocabulary size, and  $C$  is the class representation length. We additionally test two more settings with  $|V| = 8$ ,  $C = 2$  and  $|V| = 8$ ,  $C = 3$ . Note that the total number of unique class representations is  $|V|^C$ , which increases exponentially to  $C$ . We find that the classification errors are mostly similar regardless of the representation, although there is a slight performance degradation as the  $C$  increases. We suspected that it is caused by the increased sequence length. Since the class representations are taken as input by the Transformer, increasing  $C$  induces more computational cost. Nonetheless, we believe the idea of combinatorially representing a class or a new concept is interesting and might have a useful application in the future.
Table 4: Transformers with combinatorial class representation.
Method	Representation	CASIA error (%)
Transformer	|V|=20,C=1	0.4±0.0
Transformer	|V|=8,C=2	0.5±0.1
Transformer	|V|=8,C=3	0.6±0.0
B.3 Scaling Behavior of Transformers
Here we test how the model size of the Transformer is related to performance. This experiment is inspired by the scaling law of large language models [14], where the performance keeps increasing with the computational power. We test four different Transformer architectures whose number of parameters roughly ranges from 2M to 40M. The architecture details are summarized in Table 5. Note that we use the third architecture in other experiments.
In CL literature, which generally relies on SGD updates to incorporate new knowledge, increasing model size does not necessarily lead to better CL ability. On the other hand, as shown in Fig. 2a and Fig. 2b, the performance of Transformers consistently improves as more parameters are added. This is a promising result since it implies that we can simply achieve better scores by providing more computation and data. We speculate that scaling up the model size may be a necessary condition to solve more realistic and challenging scenarios.
Table 5: Transformer architecture configurations.
Method	n_layers	d_model	n_heads	d_heads	d_MLP
Transformer (S)	2	256	4	64	512
Transformer (M)	2	512	8	64	1024
Transformer	4	512	8	64	1024
Transformer (XL)	4	1024	16	64	2048
B.4 Forgetting Analysis
Fig. 3 and Fig. 4 show the forgetting analysis [7] of 100-task CASIA classification and Sine regression benchmarks. For each task  $k$ , we measure its error increase after learning  $k' (> k)$  tasks, relative to the error right after learning it. Each graph in Fig. 3 and Fig. 4 shows the average of five consecutive tasks to save space. The last plot shows the average forgetting of all tasks.
Note that forgetting alone cannot be used as an evaluation metric since it measures the relative change in performance throughout an episode. For example, if a model scores zero accuracies in all tasks throughout the whole episode, the forgetting is measured as zero. Therefore, it should rather be used as an additional tool for analysis.
For the forgetting analysis, we meta-train the Transformer and Linear Transformer to also handle test examples in the middle of the training stream. For each test example of a task  $k$ , we randomly sample the evaluation moment from the range  $[k, K]$ , i.e., an example can be evaluated at the end of any task's training as long as the corresponding task is learned. In both benchmarks, Transformers' forgetting is significantly lower than OML, demonstrating the effectiveness of our approach.
C Computational Cost Analysis
First, we emphasize that it is hard to define a single unified measure to compare the computational cost. In some environments, the total memory consumption may be a hard constraint, while running time can be the top priority in others. Moreover, the actual cost can vary dramatically depending on the software implementation or the underlying hardware. Here we compare various aspects of the computational cost using our PyTorch [27] implementation on NVIDIA A40 GPUs which have 48 GB of VRAM. We report the cost of meta-training with a batch size of 16.
The results of the Linear Transformer and Performer are omitted due to the current limitation of our implementation. Theoretically, these efficient Transformers can be far more efficient than the standard Transformer both in terms of memory and computation time. However, realizing such advantages while maintaining high GPU utilization requires sophisticated CUDA kernel programming and cannot be implemented with simple PyTorch code. Therefore, we explicitly compute the attention matrix, the same as the standard Transformer, to save implementation time. This suboptimal implementation results in a slightly worse memory and computation time compared to the standard Transformer.
Table 6 compares the number of parameters, GPU memory consumption (in GB), meta-training speed (in episodes per second), and the classification error on the CASIA benchmark. Remarkably,
Figure 3: Forgetting analysis of 100-task CASIA benchmark.
even the largest Transformer consumes less memory than OML, the most memory-efficient baseline. Although OML has much fewer parameters than Transformers, they consume far more GPU memory due to the meta-gradient computation. If we scale up OML to have a similar number of parameters with the default Transformer architecture, which is referred to as OML (XL) in Table 6, the memory consumption increases even further, requiring four A40 with data parallelism.
Transformers are also superior in terms of (meta-)training speed. Since they can process all the examples in an episode in parallel, they can utilize GPUs more effectively. On the other hand, SGD-based baselines need to sequentially process each example and also sequentially backpropagate in the reverse direction. Therefore, Transformers have several times faster training speed compared to the baselines.
Figure 4: Forgetting analysis of 100-task Sine benchmark.
Table 6: Computational cost comparison.
Episode	Method	Parameters	GPU memory	Training speed	CASIA error
20-task 5-shot	OML	1.5M	10.7 GB	51.2 ep/s	3.2±0.1
OML (XL)	11.5M	4 × 33.7 GB	3.5 ep/s	1.9±0.2
ANML	2.0M	18.4 GB	6.6 ep/s	4.3±0.5
Transformer (S)	2.4M	4.3 GB	278.5 ep/s	0.6±0.0
Transformer (M)	5.9M	4.7 GB	252.9 ep/s	0.5±0.0
Transformer	10.1M	5.2 GB	201.5 ep/s	0.4±0.0
Transformer (XL)	36.0M	6.2 GB	138.0 ep/s	0.4±0.0
100-task 5-shot	OML	1.5M	43.7 GB	9.40 ep/s	6.8±0.9
Transformer (S)	2.4M	22.0 GB	49.69 ep/s	1.5±0.0
Transformer (M)	5.9M	25.8 GB	38.78 ep/s	1.1±0.0
Transformer	10.1M	31.5 GB	26.80 ep/s	1.0±0.0
Transformer (XL)	36.0M	41.4 GB	16.34 ep/s	0.8±0.0