SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The functions  $f_{\theta}(\cdot)$  and  $h_t(\cdot)$  are trained using a similar approach to LwF [22]. After completing the training for the  $t-1$ th task, the training proceeds for the  $t$ th task, temporarily saving  $f_{\theta(t-1)}$  and  $h_{k,t-1}$  for this task. First,  $m_t$  is trained. Then, the randomly initialized  $h_t$  is pre-tuned while the rest of the model is frozen using a small portion of the dataset. For each sample  $\mathbf{x}_{t,i}, \mathbf{y}_{t,i}$ , the pseudo-label for the  $k$ -th  $(1 \leq k < t)$  task  $\hat{y}_{k,i}$  is obtained by  $\text{argmax} \cdot h_{k,t-1} \cdot f_{\theta(t-1)} \cdot m_t(\mathbf{x}_{t,i})$ . The loss is calculated for all  $t$  heads. For the outputs of the  $k$ -th task heads:  $h_k \cdot f_\theta \cdot m_t(\mathbf{x}_{t,i})$  ( $1 \leq k \leq t$ ), the loss is computed with the pseudo-labels  $\hat{y}_{k,i}$  for  $1 \leq k < t$ , and for the  $t$ -th task head, the loss is computed with  $\mathbf{y}_{t,i}$ . The final loss is the sum of the losses for each head up to  $t$ -th. The overall learning process is shown in Algorithm 1.
In this section, we first confirmed that applying our mapping function and tuning structure did not alter the learning ability of each model on tasks from various Chomsky hierarchies, compared to the previous study [11]. Next, we demonstrate CLeAR's superior performance on high-correlation, in-hierarchy, and inter-hierarchy CL-AR tasks through extensive ablations. Furthermore, we analyze the model's robustness in terms of noise within the mapping space. A detailed explanation of the experiment setting and comparison with existing CL algorithms [19, 22, 5] and their failure is shown in Appendix D.
4.1 Baseline Generalization Accuracy
In Table 1, we presented the generalization accuracy of each model for all tasks in our paper. Corresponding in-distribution accuracy and reported generalization accuracy in the previous paper [11] paper is shown in Appendix B. It was shown that applying mapping and head did not greatly affect the model's performance and did not alter the limits of the Chomsky hierarchy that each model can learn. The reported performance here was used as a benchmark for calculating Single-task accuracy and FWT.
Table 1: Baseline generalization accuracy (\%) of overall tasks. Each task was averaged on three repeats, trained for 50,000 epochs each. The accuracy value was reported with positional mapping (accuracy without positional mapping is in parentheses). For accuracy over  $90\%$ , we considered that the model successfully learned AR as suggested in [11]. The random accuracy is  $50\%$  except for Modular Arithmetics (R), Cyclic Navigation (R), and Bucket Sort (CS). Tasks that require additional positional mapping are tagged with  $\dagger$ .
Level	Task	RNN	Stack-RNN	Tape-RNN	LSTM
Regular	Even Pairs	100.0(100.0)	100.0(100.0)	51.4(53.1)	100.0(100.0)
Modular Arithmetic	99.7(98.8)	99.1(99.2)	63.1(82.4)	100.0(100.0)
Parity Check	100.0(100.0)	100.0(100.0)	99.6(99.3)	100.0(100.0)
Cycle Navigation	100.0(98.1)	99.7(100.0)	80.0(97.3)	92.1(87.9)
Context Free	Compare Occurrence	96.6(97.9)	99.5(98.5)	97.7(97.9)	99.6(99.5)
Stack Manipulation†	57.0	83.9	75.6	82.4
Reverse String	77.7(79.9)	80.1(80.0)	51.3(56.2)	79.3(79.8)
Divide by 2†	71.4	92.0	59.8	70.0
Context Sensitive	Duplicate String	54.3(53.2)	58.0(60.1)	51.3(51.6)	74.2(76.9)
Interlocked Pairing†	60.6	96.7	53.8	99.7
Odds First	56.5(56.5)	60.8(61.3)	61.4(61.4)	73.2(72.3)
Binary Addition†	48.6	55.4	51.9	71.3
Binary Multiplication†	50.1	52.3	48.9	60.9
Compute Squrt	57.9(57.4)	64.4(64.1)	56.3(60.3)	66.3(66.7)
Bucket Sort	59.3(55.9)	96.7(97.5)	71.4(78.7)	97.2(98.7)
Table 2: Generalization accuracy (%) on high-correlation CL-AR scenario, averaged on three repeats. Joint refers to simultaneously training all tasks, with tasks alternating at each step and ultimately being learned uniformly. Single refers to averaging the accuracy of each individual task. CL Initial averages the accuracy immediately after learning each task, while CL Final is the final average accuracy. CL Final values that are greater than  $90\%$  or the Joint accuracy were marked in bold, as well as the positive BWT values.
Task	Model	Average Accuracy (%)
Joint	Single	CL Initial	CL Final	BWT
Modular Arithmetic	RNN	40.32	99.70	61.08	59.68	-1.64
Stack-RNN	54.24	97.98	63.27	62.66	-0.72
Tape-RNN	43.68	84.72	30.84	24.58	-7.31
LSTM	93.07	99.96	96.38	96.25	-0.15
Cycle Navigation	RNN	96.33	99.36	98.57	98.09	-0.56
Stack-RNN	96.97	99.83	98.52	97.88	-0.75
Tape-RNN	56.93	88.65	65.46	67.76	2.67
LSTM	98.72	96.18	99.54	99.37	-0.20
Reverse String	RNN	46.46	64.11	42.64	40.42	-2.60
Stack-RNN	60.00	67.84	68.39	67.69	-0.81
Tape-RNN	24.77	40.53	42.93	39.48	-4.02
LSTM	46.35	65.98	50.73	48.91	-2.12
Bucket Sort	RNN	51.43	56.06	54.25	51.36	-3.36
Stack-RNN	93.49	94.19	95.16	92.44	-3.18
Tape-RNN	73.28	74.54	79.22	77.54	-1.96
LSTM	99.92	98.40	99.55	99.97	0.48
4.2 CLeAR on CL-AR
In this section, we demonstrate that the CLeAR can be applied to various abstract logical tasks, regardless of the model type and difficulty level. Firstly, we conducted experiments on a scenario where a single task becomes progressively more challenging (high-correlation scenario). Since the sequentially presented tasks require similar logical reasoning, there is a significant level of correlation between these tasks. For example, in the same Cycle Navigation task, the difficulty increases as the cycle size expands, while in Modular Arithmetic, the difficulty increases as the modular
Table 3: Generalization accuracy (%) on in-hierarchy CL-AR scenario, averaged on three repeats. Each task is abbreviated to: Cycle Navigation(CN), Modular Arithmetic(MA), Parity Check(PC), Even Pairs(EP), Divide by 2(DT), Reverse String(RS), Stack Manipulation(SM), Compare Occurrence(CO), Binary Addition(BA), Binary Multiplication(BM), Bucket Sort(BS), Compute Squrt(CS), Duplicate String(DS), Interlocked Pairing(IP), Odds First(OF).
Task	Model	Average Accuracy
Joint	Single	CL Initial	CL Final	BWT	FWT
Regular	RNN	97.61	99.22	99.85	47.50	-69.80	-
CN-MA-PC-EP	Stack-RNN	96.50	99.79	99.42	47.64	-69.04	-
w/o CLeAR	Tape-RNN	71.05	83.01	98.24	47.44	-67.73	-
LSTM	98.73	96.96	98.02	50.65	-63.15	-
Regular	RNN	97.79	99.22	98.72	53.80	-59.90	-
EP-PC-MA-CN	Stack-RNN	96.50	99.79	99.97	55.11	-59.82	-
w/o CLeAR	Tape-RNN	71.05	83.01	68.24	47.56	-27.58	-
LSTM	98.73	96.96	97.48	53.61	-58.48	-
Regular	RNN	97.61	99.22	96.45	96.17	-0.37	-4.39
CN-MA-PC-EP	Stack-RNN	96.50	99.79	94.89	87.09	-10.40	-10.47
Tape-RNN	71.05	83.01	76.31	76.70	0.53	-8.43
LSTM	98.73	96.96	98.21	96.49	-2.29	0.97
Regular	RNN	97.79	99.22	99.68	99.80	0.16	0.72
EP-PC-MA-CN	Stack-RNN	95.22	99.79	98.19	99.28	1.45	-0.50
Tape-RNN	59.61	83.01	58.16	74.45	21.72	-27.17
LSTM	98.72	96.96	99.99	99.96	-0.05	4.01
Context Free	RNN	65.95	75.66	64.13	61.67	-3.27	-14.13
DT-RS-SM-CO	Stack-RNN	69.64	88.88	79.14	69.95	-12.26	-17.63
Tape-RNN	54.06	71.09	62.26	62.07	-0.25	-9.77
LSTM	82.72	82.86	76.93	79.36	3.24	-4.57
Context Free	RNN	65.95	75.66	64.87	63.89	-1.30	-14.96
CO-SM-RS-DT	Stack-RNN	69.64	88.88	69.74	67.59	-2.86	-26.85
Tape-RNN	54.06	71.09	74.42	67.67	-9.01	0.63
LSTM	82.72	82.86	75.69	74.78	-1.21	-9.14
Context Sensitive	RNN	55.33	55.33	50.77	50.51	-0.34	-5.28
BA-BM-BS-CS-DS-IP-OF	Stack-RNN	52.70	69.19	61.06	60.52	-0.72	-9.53
Tape-RNN	45.97	56.43	46.77	45.97	-1.07	-12.60
LSTM	76.73	77.54	73.48	74.11	0.84	-3.38
value grows. Table 2 shows that CLeAR demonstrated very low forgetting for the most tasks, and in some cases, backward knowledge transfer occurred. In general CL, Joint is considered a soft upper bound [10]. We have highlighted CL Final when it exceeds that of Joint, as well as when it surpasses  $90\%$ , indicating successful learning of all tasks. Notably, CLeAR effectively preserve the best information they have learned even in cases where a task is not fully learned (low CL Initial) or when learning a specific task is deemed impossible for such a model [11]. Secondly, we conducted experiments on distinct tasks that belong to the same level of the Chomsky hierarchy (in-hierarchy scenario). This represents scenarios where the tasks are of different types but have similar difficulty levels. As evident from the first two rows of Table 3, simple sequential learning without using CLeAR completely forgets the previous task while learning the next task. In contrast, CLeAR exhibits low levels of forgetting for tasks across all Chomsky hierarchies and demonstrates multiple instances of backward/forward knowledge transfer (Table 3). Note that even in cases not highlighted in bold, CLeAR exhibited very little forgetting or near Joint performance. Lastly, we explored CL for tasks belonging to different levels of the Chomsky hierarchy (inter-hierarchy scenario). This entails tasks that differ both in type and difficulty level. Table 4 shows both RNN and LSTM are capable of storing information on tasks of different levels. During the learning process, which includes relatively long and completely unrelated tasks, both models successfully remember all the tasks learned from the very beginning.
Table 4: For inter-hierarchy CL-AR scenario, average generalization accuracy  $(\%)$  is obtained for each group of tasks, averaged on three repeated experiments. The scenario consisted of ten AR tasks that followed an ascending order in the Chomsky hierarchy: EP-CN-PC-CO-RS-SM-IP-OF-BS-DS (abbreviations same with Table 3)
Model	Chomsky hierarchy	Average Accuracy (%)
Joint	Single	CL Initial	CL Final	BWT
RNN	Regular	60.50	100.0	93.31	75.04	-27.40
Context Free	59.94	77.10	61.59	60.78	-0.81
Context Sensitive	49.48	57.68	47.13	46.99	-0.19
LSTM	Regular	96.76	97.37	100.0	93.46	-9.06
Context Free	79.70	87.10	80.94	79.11	-1.83
Context Sensitive	83.38	86.08	79.60	77.67	-2.57
(a) Robustness against Different Noise Level
(b) Visualization of Noise Value of Each Task
Figure 4: An analysis of the impact of noise in the mapping space on CLeAR and an assessment of the amount of noise present in each task. In (a), we show that the proposed framework is robust enough till the noise level of std 0.06 for five continual tasks. In (b), we provide the noise level of the task's mapping space distribution for all 15 tasks and show they are all smaller than the notable noise level.
4.3 Robustness to Distribution Shift
Although we aim to map the input distribution to become uniformly distributed, this is not always possible. We provide a quantitative analysis of whether the model can maintain high CL performance without catastrophic forgetting, even in situations where the mapping distribution is not uniform. We conducted experiments by adding Gaussian noise with specific standard deviation (std) to the uniform probability distribution on the model's mapping space. As shown in Fig. 4(a), the model exhibited robustness even in the presence of significant noise (std up to 0.06 induces forgetting of  $5\%$  or less for the five continual tasks). For our mapping strategy, the std of mapping space distribution is much smaller than the notable noise level for all 15 tasks as shown in Fig. 4(b). This indicates that CLEAR performs well for task mappings with distributions that deviate to some extent from a uniform distribution, demonstrating high robustness.
4.4 Joint training of AR tasks
We discovered from the experimental results in Tables 2, 3 and 4 that continual learning of tasks for CL-AR often showed higher performance compared to when all tasks were learned simultaneously (Joint). This contrasts with the conventional view of continual learning that regards joint training accuracy as a soft upper bound [10]. Therefore, we analyzed the characteristics of the model in continual learning and simultaneous learning. First, we visualized the model's internal feature space for the simplest two tasks, Parity Check (PC) and Even Pairs (EP), as described in this paper [11], to illustrate how the model's learning occurs in each scenario in Fig. 5. From all the plots, we can observe that after the model learns PC or EP, it internally learns the corresponding automata for each task. Furthermore, after CL or joint training, it's evident that the hidden feature is clustered to solve both tasks simultaneously (Fig. 5 C, D). When learning EP following PC, the distribution of the hidden features remains similar, and it exhibits a pattern where each cluster further splits. However, in the case of joint training, the hidden features occupy a broader space, and unlike CL, the states for PC are clustered into eight pieces.
Figure 5: The A represents the automata for Parity Check (PC, up) and Even Pairs (EP, down). Each state (S1-S4) is color-coded to match the features of B, C, and D, with the final states for positive labels highlighted in yellow. Plots show the distribution of the model's hidden features using PCA visualization: after PC training (B), after PC-EP CL (C), and after PC-EP joint training (D). The upper row corresponds to PC automata states, and the lower row corresponds to EP automata states.
The difference between the two learning methods can be even more clearly observed in the following experiment. We separated the features of each model, trained with CL and joint training, based on the two consecutive digits in Fig. 6. These consecutive two digits are not necessary for learning PC or EP but represent easy-to-learn patterns that the model can easily grasp. In the case of joint training, this pattern is learned, and the feature is well separated, while in the case of
CL, this information is not learned. These observations suggest that when learning tasks sequentially through CLeAR, previously learned tasks can act as a regularizer, reducing the acquisition of unnecessary information for the task.
Figure 6: Clustering on two consecutive digits
Our study elucidates the distinct features of CL-AR as compared to conventional image-based CL and highlights the limitations of traditional CL approaches. Additionally, we introduce CLeAR, a novel methodology for the continual learning of abstract logical concepts. Experiments on 15 tasks with varying difficulty validate the efficacy of CLeAR. It achieves near-zero forgetting and often improves accuracy on previous tasks during subsequent training. There are major challenges specific to CL-AR tasks: the decorrelation between input data and tasks, dynamic dimensions of datasets, and the goal of generalization for OOD data. CLeAR tackles these challenges by employing a one-to-many mapping strategy that aligns tasks of different dimensions and shares mapping space. This prevents forgetting previous tasks and enhances overall performance. Our research contributes to the development of lifelong learning abilities, resembling human cognitive processes. By overcoming the limitations of existing CL approaches and focusing on abstract concepts, CLeAR opens new possibilities for continual learning. The limitation of our approach is that the mapping strategy used in our method involves discretized mapping, making it challenging to train the accompanying model end-to-end.
This work was supported by Electronics and Telecommunications Research Institute (ETRI) grant funded by the Korean government. [23ZS1100, Core Technology Research for Self-Improving Integrated Artificial Intelligence System], Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) [2021-0-01343: Artificial Intelligence Graduate School Program (Seoul National Univ.) and 2021-0-02068: Artificial Intelligence Innovation Hub], National Research Foundation of Korea (NRF) grant funded by MSIT (2022R1A3B1077720 and 2022R1A5A708390811) and the BK21 FOUR program of the Education and Research Program for Future ICT Pioneers, Seoul National Univ. in 2023.
[39] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04671, 2016.
[40] Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International Conference on Machine Learning, pages 4548-4557. PMLR, 2018.
[41] Michael Sipser. Introduction to the theory of computation. PWS Publishing Company, 1997.
[42] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017.
[43] Mirac Suzgun, Sebastian Gehrmann, Yonatan Belinkov, and Stuart M. Shieber. Memory-augmented recurrent neural networks can learn generalized dyck languages. CoRR, 2019.
[44] Lazar Valkov, Dipak Chaudhari, Akash Srivastava, Charles Sutton, and Swarat Chaudhuri. Houdini: Lifelong learning as program synthesis. Advances in neural information processing systems, 31, 2018.
[45] Vladimir Vapnik. Statistical learning theory. Wiley, 1998.
[46] Liyuan Wang, Mingtian Zhang, Zhongfan Jia, Qian Li, Chenglong Bao, Kaisheng Ma, Jun Zhu, and Yi Zhong. Afec: Active forgetting of negative transfer in continual learning. Advances in Neural Information Processing Systems, 34:22379-22391, 2021.
[47] Zhen Wang, Liu Liu, Yiqun Duan, Yajing Kong, and Dacheng Tao. Continual learning with lifelong vision transformer. In CVPR, pages 171-181, 2022.
[48] Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry Heck, Heming Zhang, and C-C Jay Kuo. Class-incremental learning via deep model consolidation. In WACV, pages 1131-1140, 2020.