Title: CLeAR: Continual Learning on Algorithmic Reasoning for Human-like Intelligence
Venue: NeurIPS 2023
Accepted: yes
Keywords: Continual learning

Knowledge transfer

Algorithmic reasoning
Abstract: Continual learning (CL) aims to incrementally learn multiple tasks that are presented sequentially. The significance of CL lies not only in the practical importance but also in studying the learning mechanisms of humans who are excellent continual learners. While most research on CL has been done on structured data such as images, there is a lack of research on CL for abstract logical concepts such as counting, sorting, and arithmetic, which humans learn gradually over time in the real world. In this work, for the first time, we introduce novel algorithmic reasoning (AR) methodology for continual tasks of abstract concepts: CLeAR. Our methodology proposes a one-to-many mapping of input distribution to a shared mapping space, which allows the alignment of various tasks of different dimensions and shared semantics. Our tasks of abstract logical concepts, in the form of formal language, can be classified into Chomsky hierarchies based on their difficulty. In this study, we conducted extensive experiments consisting of 15 tasks with various levels of Chomsky hierarchy, ranging from in-hierarchy to inter-hierarchy scenarios. CLeAR not only achieved near zero forgetting but also improved accuracy during following tasks, a phenomenon known as backward transfer, while previous CL methods designed for image classification drastically failed.

CLeAR: Continual Learning on Algorithmic Reasoning for Human-like Intelligence

Abstract

Continual learning (CL) aims to incrementally learn multiple tasks that are presented sequentially. The significance of CL lies not only in the practical importance but also in studying the learning mechanisms of humans who are excellent continual learners. While most research on CL has been done on structured data such as images, there is a lack of research on CL for abstract logical concepts such as counting, sorting, and arithmetic, which humans learn gradually over time in the real world. In this work, for the first time, we introduce novel algorithmic reasoning (AR) methodology for continual tasks of abstract concepts: CLeAR. Our methodology proposes a one-to-many mapping of input distribution to a shared mapping space, which allows the alignment of various tasks of different dimensions and shared semantics. Our tasks of abstract logical concepts, in the form of formal language, can be classified into Chomsky hierarchies based on their difficulty. In this study, we conducted extensive experiments consisting of 15 tasks with various levels of Chomsky hierarchy, ranging from in-hierarchy to inter-hierarchy scenarios. CLeAR not only achieved near zero forgetting but also improved accuracy during following tasks, a phenomenon known as backward transfer, while previous CL methods designed for image classification drastically failed.

1 Introduction

From an early age, humans develop their ability to solve complex logical reasoning problems through lifelong learning. They gradually learn sequential cognitive skills, from basic counting methods to more advanced concepts such as addition, subtraction, and logical operations [2]. Although this sequential learning ability is inherent in humans, deep learning models experience catastrophic forgetting [30], where their performance on previous tasks rapidly declines, after learning a new task. Continual learning (CL) [6, 19] is essential not only for practical purposes but also for developing human-like artificial intelligence that can maintain its performance on previous tasks after learning new tasks. However, existing CL research has been limited to learning streams of structured data with fixed input formats, such as images, where each task involves the same classification task only with different image classes [47, 1, 48]. Current approaches fail to reflect the learning of abstract logical concepts that humans continuously acquire in the real world.

There are three major challenges in naively leveraging existing CL methodologies to continual algorithmic reasoning (AR) tasks. Firstly, in AR tasks, input data contain little information, and the input data and task are decorrelated (Fig. 1b). In traditional image CL, the inherent information in the image itself enables the model to estimate which task the image belongs to [18]. However, in CL for abstract concepts, the input data may be completely decorrelated from the task that the model needs

t

⁕

SUM of sequence

0101: Even

1001010: Odd

00101011: Even

11010100: Even

0010: Odd

3+1 = 4 (Mod5)

1

×

2-4 = 4 (Mod5)

3

×

3-1+2 = 0 (Mod5)

2

×

4

×

2 = 3 (Mod5)

2

×

1-3

×

4+1 = 1 (Mod5)

⋮

⋮

(a) Sequential Tasks of Learning Logics

110

11010

0110101

⋯

T1.

SUM is

even(odd)

T2.

First & Last digit is

same(different)

𝑋~{0,1} 𝑙𝑒𝑛𝑔𝑡ℎ

Variant tasks

Identical input distribution

(b) Input-Task Decorrelation

to learn. For example, consider a binary input sequence consisting of 0's and 1's. In the first task, the model needs to determine whether the sum of the sequence is odd or even, while in the second task, it needs to determine the square root of the sequence when treated as a binary number. In this case, the input data distribution is identical for both tasks, thus it is impossible to correctly predict the task based on the input data. Therefore, CL for AR of abstract concepts is better suited to task incremental learning. Moreover, since the input data contains little information, learning data distribution (e.g. contrastive learning based CL) is inapplicable.

Secondly, the dimensions of the datasets of each task can be dynamic in the AR of abstract concepts. In current CL, data has mostly fixed shape and dimension (ex. height, weight, and channel H × W × C ) throughout the tasks [47, 1, 48]. However, ways of representing each 'abstract logical concept' as the sequential input-output format are various [11, 26]. For example, when trying to train the model for modular arithmetic, there are various ways to introduce the concepts of numbers and operators to the model. Specifically, all used numbers and operators can be one-hot encoded numbers, and operators can be represented in different dimensions, or numbers can be represented in binary format. Furthermore, input sequences of different tasks result in different dimensions and lengths. Therefore, it is impossible to apply the same model directly to each task's data like the traditional image CL methodology, and additional operations adjusting the data shape are necessary.

Thirdly, the goal of generalization that the AR task aims to achieve is different. In traditional CL, the model learns the input data distribution itself, assuming that the train dataset and test dataset are identical and independently distributed (IID) [45]. However, in AR tasks, the model learns the general rule that generates labels from input distribution. To identify whether the model has appropriately learned to generate the correct answers rather than simply memorizing them, the test set is composed of out-of-distribution (OOD) data [33] that was never seen during training. Therefore, the goal of AR is not generalization performance for IID as in traditional CL, but generalization for OOD [14]. Due to these characteristics, the performance of replay-based CL methods that use a replay buffer to represent the input data distribution decreases. The core of the replay buffer is to extract representative samples or prototypes that can represent the input data distribution [5]. However, unlike a sharp-eared and long-tailed image that represents a cat class, '5+8=13' cannot be considered as representative of the concept of arithmetic.

This work. We have developed a CL methodology that can be applied in the aforementioned scenario. We have applied a novel one-to-many mapping strategy to handle input data of different formats, and through this mapping, we have proposed ways to prevent forgetting previous tasks when learning new tasks, and even increased performance on previous tasks during the process of learning new tasks. We conducted extensive experiments on sequential models such as RNN, LSTM, and Tape RNN [12, 15, 17, 43, 11], which have memory, and used AR tasks with various levels of Chomsky hierarchy [11] to conduct extensive experiments on sequential tasks of the same level and different levels.

Weintroduce the scenario of CL on AR for the first time. This scenario is of great importance as it is similar to how humans learn logical reasoning abilities incrementally in the real world and has very different characteristics from conventional CL scenarios as described above.

We introduce new data mapping methods and CL algorithms that are applicable to the aforementioned scenario and have demonstrated high performance not only in catastrophic forgetting but also in knowledge transfer, which is one of the ultimate goals of CL.

Wedemonstrated the superiority of our method over the existing CL methods in our proposed CL scenario for AR through extensive experiments on various combinations of AR tasks. We also showed that existing CL methods perform poorly due to the reasons described above.

2 Preliminary

Continual learning. CL is one of the long-standing challenges in deep learning [35], aiming to preserve the performance of previous tasks while effectively learning new tasks (without access to data from previous tasks) in situations where multiple tasks are learned sequentially. The existing CL methodologies can be broadly classified into three categories based on the methods used to prevent catastrophic forgetting [24]. First, regularization-based methods [19, 22, 21] constrain the model's important parameters for the previous task from changing when learning a new task. Elastic Weight Consolidation (EWC) [19] approximates the Fisher information matrix to learn a loss-minimizing weight space and regularizes the gradient direction to maintain the low loss for previous tasks. Learning without forgetting (LwF) [22] regularizes the loss through knowledge distillation by using the output of previous tasks as pseudo-labels during the learning of new tasks. Second, memory-based methods [5, 27, 4, 37] are further divided into experiment-replay and orthogonal projection methods. Experiment replay (ER) [5] stores small amounts of data from previous tasks in memory to use their information when learning new tasks. The performance of this method depends on the representative sample selected to capture the entire distribution of data. Orthogonal projection methods [27, 4] stores gradient and update weight in a direction orthogonal to the gradient from the previous task. Third, parameter isolation-based methods [39, 28, 40] use different parts of the model's parameters for each task. The parameters used in previous task learning are masked during the learning of the next task, and the model can also be extended as new tasks are introduced.

Knowledge Transfer. Recent CL models aim not only to prevent catastrophic forgetting of previous tasks but also to enhance the overall performance of the entire task by transferring common knowledge between tasks [24]. This is a property that humans, as excellent continual learners, naturally possess, allowing them to learn new tasks more easily from previous experiences and gain a deeper understanding of previously acquired knowledge as they learn new tasks. However, such knowledge transfer is not easy for deep learning models to obtain. Backward transfer refers to improving the performance of previously learned tasks by learning the current task, while forward transfer refers to previously learned tasks helping to learn a new task. Although there have been studies in this area [39, 25], knowledge backward transfer is still hard to achieve [46].

Algorithmic Reasoning. The core of human logical reasoning ability lies in inductive inference [20, 36]. This is the ability to derive general rules from finite examples and use them to infer the correct answers from unseen data. From the perspective of deep learning, this is AR [11, 23], where the model receives a finite number of input-output data and infers the underlying general rule to generate output for input sequences that have not been seen before. There has been much research on program induction [34, 23, 32, 31, 11], however, conventional approaches for serial AR have a critical limitation in that they require prior human knowledge about the task to pre-configure the model [29, 44, 13]. In general deep learning, the model learns through empirical risk minimization of statistical learning theory to achieve the theoretical bound of its performance and estimates the generalization error using test data [45]. An important assumption here is that the train and test data are independent and identically distributed (IID). However, in AR, where the model needs to learn general rules, this assumption is violated [11]. Therefore, in AR tasks, we have to measure the model's performance not only on the train data distribution but also on out-of-distribution (OOD) distribution [11].

Formal Language and Chomsky Hierarchy. Computational problems that generate output through learnable rules can be represented in formal language [41, 8, 38]. Formal language can be classified into hierarchies based on their complexity, which is known as the Chomsky hierarchy [3, 9]. The hierarchy ranges from regular language, which requires the lowest complexity of memory, contextfree language, which requires stack-type memory, context-sensitive language, which requires linear tape memory, and recursively enumerable language, which requires infinite tape memory. Research has been conducted on whether various deep learning models can learn each level of formal language in this hierarchy [11, 12, 7]. Results have shown that there is a Chomsky hierarchy bound on the formal language that each deep learning model can learn, whereas models with external memory can learn languages of a high Chomsky hierarchy.

3 Method

Problem Statement. Our goal is to train a fixed-size model that sequentially learns various abstract logical tasks, following a standard CL scenario. We assumed that predicting the output sequence corresponding to the input sequence, as in [11], is closer to real-world scenarios which are learned through inductive inference than inferring the internal state of automata, as in [26]. There are various methods of representing the same abstract logical concept into an input sequence that a model can understand. However, different forms of input can affect the performance of the model. Therefore, for fairness, we used the data format proposed in the previous paper [11].

In a CL scenario, a sequence of tasks T = { task t } T t =1 is given one after another. Each task t is given as a dataset D t = { ( x t,i , y t,i ) } S t i =1 with S t sample pairs. Each pair of x t,i , y t,i comes from { One-hot ( L t in ) ∈ { 0 , 1 } ( | ∑ t in | , l in ≤ N ) , One-hot ( L t out ) ∈ { 0 , 1 } ( | ∑ t out | , l out ) } , where L t in ∈ { ∑ t in } l in ≤ N and L t out ∈ { ∑ t out } l out is input and output language pair from task t . N is given as the input sequence length limit. ∑ t in and ∑ t out are corresponding alphabets.

Consider a sequence-to-sequence model f θ ( · ) parameterized by fixed-sized θ . To match different sized input x t,i of each task to model input, we require a task-wise mapping function m t ( · ) . Also, to project the model output vector to match different sized output y t,i , we applied task-wise linear projection head h t ( · ) which consists of a single layer perceptron. After learning a k -th task, the model is parameterized by θ ( k ) , and the projection head becomes h t,k ( · ) .

During training, the model can only access data from the current task. And after training, to evaluate whether the model actually learned the general output sequence generation rule, the model is tested on OOD dataset D test t = { ( x test t,i , y test t,i ) } S test t i =1 which are created in the same method as train dataset from L t in ∈ { ∑ t in } N<l in ≤ M and its corresponding output sequence L t out ∈ { ∑ t out } l out . After learning a k -th task, model performance on task t is measured using an average of per-sequence accuracy of test data, i.e., ACC test k,t .

Following [11], we set the maximum training input sequence length to be 40 ( N = 40) and sampled input sequence length from the uniform distribution U (1 , N ) . OOD test set input sequence length was sampled from the uniform distribution U ( N +1 , M ) where N = 40 and M = 100.

Tasks. We conducted a comprehensive evaluation of CL on AR (CL-AR), involving 15 diverse logical reasoning tasks. These tasks exhibit various levels of complexity, resulting in different Chomsky hierarchies. Table 1 shows our AR tasks and the corresponding Chomsky hierarchy. The detailed information for each task is in Appendix A. We conducted three major CL scenarios. (a) High-correlation CL-AR scenario continually learns the same kind of task with varying difficulty. One example is the Modular Arithmetic task with increasing modular value enabling the model to access larger numbers. Another example is the Cycle Navigation task with increasing cycle length. This scenario allows us to interpret the model's behavior upon learning highly correlated reasoning tasks. (b) In-hierarchy CL-AR scenario continually learns various tasks of the same level of Chomsky hierarchy. This scenario allows us to analyze the interaction among tasks of different natures but with similar complexity. (c) Inter-hierarchy CL-AR scenario consists of tasks of different Chomsky hierarchies. This is by far the most challenging scenario for a neural network that has to manipulate its memory structure.

Models. Recently, there have been numerous quantitative studies on whether various architectures of neural networks can learn formal languages. In recent research, it has been shown that regardless of the increase in the number of training data, models have "hard limitations for scaling laws" and cannot learn generalization rules based on the complexity of the task and the inductive bias of the model. Furthermore, to learn high-level tasks, models with external memory are necessary, and the commonly used transformer model has demonstrated very limited AR performance. Therefore, in

Task 2

Task 1

Range = Codomain

Mapping space

Domain of valid input

Domain of valid input

Valid input

for both tasks

Model

{0,1}

(𝑫𝒊𝒎 𝟏, 𝑙)

{0,1}

(𝑫𝒊𝒎 𝟐, 𝑙)

Figure 2: m t maps this input space to the entire mapping space, which is shared across all tasks. In order to map the entire space, mapping from the input space to the mapping space become one-tomany, making an arbitrary point in mapping space to be mapped from a valid input sequence for every task. Also, this mapping function ensures that the distribution of inputs from the input space becomes a uniform distribution in the mapping space.

this paper, we used RNN [12] and LSTM [15], as well as stack RNN [17] and tape RNN [43, 11] which have additional differentiable memory structures added to the standard RNN. The length of the external memory stack and tape in stack RNN and tape RNN was set to 40, and each element had 8 dimensions [11] and dimensions were doubled with CL of more than 5 tasks.

Evaluation Metrics. We adopt generalization accuracy as the basic evaluation metric. Final model accuracy (ACC) is the average accuracy of all tasks after learning all tasks as follows:

In order to evaluate not only the catastrophic forgetting but also the model's knowledge transfer capability, we introduced a metric that measures backward/forward knowledge transfer [42, 16]. Backward knowledge transfer (BWT) [27] indicates how much learning on the next tasks has influenced the performance of the previous task. BWT > 0 indicates that there has been a transfer of beneficial knowledge to the previous task during the CL process. Forward knowledge transfer (FWT) measures how helpful previously learned knowledge is when learning a new task. FWT is obtained by calculating the difference between the accuracy achieved in the CL scenario and the accuracy achieved by training an independent model (as Table 1). BWT and FWT are expressed as follows:

CLeAR. We propose a novel CL algorithm for the CL-AR scenario: CLeAR. The CLeAR model is divided into two main parts. The first part is a mapping function m t that maps sequences sampled from the input data distribution to a shared mapping space. The second part consists of the main model f θ ( · ) , which learns labels from the features in the feature space, and the task-wise single-layer projection head h t .

m t maps the input sequence sampled from the input space of task t to a feature space (mapping space), preserving length but changing dimension to fixed size D map . The input sequence consists of binary values, denoted as { 0 , 1 } ( D t in ,l ) , and the range of the mapping space is expressed as { 0 , 1 } ( D map ,l ) . As described in Fig. 2,

Algorithm 1 CLeAR -training procedure for a given task t

3. Mean 0.5

2. Decorrelation

Mapped Sequence

1  0  0  1  0

1  1  0  1  1

1  0  0  1  1

1  0  0  1  0

0  1  0  0  1

0  1  0  0

0  0  0  0

Mapped Sequence

0/1

0/1

1  0  0  1  0

0  1  0  0  1

0  1  0  0

0  0  0  0

1  1  0  1  0

0  0  1  0

Decoder

z

1  0  0  1  0

1  1  0  1  1

1  0  0  1  1

0/1

0/1

0/1

Encoder

0/1

1. Perfect Reconstruction

our mapping function m t has the following characteristics. First, an arbitrary sample from the input distribution is mapped to one or more points in the mapping space (one-to-many). Second, for any point in the mapping space, a corresponding sample exists in the input space. Third, the mapping of input distribution resembles uniform sampling in the mapping space: m t ( inputdistribution ) ≈ U ( { 0 , 1 } ( D map ,l ) ) . Therefore, the mapping function of each task expands the input distribution to fill the mapping space fully. Thus, an arbitrary sample drawn from the mapping space can correspond to a valid input sample for any task. This allows the model to consider a single sequence in the mapping space as valid input for each previous task. It is similar to how we can perceive the number 395 in various ways, such as being odd, multiple of 5, or having a sum of digits equal to 17. These mapping functions can be expressed very simply, requiring minimal computational resources in the overall model. Additional details about the mapping function, positional mapping, and specific mapping for each task are explained in Appendix C.

To create such mapping functions, an encoder-decoder structure is considered for tasks where the distribution of alphabets is preserved across all positions of the input language. As shown in Fig. 3, the encoder takes the input sequence sampled from the input space and additional random noise to generate a mapped sequence. Then, the decoder reconstructs the input sequence from the mapped sequence. In this process, each column of the mapped sequence should have a mean of 0.5 and should be decorrelated from each other. The encoder serves as the mapping function, and the decoder is discarded.

The functions f θ ( · ) and h t ( · ) are trained using a similar approach to LwF [22]. After completing the training for the t -1 th task, the training proceeds for the t th task, temporarily saving f θ ( t -1) and h k,t -1 for this task. First, m t is trained. Then, the randomly initialized h t is pre-tuned while the rest of the model is frozen using a small portion of the dataset. For each sample x t,i , y t,i , the pseudo-label for the k -th ( 1 ≤ k < t ) task ˆ y k,i is obtained by argmax · h k,t -1 · f θ ( t -1) · m t ( x t,i ) . The loss is calculated for all t heads. For the outputs of the k -th task heads: h k · f θ · m t ( x t,i ) ( 1 ≤ k ≤ t ), the loss is computed with the pseudo-labels ˆ y k,i for 1 ≤ k < t , and for the t -th task head, the loss is computed with y t,i . The final loss is the sum of the losses for each head up to t -th. The overall learning process is shown in Algorithm 1.

4 Experiments

In this section, we first confirmed that applying our mapping function and tuning structure did not alter the learning ability of each model on tasks from various Chomsky hierarchies, compared to the previous study [11]. Next, we demonstrate CLeAR's superior performance on high-correlation, in-hierarchy, and inter-hierarchy CL-AR tasks through extensive ablations. Furthermore, we analyze the model's robustness in terms of noise within the mapping space. A detailed explanation of the experiment setting and comparison with existing CL algorithms [19, 22, 5] and their failure is shown in Appendix D.

4.1 Baseline Generalization Accuracy

In Table 1, we presented the generalization accuracy of each model for all tasks in our paper. Corresponding in-distribution accuracy and reported generalization accuracy in the previous paper [11] paper is shown in Appendix B. It was shown that applying mapping and head did not greatly affect the model's performance and did not alter the limits of the Chomsky hierarchy that each model can learn. The reported performance here was used as a benchmark for calculating Single-task accuracy and FWT.

4.2 CLeAR on CL-AR

In this section, we demonstrate that the CLeAR can be applied to various abstract logical tasks, regardless of the model type and difficulty level. Firstly , we conducted experiments on a scenario where a single task becomes progressively more challenging (high-correlation scenario). Since the sequentially presented tasks require similar logical reasoning, there is a significant level of correlation between these tasks. For example, in the same Cycle Navigation task, the difficulty increases as the cycle size expands, while in Modular Arithmetic , the difficulty increases as the modular

value grows. Table 2 shows that CLeAR demonstrated very low forgetting for the most tasks, and in some cases, backward knowledge transfer occurred. In general CL, Joint is considered a soft upper bound [10]. We have highlighted CL Final when it exceeds that of Joint, as well as when it surpasses 90%, indicating successful learning of all tasks. Notably, CLeAR effectively preserve the best information they have learned even in cases where a task is not fully learned (low CL Initial) or when learning a specific task is deemed impossible for such a model [11]. Secondly , we conducted experiments on distinct tasks that belong to the same level of the Chomsky hierarchy (in-hierarchy scenario). This represents scenarios where the tasks are of different types but have similar difficulty levels. As evident from the first two rows of Table 3, simple sequential learning without using CLeAR completely forgets the previous task while learning the next task. In contrast, CLeAR exhibits low levels of forgetting for tasks across all Chomsky hierarchies and demonstrates multiple instances of backward/forward knowledge transfer (Table 3). Note that even in cases not highlighted in bold, CLeAR exhibited very little forgetting or near Joint performance. Lastly , we explored CL for tasks belonging to different levels of the Chomsky hierarchy (inter-hierarchy scenario). This entails tasks that differ both in type and difficulty level. Table 4 shows both RNN and LSTM are capable of storing information on tasks of different levels. During the learning process, which includes relatively long and completely unrelated tasks, both models successfully remember all the tasks learned from the very beginning.

0.6

0.7

0.8

0.9

1

t = 1

t = 2

t = 3

t = 4

t = 5

Accuracy of first task

noise std 0.02

noise std 0.04

noise std 0.06

noise std 0.08

noise std 0.10

(a) Robustness against Different Noise Level

0

0.01

0.02

0.03

EP MA PC CN CO SM RS DT DS

IP

OF BA BM CS BS

Noise level (std)

Tasks

(b) Visualization of Noise Value of Each Task

4.3 Robustness to Distribution Shift

Although we aim to map the input distribution to become uniformly distributed, this is not always possible. We provide a quantitative analysis of whether the model can maintain high CL performance without catastrophic forgetting, even in situations where the mapping distribution is not uniform. We conducted experiments by adding Gaussian noise with specific standard deviation (std) to the uniform probability distribution on the model's mapping space. As shown in Fig. 4(a), the model exhibited robustness even in the presence of significant noise (std up to 0.06 induces forgetting of 5% or less for the five continual tasks). For our mapping strategy, the std of mapping space distribution is much smaller than the notable noise level for all 15 tasks as shown in Fig. 4(b). This indicates that CLeAR performs well for task mappings with distributions that deviate to some extent from a uniform distribution, demonstrating high robustness.

4.4 Joint training of AR tasks

We discovered from the experimental results in Tables 2, 3 and 4 that continual learning of tasks for CL-AR often showed higher performance compared to when all tasks were learned simultaneously (Joint). This contrasts with the conventional view of continual learning that regards joint training accuracy as a soft upper bound [10]. Therefore, we analyzed the characteristics of the model in continual learning and simultaneous learning. First, we visualized the model's internal feature space for the simplest two tasks, Parity Check (PC) and Even Pairs (EP), as described in this paper [11], to illustrate how the model's learning occurs in each scenario in Fig. 5. From all the plots, we can observe that after the model learns PC or EP, it internally learns the corresponding automata for each task. Furthermore, after CL or joint training, it's evident that the hidden feature is clustered to solve both tasks simultaneously (Fig. 5 C, D). When learning EP following PC, the distribution of the hidden features remains similar, and it exhibits a pattern where each cluster further splits. However, in the case of joint training, the hidden features occupy a broader space, and unlike CL, the states for PC are clustered into eight pieces.

1

0

1

0

S1

S3

S2

S1

S3

S2

1

1

0

0

1

0

1

0

0

1

1

0

1

0

1

0

A

B

C

D

even+0

even+0

even+0

even+1

odd+0

[+uaAa

0+ppo

odd+0

even+1

odd+1

odd+1

odd+1

Z-

1.5

-5.0

2.5

00

2.5

5.0

7.5

10.0

0-1

0-0

1-1

0-0

1·1

0-1

0-1

1-1

0-0

1-0

1-0

1-0

7.5

5.0

0.0

2.5

5.0

7.5

10.C

The difference between the two learning methods can be even more clearly observed in the following experiment. We separated the features of each model, trained with CL and joint training, based on the two consecutive digits in Fig. 6. These consecutive two digits are not necessary for learning PC or EP but represent easy-to-learn patterns that the model can easily grasp. In the case of joint training, this pattern is learned, and the feature is well separated, while in the case of

(a) CL

(b) Joint training

1-1

0-0

8

0-0

1·1

0-1

1-0

0-1

1-0

0

D

6

CL, this information is not learned. These observations suggest that when learning tasks sequentially through CLeAR, previously learned tasks can act as a regularizer, reducing the acquisition of unnecessary information for the task.

5 Conclusion

Our study elucidates the distinct features of CL-AR as compared to conventional image-based CL and highlights the limitations of traditional CL approaches. Additionally, we introduce CLeAR, a novel methodology for the continual learning of abstract logical concepts. Experiments on 15 tasks with varying difficulty validate the efficacy of CLeAR. It achieves near-zero forgetting and often improves accuracy on previous tasks during subsequent training. There are major challenges specific to CL-AR tasks: the decorrelation between input data and tasks, dynamic dimensions of datasets, and the goal of generalization for OOD data. CLeAR tackles these challenges by employing a one-to-many mapping strategy that aligns tasks of different dimensions and shares mapping space. This prevents forgetting previous tasks and enhances overall performance. Our research contributes to the development of lifelong learning abilities, resembling human cognitive processes. By overcoming the limitations of existing CL approaches and focusing on abstract concepts, CLeAR opens new possibilities for continual learning. The limitation of our approach is that the mapping strategy used in our method involves discretized mapping, making it challenging to train the accompanying model end-to-end.

References

Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In ECCV , pages 139-154, 2018.

Michael A Arbib. Brains, machines, and mathematics . Springer Science & Business Media, 2012.

Mikael Bodén and Janet Wiles. On learning context-free and context-sensitive languages. IEEE Trans. Neural Networks , 2002.

Arslan Chaudhry, Marc'Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. In International Conference on Learning Representations , 2018.

Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc'Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv:1902.10486 , 2019.

Zhiyuan Chen and Bing Liu. Lifelong machine learning. Synthesis Lectures on Artificial Intelligence and Machine Learning , 12(3):1-207, 2018.

Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoderdecoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing , 2014.

Noam Chomsky. Three models for the description of language. IRE Trans. Inf. Theory , 1956.

Noam Chomsky. Syntactic structures . Mouton de Gruyter, 2002.

Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence , 44(7):33663385, 2021.

Gregoire Deletang, Anian Ruoss, Jordi Grau-Moya, Tim Genewein, Li Kevin Wenliang, Elliot Catt, Chris Cundy, Marcus Hutter, Shane Legg, Joel Veness, and Pedro A Ortega. Neural networks and the chomsky hierarchy. In The Eleventh International Conference on Learning Representations , 2023.

Jeffrey L. Elman. Finding structure in time. Cogn. Sci. , 1990.

Alexander L Gaunt, Marc Brockschmidt, Nate Kushman, and Daniel Tarlow. Differentiable programs with neural libraries. In International Conference on Machine Learning , pages 1213-1222. PMLR, 2017.

Faustino J. Gomez, Jürgen Schmidhuber, and Risto Miikkulainen. Accelerated neural evolution through cooperatively coevolved synapses. J. Mach. Learn. Res. , 2008.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput. , 1997.

Mingcheng Hou and Issei Sato. A closer look at prototype classifier for few-shot image classification. arXiv preprint arXiv:2110.05076 , 2021.

Armand Joulin and Tomás Mikolov. Inferring algorithmic patterns with stack-augmented recurrent nets. In Advances in Neural Information Processing Systems 28 , 2015.

Gyuhak Kim, Changnan Xiao, Tatsuya Konishi, Zixuan Ke, and Bing Liu. A theoretical study on solving continual learning. arXiv preprint arXiv:2211.02633 , 2022.

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences , 114(13):3521-3526, 2017.

Karl Josef Klauer and Gary D Phye. Inductive reasoning: A training approach. Review of educational research , 78(1):85-123, 2008.

Sang-Woo Lee, Jin-Hwa Kim, Jaehyun Jun, Jung-Woo Ha, and Byoung-Tak Zhang. Overcoming catastrophic forgetting by incremental moment matching. Advances in neural information processing systems , 30, 2017.

Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence , 40(12):2935-2947, 2017.

Percy Liang, Michael I. Jordan, and Dan Klein. Learning dependency-based compositional semantics. Comput. Linguistics , 2013.

Sen Lin, Li Yang, Deliang Fan, and Junshan Zhang. Beyond not-forgetting: Continual learning with backward knowledge transfer. arXiv preprint arXiv:2211.00789 , 2022.

Sen Lin, Li Yang, Deliang Fan, and Junshan Zhang. Trgp: Trust region gradient projection for continual learning. arXiv preprint arXiv:2202.02931 , 2022.

Bingbin Liu, Jordan T Ash, Surbhi Goel, Akshay Krishnamurthy, and Cyril Zhang. Transformers learn shortcuts to automata. arXiv preprint arXiv:2210.10749 , 2022.

David Lopez-Paz and Marc'Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems , 30, 2017.

Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition , pages 7765-7773, 2018.

Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig, Thomas Demeester, and Luc De Raedt. Deepproblog: Neural probabilistic logic programming. Advances in neural information processing systems , 31, 2018.

Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation , volume 24, pages 109-165. Elsevier, 1989.

William Merrill, Ashish Sabharwal, and Noah A Smith. Saturated transformers are constantdepth threshold circuits. Transactions of the Association for Computational Linguistics , 10:843856, 2022.

William Merrill, Gail Weiss, Yoav Goldberg, Roy Schwartz, Noah A. Smith, and Eran Yahav. A formal hierarchy of RNN architectures. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , 2020.

John P Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization. In International Conference on Machine Learning , pages 7721-7735. PMLR, 2021.

Peter Nordin. Evolutionary program induction of binary machine code and its applications . PhD thesis, Dortmund University of Technology, 1997.

German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural networks , 113:54-71, 2019.

Michael Pazzani and Dennis Kibler. The utility of knowledge in inductive learning. Machine learning , 9:57-94, 1992.

Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition , pages 2001-2010, 2017.

Elaine Rich. Automata, Computability and Complexity . Prentice-Hall, 2007.

Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04671 , 2016.

Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International Conference on Machine Learning , pages 4548-4557. PMLR, 2018.

Michael Sipser. Introduction to the theory of computation . PWS Publishing Company, 1997.

Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems , 30, 2017.

Mirac Suzgun, Sebastian Gehrmann, Yonatan Belinkov, and Stuart M. Shieber. Memoryaugmented recurrent neural networks can learn generalized dyck languages. CoRR , 2019.

Lazar Valkov, Dipak Chaudhari, Akash Srivastava, Charles Sutton, and Swarat Chaudhuri. Houdini: Lifelong learning as program synthesis. Advances in neural information processing systems , 31, 2018.

Vladimir Vapnik. Statistical learning theory . Wiley, 1998.

Liyuan Wang, Mingtian Zhang, Zhongfan Jia, Qian Li, Chenglong Bao, Kaisheng Ma, Jun Zhu, and Yi Zhong. Afec: Active forgetting of negative transfer in continual learning. Advances in Neural Information Processing Systems , 34:22379-22391, 2021.

Zhen Wang, Liu Liu, Yiqun Duan, Yajing Kong, and Dacheng Tao. Continual learning with lifelong vision transformer. In CVPR , pages 171-181, 2022.

Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry Heck, Heming Zhang, and C-C Jay Kuo. Class-incremental learning via deep model consolidation. In WACV , pages 1131-1140, 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work tackles the problem of continual learning for rule-based tasks. The method first maps algorithmic inputs to a discrete latent space. This mapping is one-to-many, which allows the method to learn multiple features for the inputs of each task. A model then consumes results from this discrete latent space to perform the task. The authors test and verify the method works on various algorithmic tasks drawn from the Chomsky hierarchy.\n\nSTRENGTHS:\n- The authors define a new set of problems for continual learning\n- The approach for mapping inputs to learned features makes sense\n- Continual learning is becoming more prevalent as models become more expensive to train from scratch, so the work is timely\n\nWEAKNESSES:\n- The framing uses catastrophic forgetting, but whether catastrophic forgetting is a pressing issue in today’s deep models is an open question. We do not know how increasing model scale impacts catastrophic forgetting, and deep networks like GPT-3 are fine-tuned often in production. Recent empirical results on catastrophic forgetting in neural networks are not as dramatic as the authors imply, and these works should be cited. See my comments in the questions section for references.\n- It’s unclear how the experiments in the paper map onto real world problems. Can the authors say more about this? When would we want to train a model sequentially over algorithmic tasks?\n- It seems CLeAR gets outperformed by multitask and single task training. Is there a baseline that would demonstrate the lift of this approach?\n\nSee questions for other concerns.\n\nQUESTIONS:\nMajor questions:\n- Line 229: How do you control the size of the mapping set?\n- How much of the performance is explained by learning regular languages well and nothing else? For example, in the context-free tasks, are the models just solving the subset of examples that can be modeled via a regular language? \n- Is the CL setting better than the pretrain-finetune or meta-learning settings, where we might pretrain/meta-train over multiple tasks and then adapt to one task?\n\nMinor\n- Line 24: The decline is not necessarily rapid. See [this](https://arxiv.org/abs/1910.08475) [this](https://arxiv.org/abs/2102.07686) for more recent citations to catastrophic forgetting.\n- Line 113: “the model performing backward transfer is still limited.” What does this mean?\n- Line 126: Missing citations to [this](https://arxiv.org/abs/2106.16213) [this](https://aclanthology.org/2020.acl-main.43/) and related works on the languages that can be recognized by transformers and RNNs. These works essentially define the upper bounds to your empirical work.\n- Table 3: What is the logic behind the bold numbers here?\n\nNits:\n- Line 16: typo “during following”\n- Line 83: could you spell out LwF?\n- Line 365: duplicated references\n\nLIMITATIONS:\nI would argue that an additional limitation is the latent space design itself. More accurately, the discrete latent space comes with tradeoffs. Because the method does not define a distance metric over the discrete latent space, there is no notion of exemplars or typicality in the mapped sequences. By design, all the sequences represent features of equal importance. However, one might presume that for some tasks, certain features of task deserve a higher weighting than others. This seems easier to solve in a continuous latent space, where you can map the importance of a feature onto distance.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces CLeAR, a novel algorithmic reasoning (AR) methodology for continual learning (CL) of abstract logical concepts. The main component of CLeAR is a one-to-many mapping strategy that aligns various tasks of different dimensions in a shared mapping space, allowing for the gradual learning of abstract concepts. The paper addresses the challenges specific to CL for AR tasks, such as decorrelated input data, dynamic dimensions of datasets, and the goal of generalization for out-of-distribution data. Extensive experiments are conducted, demonstrating that CLeAR achieves near-zero forgetting and improved accuracy during subsequent training, outperforming existing CL methods designed for image classification. The paper highlights the importance of studying CL in the context of abstract logical concepts and offers valuable insights into human learning mechanisms.\n\nSTRENGTHS:\nThe paper introduces a novel methodology, CLeAR, for continual learning of abstract logical concepts. It addresses the lack of research in this area and presents a unique approach to aligning tasks of different dimensions and sharing semantics. By focusing on abstract algorithmic reasoning tasks, the paper bridges the gap between CL and real-world cognitive skills development. The proposed one-to-many mapping strategy addresses the challenges unique to abstract concept learning and offers a solution for preventing forgetting and improving accuracy during subsequent training.\n\nWEAKNESSES:\nFrom my point of view, I'm not sure why end-to-end algorithmic reasoning is important when this problem can easily be solved via symbolic reasoning with pre-defined rules and structures. Also, I think the paper should include some other baseline experiments, like standard CL methods, for comparison. Foundation models can also be considered if possible. This would provide a better understanding of the performance improvement achieved by CLeAR and allow for a more comprehensive assessment of its effectiveness.\n\nQUESTIONS:\n- Could the authors provide further insights on why end-to-end approaches are necessary or beneficial in comparison to symbolic reasoning with pre-defined rules and structures? \n- Have the authors considered including other CL methods or foundation models in the evaluation? I guess most of the tasks listed in the paper can be solved by models like gpt4. So why learning Algorithmic Reasoning important?\n\nLIMITATIONS:\nSee weaknesses and questions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a training framework for continual learning in which task inputs are mapped to regions of or distributions over an embedding space, before the resulting embedding is passed to a single model that learns continually to solve the tasks.  They apply this continual learning framework to tasks based on learning algorithms, testing against input-output pairs from outside the training distribution. The resulting training and task framework can solve a number of algorithmic reasoning tasks, up to the limitations imposed by model architectures.\n\nSTRENGTHS:\nThe paper is one of the first I've seen to tackle algorithmic reasoning/algorithm induction tasks along with continual learning. The core idea of mapping tasks and inputs to an embedding space from which a unified model learns continually to solve tasks (via a task-specific output head) is, to my knowledge, highly novel. The paper is duly careful about which levels of the Chomsky hierarchy it considers as algorithms for purposes of defining the algorithmic reasoning task.\n\nWEAKNESSES:\nThe paper needs to more clearly present its comparisons between a baseline and a contribution.  I understand that the authors do not claim every task on which they evaluated to be \"solved\", especially since many of them require positional encodings and a context-free or context-sensitive memory.  However, since the message is more complicated, the authors could do a better job presenting exactly where their training framework and meta-model architecture makes a difference against, for instance, just training the baseline neural network architectures with no continual learning, or training them and then retraining with a baseline level of catastrophic forgetting.\n\nThe authors have since conducted a number of additional experiments to address this weakness.\n\nQUESTIONS:\nWhere do the authors think some form of hierarchical representation or compression of previous \"knowledge\" is happening to enable continual learning?\n\nThe authors have submitted supplementary material on exactly this question.\n\nLIMITATIONS:\nThe authors duly discuss the chief limitation of their approach being a discretization in the embedding space.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors introduce CLeAR, a new method of Continual Learning (CL) for Algorithmic Reasoning (AR). In doing so, several relaxations to standard CL are discussed: (1) scenarios where the same data are used for different tasks, (2) the variation of the input, being not of fixed size, and (3) analysis of performances on OOD data. The main algorithm proposed, namely CLeAR, combines a map to a shared feature space with a RCNN or a LSTM where learning happens, and uses an additional MLP to project in the output space. The CL strategy for catastrophic forgetting builds on LwF. \n\nTo evaluate the proposed model, the authors propose 15 benchmarks of increasing difficulty, based on the use of different Chomsky hierarchies. The experiments are conducted for the method and several baselines, some of which include known CL strategies. Overall the results show that CLeAR addresses the CL in AR, providing also positive Backward transfer.\n\nSTRENGTHS:\nThe authors address for the first time the CL adaptation of AR. The setup proposed differentiates from standard CL, where input data dimensions are fixed, and evaluation is typically done in an IID test-set w.r.t. to the joint task distribution. This leads to a more general setting w.r.t. standard CL which can improve current research and lead to new methodologies. \n\nThe proposed method well adapts to variegated AR tasks, being designed to use a shared feature space where learning takes place. The experiments show that this choice has the desired effects, addressing catastrophic forgetting in several scenarios. In addition, the newly designed benchmarks (to be released on github) offer space for further research in this new field. \n\nThe presentation is overall clear, with much focus on benchmarking the method in several experiments.\n\nWEAKNESSES:\nIt seems some information is missing in the current text, and the experimental evaluation can be improved. Referring to Table 3, the discussion in Section 4.2 only mentions the first 4 rows, but nothing is mentioned about the last three. How should those results be interpreted and what do they convey? It seems that the method is not solving the task, accordingly to the authors' claim that the threshold should be above 90% of the task's accuracy.\n\nAs a suggestion, the comparison with existing CL strategies is very important and should appear or at least be discussed in the main text.\n\nAll experiments are conducted in a single run, contrary to common practice, especially in CL. This may lead to a confirmation bias over the proposed method and should be clarified at least here why the choice of single runs does convey all relevant information. Based on the experiments in Appendix D, I'm not entirely sure that LwF combined with the shared feature mapping would drastically differentiate from CLeAR (Tables 4 and 5), since it is showing very similar classification scores. To this end, I sense that adopting LwF is particularly suited for the task the authors are proposing since it is likely adapting previous tasks to new observed sequences.  \n\n**Typo** In the bibliography refs [33] and [34] coincide.\n\nQUESTIONS:\nCan you provide more details about why LwF is performing very well in CL-AR? Is it because of the pseudo-labels in the initial training phase?\n\nLIMITATIONS:\n**Major** The related work connected to AR is poorly discussed and only one recent work is mentioned in the related work. This gives somehow the feeling that research in AR is niche. To this end, more works should be cited or, otherwise, the motivation for CL in AR should be strengthened. \n\n**Minor** Details on the benchmarks are not entirely clear, making a bit fuzzy the decision behind their design. Such explanation can be improved even in the Supplementary to make understand how different tasks combine in increasing level of difficulty", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper looks at the continual learning (CL) setting involving tasks which require more abstract reasoning (reusable across input domains), s.a. addition and multiplication. The authors imagine that the inputs of all encountered tasks can be mapped into a common space, which, in turn, can be similarly transformed across tasks, in order to achieve transfer on a higher level. To take advantage of this fact, the inputs are processed by three NNs: 1) task-specific m_t which maps the inputs to a common space; 2) f which processes the resulting latent embeddings; 3) h_t which is a task-specific single-layer projection head. m_t is trained using an additional decoder. f and h_t are trained using a similar approach to Learning without Forgetting (LwF). The paper introduces a new set of tasks for evaluating abstract reasoning in a CL setting. The results demonstrate that this approach is capable of backward transfer on the created tasks.\n\nSTRENGTHS:\nThe paper looks towards an interesting direction in CL. I found the discussion on the challenges around using most CL methods for abstract reasoning (AR) to be interesting.\n\nThe paper introduces a novel set of logical reasoning tasks which are categorised according to a class in the Chomsky hierarchy.\n\nThe experiments demonstrate the method’s capability of achieving backward transfer.\n\nThe use of the auto-encoder objective to learn a common mapping between tasks appears novel.\n\nWEAKNESSES:\nThe method claims to introduce the first CL scenario for abstract reasoning, but simpler AR tasks have been explored in NeuralTerpret, HOUDINI, DeepProbLog.\n\nThe description of the algorithm seems rushed. The approach is presented but not well motivated, e.g. the properties of the mapping m_t on lines 218-222. The continual learning approach’s novelty seems limited, being mainly modelled after LwF.\n\nMy impression is that the method is not evaluated on a difficult enough setting in order to determine whether it is promising. Each input consists of one-hot encodings (as opposed to images) which I imagine are easier to learn how to map to a common space.\n\nQUESTIONS:\nWhat do you think would happen if two tasks have partially overlapping input alphabets, e.g. (task1: 1 2 3 4 , task2:  3 4 5)? Would this idea of mapping both input distributions to the same space be problematic?\n\nWhat would you say is learned in f_theta across disparate tasks, which allows for transfer between them?\n\nLIMITATIONS:\nThe fixed-size model limits the effective number of tasks which can be solved.\n\nIt appears that the approach relies on some level of similarity between tasks, as it uses the new inputs to create pseudo-labels.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}