Metalearning to Continually Learn In Context

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 18:09:51

Abstract: General-purpose learning systems should improve themselves in open-ended fashion in ever-changing environments. Conventional learning algorithms for neural networks, however, suffer from catastrophic forgetting (CF)---previously acquired skills are forgotten when a new task is learned. Instead of hand-crafting new algorithms for avoiding CF, we propose Automated Continual Learning (ACL) to train self-referential neural networks to meta-learn their own in-context continual (meta-)learning algorithms.  ACL encodes continual learning desiderata---good performance on both old and new tasks---into its meta-learning objectives. Our experiments demonstrate that, in general, in-context learning algorithms also suffer from CF but  ACL effectively solves such "in-context catastrophic forgetting". Our ACL-learned algorithms outperform hand-crafted ones and existing meta-continual learning methods on the Split-MNIST benchmark in the replay-free setting,  and enables continual learning of diverse tasks consisting of multiple few-shot and standard image classification datasets. Going beyond, we also highlight the limitations of in-context continual learning, by investigating the possibilities to extend ACL to the realm of state-of-the-art CL methods which leverage pre-trained models.

Main paper: 
 introduction: 
 Enemies of memories are other memories [1] . Continually-learning artificial neural networks (NNs) are memory systems in which their weights store memories of task-solving skills or programs, and their learning algorithm is responsible for memory read/write operations. Conventional learning algorithms-used to train NNs in the standard scenarios where all training data is available at onceare known to be inadequate for continual learning (CL) of multiple tasks where data for each task is available sequentially and exclusively, one at a time. They suffer from "catastrophic forgetting" (CF; [2] [3] [4] [5] ); the NNs forget, or rather, the learning algorithm erases, previously acquired skills, in exchange of learning to solve a new task. Naturally, a certain degree of forgetting is unavoidable when the memory capacity is limited, and the amount of things to remember exceeds such an upper bound. In general, however, capacity is not the fundamental cause of CF; typically, the same NNs, suffering from CF when trained on two tasks sequentially, can perform well on both tasks when they are jointly trained on the two tasks at once instead (see, e.g., [6] ).The real root of CF lies in the learning algorithm as a memory mechanism. A "good" CL algorithm should preserve previously acquired knowledge while also leveraging previous learning experiences to improve future learning, by maximally exploiting the limited memory space of model parameters.All of this is the decision-making problem of learning algorithms. In fact, we can not blame the conventional learning algorithms for causing CF, since they are not aware of such a problem. They are designed to train NNs for a given task at hand; they treat each learning experience independently (they are stationary up to certain momentum parameters in certain optimizers), and ignore any potential influence of current learning on past or future learning experiences. Effectively, more sophisticated algorithms previously proposed against CF [7, 8] , such as elastic weight consolidation [9, 10] or synaptic intelligence [11] , often introduce manually-designed constraints as regularization terms to explicitly penalize current learning for deteriorating knowledge acquired in past learning.Here, instead of hand-crafting learning algorithms for continual learning, we train self-referential neural networks [12, 13] to meta-learn their own "in-context" continual learning algorithms. We train them through gradient descent on learning objectives that reflect desiderata for continual learning algorithms-good performance on both old and new tasks, including forward and backward transfer. In fact, by extending the standard settings of few-shot or meta-learning based on sequenceprocessing NNs [14] [15] [16] [17] [18] , the continual learning problem can also be formulated as a long-span sequence processing task [19] . Corresponding CL sequences can be obtained by concatenating multiple few-shot/meta-learning sub-sequences, where each sub-sequence consists of input/target examples corresponding to the task to be learned in-context. As we'll see in Sec. 3, this setting also allows us to seamlessly express classic desiderata for CL as part of objective functions of the meta-learner.Once formulated as such a sequence-learning task, we let gradient descent search for CL algorithms achieving the desired CL behaviors in the program space of NN weights. In principle, all typical challenges of CL-such as the stability-plasticity dilemma [20] -are automatically discovered and handled by the gradient-based program search process. Once trained, CL is automated through recursive self-modification dynamics of the trained NN, without requiring any human intervention such as adding extra regularization or setting hyper-parameters for CL. Therefore, we call our method, Automated Continual Learning (ACL).Our experiments focus on supervised image classification, making use of standard few-shot learning datasets for meta-training, namely, Mini-ImageNet [21, 22] , Omniglot [23] , and FC100 [24] , while we also meta-test on other datasets including MNIST [25] , FashionMNIST [26] and CIFAR-10 [27] .Our core contribution is a set of focused experiments showing various facets of in-context CL: (1) We first reveal the "in-context catastrophic forgetting" problem using two-task settings (Sec. 4.1) and analyse its emergence (Sec. 4.2). We are not aware of any prior work discussing this problem. (2) We show very promising results of our ACL-trained learning algorithm on the classic Split-MNIST [6, 28] benchmark, outperforming hand-crafted learning algorithms and prior meta-continual learning methods [29] [30] [31] . (3) We experimentally illustrate the limitations of ACL on 5-datasets [32] and Split-CIFAR100 by comparing to more recent prompt-based state-of-the-art CL methods [33, 34] . 
continual learning: 
 The main focus of this work is on continual learning [35, 36] in supervised learning settings even though high-level principles we discuss here also transfer to reinforcement learning settings [37] .In addition, we focus on the realm of CL methods that keep model sizes constant (unlike certain CL methods that incrementally add more parameters as more tasks are presented; see, e.g., [38] ), and do not make use of any external replay memory (used in other CL methods; see, e.g., [39] [40] [41] [42] [43] ).Classic desiderata for a CL system (see, e.g., [44, 45] ) are typically summarized as good performance on three metrics: classification accuracies on each dataset (their average), backward transfer (i.e., impact of learning a new task on the model's performance on previous tasks; e.g., catastrophic forgetting is a negative backward transfer), and forward transfer (impact of learning a task for the model's performance on a future task). From a broader perspective of meta-learning systems, we may also measure other effects such as learning acceleration (i.e., whether the system leverages previous learning experiences to accelerate future learning); here our primary focus remains the classic CL metrics above.In Sec. 3, we'll formulate continual learning as a long-span sequence processing task. This is a direct extension of the classic few-shot/meta learning formulated as a sequence learning problem. In fact, since the seminal works [14] [15] [16] [17] (see also [46] ), many sequence processing neural networks (see, e.g., [47] [48] [49] [50] [51] [52] [53] [54] [55] [56] [57] [58] including Transformers [59, 18] ) have been trained as a meta-learner [13, 12] that learn by observing sequences of training examples (i.e., pairs of inputs and their labels) in-context. Here we briefly review such a formulation. Let d, N , K, P be positive integers. In sequential N -way K-shot classification settings, a sequence processing NN with a parameter vector θ ∈ R P observes a pair (x t , y t ) where x t ∈ R d is the input and y t ∈ {1, ..., N } is its label at each step is fed to the model without its true label but with an "unknown label" token ∅ (number of input labels accepted by the model is thus N + 1). The model is trained to predict its true label, i.e., the parameters of the model θ are optimized to maximize the probability p(y|(x 1 , y 1 ), ..., (x N •K , y N •K ), (x, ∅); θ) of the correct label y ∈ {1, ..., N } of the input query x. Since class-to-label associations are randomized and unique to each sequence ((x 1 , y 1 ), ..., (x N •K , y N •K ), (x, ∅)), each such a sequence represents a new (few-shot or meta) learning example to train the model. To be more specific, this is the synchronous label setting of Mishra et al. [18] where the learning phase (observing examples, (x 1 , y 1 ) etc.) is separated from the prediction phase (predicting label y given (x, ∅)). We opt for this variant in our experiments as we empirically find this (at least in our specific settings) more stable than the delayed label setting [14] where the model has to make a prediction for every input, and the label is fed to the model with a delay of one time step.Our method (Sec. 3) can be applied to any sequence-processing NN architectures in principle.Nevertheless, certain architectures naturally fit better to parameterize a self-improving continual learner. Here we use the modern self-referential weight matrix (SRWM; [19, 60] ) to build a generic self-modifying NN. An SRWM is a weight matrix that sequentially modifies itself as a response to a stream of input observations [12, 61] . The modern SRWM belongs to the family of linear Transformers (LTs) a.k.a. Fast Weight Programmers (FWPs; [62] [63] [64] [65] [66] [67] [68] ). Linear Transformers and FWPs are an important class of the now popular Transformers [59] : unlike the standard ones whose computational requirements grow quadratically and whose state size grows linearly with the context length, LTs/FWPs' complexity is linear and the state size is constant w.r.t. sequence length (like in the standard RNNs). This is an important property for in-context CL, since, conceptually, we want such a CL system to continue to learn for an arbitrarily long, lifelong time span. Moreover, the duality between linear attention and FWPs [67] -and likewise, between linear attention and gradient descent-trained linear layers [69, 70] -have played a key role in certain theoretical analyses of in-context learning capabilities of Transformers [71, 72] .The dynamics of an SRWM [19] are described as follows. Let d in , d out , t be positive integers, and ⊗ denote outer product. At each time step t, an SRWM W t-1 ∈ R (dout+2 * din+1)×din observes an input x t ∈ R din , and outputs y t ∈ R dout , while also updating itself to W t as:EQUATIONwhere v t , vt ∈ R (dout+2 * din+1) are value vectors, q t ∈ R din and k t ∈ R din are query and key vectors, and σ(β t ) ∈ R is the learning rate. σ and ϕ denote sigmoid and softmax functions respectively. ϕ is typically also applied to x t in Eq. 1; here we follow Irie et al. [19] 's few-shot image classification setting, and use the variant without it. Eq. 3 corresponds to a rank-one update of the SRWM, from W t-1 to W t , through the delta learning rule [73, 67] where the self-generated patterns, v t , ϕ(k t ), and σ(β t ), play the role of target, input, and learning rate of the learning rule respectively. The delta rule is crucial for the performance of LTs [67, 68, 74, 75] .The initial weight matrix W 0 is the only trainable parameters of this layer, that encodes the initial self-modification algorithm. We use the layer above as a direct replacement to the self-attention layer in the Transformer architecture [59] ; and use the multi-head version of the computation above [19] . (which is extensible to mini-batches with multiple such sequences), where θ denotes the model parameters (for the SRWM layer, it is the initial weights W 0 ). Figure 1 illustrates the overall meta-training process of ACL. Overall Model Architecture. As we mention in Sec. 2, in our NN architecture, the core sequential dynamics of CL are learned by the self-referential layers. However, as an image-processing NN, our model makes use of a vision backend. We use the "Conv-4" architecture [21] (typically used in the context of few-shot learning) in all our experiments, except in the last one where we use a pre-trained vision Transformer [76] . Overall, the model takes an image as input, process it through a feedforward vision NN, whose output is fed to the SRWM-layer block. Note that this is one of the limitations of this work: more general ACL should also learn to modify the vision components. 2Another crucial architectural choice that is specific to continual/multi-task image processing is normalization layers (see also Bronskill et al. [78] ). Typical NNs used in few-shot learning (e.g., Vinyals et al. [21] ) contain batch normalization (BN; [79] ) layers. All our models use instance normalization (IN; [80] ) instead of BN because in our preliminary experiments, we expectably found IN to generalize much better than BN layers in the CL setting.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss limitations of our method in Sec. 4 and 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
theory assumptions and proofs: 
 Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
two-task setting: comprehensible study: 
 We first reveal the problem of "in-context catastrophic forgetting" and show how our ACL method (Sec. 3) can overcome it. As a minimum setting for this, we focus on the two-task "domain- 1 and 2 ). Here Task A is Omniglot and Task B is Mini-ImageNet. We observe that, in both cases, without explicit minimization, backward transfer capability (purple and brown curves) of the learned learning algorithm gradually degrades as it learns to learn a new task (all other colors), causing in-context catastrophic forgetting. Note that blue/orange and green/red curve pairs almost overlap; indicating that when a task is learned, the model can learn it whether it is in the first or second segment of the continual learning sequence.incremental" CL setting (see Appendix A.1). We consider two meta-training task combinations:Omniglot [23] and Mini-ImageNet [21, 22] or FC100 [24] (which is based on CIFAR100 [27] ) and Mini-ImageNet. The order of appearance of two tasks within meta-training sequences is alternated for every batch. Appendix A.2 provides further details. We compare systems trained with or without the backward transfer term in the ACL loss (the last term in Eq. 4).Unless otherwise indicated (e.g, later for classic Split-MNIST; Sec. 4.3), all tasks are configured to be a 5-way classification task. This is one of the classic configurations for few-shot learning tasks, and also allows us to evaluate the principle of ACL with reasonable computational costs (like any sequence learning-based meta-learning methods, scaling this to many more classes is challenging; we also discuss this in Sec. 5). For standard datasets such as MNIST, we split the dataset into sub-datasets of disjoint classes [81] : for example for MNIST which is originally a 10-way classification task, we split it into two 5-way tasks, one consisting of images of class '0' to '4' ('MNIST-04'), and another one made of class '5' to '9' images ('MNIST-59'). When we refer to a dataset without specifying the class range, we refer to the first sub-set. Unless stated otherwise, we concatenate 15 examples from each class for each task in the context for both meta-training and meta-testing (resulting in sequences of length 75 for each task). All images are resized to 32 × 32-size 3-channel images, and normalized according to the original dataset statistics. We refer to Appendix A for further details.Table 1 shows the results when the models are meta-tested on the test sets of the corresponding few-shot learning datasets used for meta-training. We observe that for both pairs of meta-training tasks, the models without the ACL loss catastrophically forget the first task after learning the second one: the accuracy on the first task is at the chance level of about 20% for 5-way classification after learning the second task in-context (see rows with "ACL No"). The ACL loss clearly addresses this problem: the ACL-learned CL algorithms preserve the performance of the first task. This effect is particularly pronounced in the Omniglot/Mini-ImageNet case (involving two very different domains).Table 2 shows evaluations of the same models but using two standard datasets, 5-way MNIST and CIFAR-10, for meta-testing. Again, ACL-trained models better preserve the memory of the first task after learning the second one. In the Omniglot/Mini-ImageNet case, we even observe certain positive backward tranfer effects: in particular, in the "MNIST-then-CIFAR10" continual learning case, the performance on MNIST noticeably improves after learning CIFAR10 (possibly leveraging 'more data' provided in-context).Now we closely look at the emergence of "in-context catastrophic forgetting" during meta-training for the baseline models trained without the backward transfer term (the last/third term in Eq. 4) in the ACL objective loss (corresponding to the ACL/No cases in Tables 1 and 2 ). We focus on the Omniglot/Mini-ImageNet case, but similar trends can also be observed in the FC100/Mini-ImageNet Figure 2a shows the case where two tasks are learned about at the same time. We observe that when the learning curves go down, the ACL losses go up, indicating that more the model learns, more it tends to forget the task in-context learned previously. We also find this same trend when one task is learned before the other one as is the case in Figure 2b . Here Task A alone is learned first; while Task B is not learned, both learning and ACL curves go down for Task A (essentially, as the model does not learn the second task, there is no force that encourages forgetting). After around 3000 steps, the model also starts learning Task B. From this point, the ACL loss for Task A also starts to go up, indicating again an opposing force effect between learning a new task and remembering a past task. These observations clearly indicate that, without explicitly taking into account the backward transfer loss as part of learning objectives, our gradient descent search tends to find solutions/CL algorithms that prefer to erase previously learned knowledge (this is rather intuitive; it seems easier to find such algorithms that ignore any influence of the current learning to past learning than those that also preserve prior knowledge). In all cases, we find our ACL objective to be crucial for the learned CL algorithms to be capable of remembering the old task while also learning the new one.Evaluation on Standard Split-MNIST. Here we evaluate ACL on the standard Split-MNIST task in domain-incremental and class-incremental settings [6, 28] , and compare its performance to existing CL and meta-CL algorithms (see Appendix A.7 for full references of these methods). Our comparison focuses on methods that do not require replay memory. We are not aware of any existing hand-crafted CL algorithms that can achieve ACL's performance without any replay memory. We refer to Appendix A.7/B for further discussions and ablation studies.Evaluation on diverse task domains. Using the setting of Sec. 4.1, we also evaluate our ACL-trained models for CL involving more tasks/domains; using meta-test sequences made of MNIST, CIFAR-10, and Fashion MNIST. We also evaluate the impact of the number of tasks in the ACL objective: in addition to the model meta-trained on Omniglot/Mini-ImageNet (Sec. 4.1), we also meta-train a model (with the same architecture and hyper-parameters) using 3 tasks, Omniglot, Mini-ImageNet, and FC100, using the 3-task ACL objective (see Appendix A.5); which is meta-trained not only on longer CL sequences but also on more data. The full results of this experiment can be found in Appendix B. 4 . We find that the two ACL-trained models are indeed capable of retaining the knowledge without catastrophic forgetting for multiple tasks during meta-testing, while the performance on prior tasks gradually degrades as the model learns new tasks, and performance on new tasks becomes moderate (see also Sec. 5 on limitations). The 3-task version outperforms the 2-task one overall, encouragingly indicating a potential for further improvements even with a fixed parameter count.Going beyond: limitations and outlook. The experiments presented above effectively demonstrate the possibility to encode a continual learning algorithm into self-referential weight matrices, that outperforms handcrafted learning algorithms and existing metalearning approaches for CL. While we consider this as an important result for metalearning and in-context learning in general, we note that current state-of-the-art CL methods use neither regularization-based CL algorithms nor metacontinual learning methods we mention above, but the so-called learning to prompt (L2P)-family of methods [33, 34] that leverage pre-trained models, namely a vision Transformer (ViT) pre-trained on ImageNet [76] . A natural question we should ask is whether we could foresee ACL beyond the scope considered so far, and evaluate it in such a setting. To study this, we take a pre-trained (frozen) vision model, and add self-referential layers (to be meta-trained from scratch) on top of it to build a continual learner. This allows us to highlight an important challenge of in-context CL in what follows.We use two tasks from the L2P works above [33, 34] : 5-datasets [32] and Split-CIFAR-100, in the class-incremental setting, but we focus on a "mini" versions thereof: we only use the two first classes within each task (i.e., 2-way version) and for Split-CIFAR100, we only use the 5 first tasks; as we'll see, this setting is enough to illustrate an important limitation of in-context CL. Again following L2P [33, 34] , we use ViT-B/16 [76] (available via PyTorch) as the pre-trained vision model, which we keep frozen. We use the same configuration for the self-referential component from the Split-MNIST experiment. We meta-train the resulting model using Mini-ImageNet and Omniglot with the 5-task ACL loss. Table 4 shows the results. Even in this simple "mini" version of the tasks, ACL's performance is far behind that of L2P methods. Notably, the frozen ImageNet-pre-trained features with the meta-learner trained on Mini-ImageNet and Omniglot are not enough to perform well on the 5-th task of Split-CIFAR100, and SVHN and notMNIST of 5-datasets. This shows the necessity to meta-train on more diverse tasks for in-context CL to be possibly successful in more general settings.  
discussion: 
 Other Limitations. In addition to the limitations already mentioned above, here we discuss others.First of all, as an in-context/learned learning algorithm, there are challenges in terms of both domain and length generalization (we qualitatively observe these to some extent in Sec. 4; further discussion and experimental results are presented in Appendix B.3 & B.5). Regarding the length generalization, we note that unlike the standard "quadratic" Transformers, linear Transformers/FWPs-like SRWMs can be trained by carrying over states across two consecutive batches for arbitrarily long sequences.Such an approach has been successfully applied to language modeling with FWPs [67] . This possibility, however, has not been investigated here, and is left for future work. Also, directly scaling ACL for real-world tasks requiring many more classes does not seem straightforward: it would require very long training sequences. That said, it may be possible that ACL could be achieved without exactly following the process we propose; as we discuss below for the case of LLMs, certain real-world data may naturally give rise to an ACL-like objective. This work is also limited to the task of image classification, which can be solved by feedforward NNs. Future work may investigate the possibility to extend ACL to continual learning of sequence learning tasks, such as continually learning new languages. Finally, ACL learns CL algorithms that are specific to the pre-specified model architecture; more general meta-learning algorithms may aim at achieving learning algorithms that are applicable to any model, as is the case for many classic learning algorithms.Related work. There are several recent works that are catagorized as 'meta-continual learning' or 'continual meta-learning' (see, e.g., [29, 30, [82] [83] [84] 51] ). For example, Javed and White [29] , Beaulieu et al. [30] use "model-agnostic meta-learning" (MAML; [85, 86] ) to meta-learn representations for CL while still making use of classic learning algorithms for CL; this requires tuning of the learning rate and number of iterations for optimal performance during CL at meta-test time (see, e.g., Appendix A.7). In contrast, our approach learn learning algorithms in the spirit of Hochreiter et al. [14] , Younger et al. [15] ; this may be categorized as 'in-context continual learning.' Several recent works (see, e.g., [87, 88] ) mention the possibility of such in-context CL but existing works [19, 89, 90] that learn multiple tasks sequentially in-context do not focus on catastrophic forgetting which is one of the central challenges of CL. Here we show that in-context learning also suffers from catastrophic forgetting in general (Sec. 4.1-4.2) and propose ACL to address this problem. We also note that the use of SRWM is relevant to 'continual meta-learning' since with a regular sequence processor with slow weights, there remains the question of how to continually learn the slow weights (meta-parameters). In principle, recursive self-modification as in SRWM is an answer to this question as it collapses such meta-levels into single self-reference [12] . We also refer to [91] [92] [93] for other prior work on meta-continual learning.Artificial v. Natural ACL in Large Language Models? Recently, "on-the-fly" few-shot/meta learning capability of sequence processing NNs has attracted broader interests in the context of large language models (LLMs; [94] ). In fact, the task of language modeling itself has a form of sequence processing with error feedback (essential for meta-learning [95] ): the correct label to be predicted is fed to the model with a delay of one time step in an auto-regressive manner. Trained on a large amount of text covering a wide variety of credit assignment paths, LLMs exhibit certain sequential few-shot learning capabilities in practice [96] . This was rebranded as in-context learning, and has been the subject of numerous recent studies (e.g., [97-103, 71, 72] ). Here we explicitly/artificially construct ACL meta-training sequences and objectives, but in modern LLMs trained on a large amount of data mixing a large diversity of dependencies using a large backpropagation span, it is conceivable that some ACL-like objectives may naturally appear in the data. 
conclusion: 
 Our Automated Continual Learning (ACL) trains sequence-processing self-referential neural networks (SRNNs) to learn their own in-context continual (meta-)learning algorithms. ACL encodes classic desiderata for continual learning (e.g., forward and backward transfer) into the objective function of the meta-learner. ACL uses gradient descent to deal with classic challenges of CL, to automatically discover CL algorithms with good behavior. Once trained, our SRNNs autonomously run their own CL algorithms without requiring any human intervention. Our experiments reveal the original problem of in-context catastrophic forgetting, and demonstrate the effectiveness of the proposed approach to combat it. We demonstrate very promising results on the classic Split-MNIST benchmark where existing hand-crafted algorithms fail, while also discussing its limitations in more general scenarios. We believe this comprehensive study to be an important step for in-context CL research.[ the meta-training phase, we modify the trainable parameters of the meta-learner through gradient descent minimizing the meta-learning loss function (using backpropagation through time). During meta-testing, no human-designed optimization for weight modification is used anymore; the SRWMs modify their own weights following their own learning rules defined as their forward pass (Eqs. 1-3).In connection with the now-popular in-context learning [96] , we also refer to a (meta-training or meta-test) training-example sequence as context. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We provide compute resource related information in Appendix A.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [NA] Justification: We do not have anything to report.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: Our work does not have any such impacts.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our work does not imply any such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Our codebase includes certain publicly available code. The corresponding license files are included in the supplemental material. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: For classic image classification datasets such as MNIST [25] , CIFAR10 [27] , and FashionMNIST (FMNIST; Xiao et al. [26] ) we refer to the original references for details.For Omniglot [23] The "5-datasets" dataset [32] consists of 5 datasets: CIFAR10, MNIST, FashionMNST, SVNH [105] ,and notMNIST [106] .Split-CIFAR100 is also based on CIFAR100. The standard setting splits CIFAR100 into 10 10-way classification tasks.Meta-train/test sequence construction procedure. We use torchmeta [107] which provides common few-shot/meta learning settings for these datasets to sample and construct their metatrain/test datasets. The construction of "meta-training training" sequences for an N -way classification, using a dataset containing C classes works as follows; for each sequence, we sample N random but distinct classes out of C (N < C). The resulting classes are re-labelled such that each class is assigned to one out of N distinct random label index which is unique to the sequence. For each of these N classes, we sample K examples. We randomly order these N * K examples to obtain a sequence. Each such a sequence "simulates" an unknown task the model has to learn.We use the same model and training hyper-parameters in all our experiments. All hyper-parameters are summarized in Table 5 . We use the Adam optimizer with the standard Transformer learning rate warmup scheduling [59] . The vision backend is the classic 4-layer convolutional NN of Vinyals et al. [21] . Most configurations follow those of Irie et al. [19] ; except that we initialize the 'query' sub-matrix in the self-referential weight matrix using a normal distribution with a mean value of 0 and standard deviation of 0.01/ √ d head while other sub-matrices use an std of 1/ √ d head (motivated by the fact that a generated query vector is immediately multiplied with the same SRWM to produce a value vector). For any further details, we'll refer the readers to our public code we'll release upon acceptance. We conduct our experiments using a single V100-32GB, 2080-12GB or P100-16GBGPUs, and the longest single training run takes about one day. For evaluation on few-shot learning datasets (i.e., Omniglot, Mini-Imagenet and FC100), we use 5 different sets consisting of 32 K random test episodes each, and report mean and standard deviation.For evaluation on standard datasets, we use 5 different random support sets for in-context learning, and evaluate on the entire test set. We report the corresponding mean and standard deviation across these 5 evaluation runs.For the Split-MNIST experiment, we do 10 meta-testing runs to compute the mean and standard deviation as the baseline models are also trained for 10 runs in Hsu et al. [6] (see other details in Table 6 : Impact of the choice of meta-validation datasets. Classification accuracies (%) on three datasets: Split-CIFAR-10, Split-Fashion MNIST (Split-FMNIST), and Split-MNIST in the domainincremental setting (we omit "Split-" in the second column). "OOB" denotes "out-of-the-box". "mImageNet" here refers to mini-ImageNet. Ideally, we want these models to 'systematically generalize' to more tasks even when they are trained with only a handful of them [108] . This is an interesting research question on generalization to be studied in a future work.In practice, instead of training the models only for "15-shot learning," we also add an auxiliary loss for 1-shot learning. This naturally encourages the models to learn in-context from the first examples.Here we provide details of the Split-MNIST experiments presented in Sec. 4 and Table 3 . The baseline methods presented in Table 3 include: standard SGD and Adam optimizers, Adam with the L2 regularization, elastic weight consolidation [9] and its online variant [10] , synaptic intelligence [11] , memory aware synapses [109] , learning without forgetting (LwF; Li and Hoiem [110] ). For these methods, we directly take the numbers reported in Hsu et al. [6] for the 5-task domain/class-incremental settings.For the 2-task class incremental setting, we use Hsu et al. [6] 's code to train the correspond models (the number for LwF is currently missing as it is not implemented in their code base; we plan to add the corresponding/missing entry in Table 3 for the final version of this paper).Finally we also evaluate two meta-CL baselines: Online-aware Meta-Learning (OML; Javed and White [29] ) and Generative Meta-Continual Learning (GeMCL; Banayeeanzade et al. [31] ). OML is a MAML-based meta-learning approach. We note that as reported by Javed and White [29] in their public code repository; after some critical bug fix, the performance of their OML matches that of Beaulieu et al. [30] (which is a direct application of OML to another model architecture). Therefore, we focus on OML as our main MAML-based baseline. We take the out-of-the-box model (metatrained for Omniglot, with a 1000-way output) made publicly available by Javed and White [29] . We evaluate the corresponding model in two ways. In the first, 'out-of-the-box' case, we take the metapre-trained model and only tune its meta-testing learning rate (which is done by Javed and White [29] even for meta-testing in Omniglot). We find that this setting does not perform very well; in the other case ('optimized # meta-testing iterations'), we additionally tune the number of meta-test training iterations. We've done a grid search of the meta-test learning rate in 3 * {1e -2 , 1e -3 , 1e -4 , 1e -5 } and the number of meta-test training steps in {1, 2, 5, 8, 10} using a meta-validation set based on an MNIST validation set (5 K held-out images from the training set); we found the learning rate of 3e -4 and 8 steps to consistently perform the best in all our settings. We've also tried it 'with' and 'without' the standard mean/std normalization of the MNIST dataset; better performance was achieved without such normalization (which is in fact consistent as they do not normalize the Omniglot dataset for their meta-training/testing). Their performance on the 5-task class-incremental setting is somewhat surprising/disappointing (since genenralization from Omniglot to MNIST is typically straightforward, at least, in common non-continual few-shot learning settings; see, e.g., Munkhdalai and Yu [51] ). At the same time, to the best of our knowledge, OML-trained models have not been tested in such a condition in prior work; from what we observe, the publicly available out-of-the-box model might be overtuned for Omniglot/Mini-ImageNet or the frozen 'representation network' is not ideal for genenralization. We note that the sensitivity of these MAML-based methods [29, 30] w.r.t. meta-test hyper-parameters has been also noted by Banayeeanzade et al. [31] ; these are characteristics of hand-crafted learning algorithms that we want to avoid with learned learning algorithms.We use code and a pre-trained model (trained on Omniglot) made public by Banayeeanzade et al. [31] for the GeMCL baseline (see also Table 7 ); like our method, GeMCL also do not require any special tuning at test-time. 7 ). To obtain a sequence of 5 tasks, we simply sample 5 tasks from Omniglot (in principle, we should make sure that different tasks in the same sequence have no class overlap; in practice, our current implementation simply randomly draws 5 independent tasks from Omniglot).A.8 Details of the Split-CIFAR100 and 5-datasets experiment using ViTAs we described in Sec. 4, for the experiments on Split-CIFAR100 and 5-datasets, following Wang et al. [33, 34] , we use ViT-B/16 pre-trained on ImageNet [76] which is available through torchvision [111] . In this experiments, we resize all images to 3x224x224 and feed them to the ViT. We remove the output layer of the ViT, and use its 768-dimensional feature from the penultimate layer as the image encoding. The self-referential component which is added to this encoder has the same architecture (2 layers, 16 heads) as the rest of the paper (see all hyper-parameters in Table 5 )All ViT parameters are frozen during meta-training.Here we conduct ablation studies on the choice of meta-validation sets to select model checkpoints. In general, when dealing with out-of-domain generalization, the choice of validation procedures to select final model checkpoints plays a crucial role in the evaluation of the corresponding method [112, 113] .The out-of-the-box models are chosen based on the average meta-validation performance on the validation set corresponding to the few-shot learning datasets used in meta-training: Omniglot and Table 8 : Meta-testing on sequences that are longer than those from meta-training. Classification accuracies (%) on 5-task Split-FMNIST and 5-task Split-MNIST in the domain-incremental settings. The model is the one finetuned with 5-task ACL loss using Omniglot as the meta-finetuning set and FMNIST as the meta-validation set (i.e., the numbers in the top part of the table are taken from Table 6 ). In the first column, "Split-FMNIST, Split-MNIST" indicates continual learning of 5 Split-FMNIST tasks followed by 5 tasks of Split-MNIST (and "Split-MNIST, Split-FMNIST" is the opposite order). Performance is measured at the end of the entire sequence. mini-ImageNet (or Omniglot, mini-ImageNet, and FC100 in the case of 3-task ACL), independently of any potential meta-test datasets. In contrast, in the meta-finetuning process of Table 3 , we selected our model checkpoint by meta-validation on the MNIST validation dataset (we held out 5 K images from the training set). Here we evaluate ACL models meta-finetuned for the "5-task domain-incremental binary classification" on three Split-'X' tasks where 'X' is MNIST, FashionMNIST (FMNIST) or CIFAR-10 for various choices of meta-validation sets (in each case we hold out 5 K images from the corresponding training set). In addition, we also evaluate the effect of meta-finetuning datasets (Omniglot only v. Omniglot and mini-ImageNet). Table 6 shows the results (we use 15 meta-training and meta-testing examples except for the Omniglot-finedtuned/MNIST-validated model from Table 3 which happens to be configured with 5 examples; this will be fixed in the final version). Effectively, meta-validation on the matching validation set is useful. Also, meta-finetuning only on Omniglot is beneficial for the performance on MNIST when meta-validated on MNIST or FMNIST. However, importantly, we emphasize that our ultimate goal is not to obtain a model that is specifically tuned for certain datasets; we aim at building models that generally work well across a wide range of tasks (ideally on any tasks); in fact, several existing works in the few-shot learning literature evaluate their methods in such settings (see, e.g., Requeima et al. [114] , Bronskill et al. [78] , Triantafillou et al. [115] ). This also goes hand-in-hand with scaling up ACL (our current model is tiny; see hyper-parameters in Table 5 ; the vision component is also a shallow 'Conv-4' net) and various other considerations on self-improving continual learners (see, e.g., Schmidhuber [116] ), such as automated curriculum learning [117] .Here we report the performance of the models used in the Split-MNIST experiment (Sec. 4.3) on "in-domain" 5-task 2-way Split-Omniglot. Table 9 shows the result. Performance is very similar between our ACL and the baseline GeMCL on this task in the class incremental setting, unlike on Split-MNIST (Table 3 ) where we observe a larger performance gap between these same models. Here we also include the "domain incremental" setting for the sake of completeness but note that GeMCL is not originally trained for this setting. Here we provide further discussion and experimental results on the limitations of our approach as a learned algorithm.As a data-driven learned algorithm, the domain generalization capability is a typical limitation as it depends on the meta-trained data. Certain results we presented above are representative of this limitation. In particular, in Table 6 , the model meta-trained/finetuned on Omniglot using Split-MNIST as meta-validation set do not perform well on Split-CIFAR10. While meta-training and meta-validating on a larger/diverse set of datasets may be an immediate remedy to obtain more robust ACL models, we note that since ACL is also a "continual meta-learning" algorithm (Sec. 5), an ideal ACL model should also continually incorporate and learn from more data during potentially lifelong meta-testing; we leave such an investigation for future work.Length generalization. We already qualitatively observed the limited length generalization capability in Table 10 (meta-trained with up to 3 tasks and meta-tested with up to 4 tasks). Here we provide one more experiment evaluating ACL models meta-trained for 5 tasks on a concatenation of two 5-task Split-MNIST and Split-FMNIST tasks (resulting in 10 tasks). Table 8 shows the results. Again, while the model does not completely break, increasing the number of tasks to 10 rapidly degrades the performance compared to the 5-task setting the model is meta-trained for. Similarly, its performance on the Split-Omniglot domain incremental setting (Sec. B.2) degrades with increased numbers of tasks: accuracies for 5, 10 and 20 tasks are 92.3% ± 0.4, 82.0% ± 0.4 and 67.6% ± 1.1 respectively.As noted in Sec. 5, this is a general limitation of sequence processing neural networks, and there is a potential remedy for this limitation (meta-training on more tasks and "context carry-over") which we leave for future work.We also note that in general, "unseen" datasets do not necessarily imply that they are harder tasks than "in-domain" test sets; when meta-trained on Omniglot and mini-ImageNet, meta-generalization on "unseen" MNIST is easier (the accuracy is higher) than on the "in-domain" test set of mini-ImageNet with heldout/unseen classes (compare Tables 1 and 2 ).NeurIPS Paper Checklist Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Here we'll add a link to our public GitHub code repository. Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS2024). Do not distribute.One "straightforward" architecture fitting the bill is an MLP-mixer architecture (Tolstikhin et al.[77]; built of several linear layers), where all linear layers are replaced by the self-referential linear layers of Sec. 2.3. While we implemented such a model, it turned out to be too slow for us to conduct corresponding experiments. Our public code will include a "self-referential MLP-mixer" implementation, but for further experiments, we leave the future work on such an architecture using more efficient CUDA kernels. 


Decision: This paper presents a meta continual learning method to deal with catastrophic forgetting. The method is based on ‘self-referential weight matrices’ (SRWM).
The key innovation is to define the loss function of SRWM training for optimizing both forward (improving performance of subsequent CL tasks) and backward (improving performance of previous CL tasks) transfer while achieving good performance on the current task.
Experiments demonstrate the effectiveness of the proposed method on common image classification meta-learning benchmarks such as Split-MNIST and Mini-ImageNet.

The paper received borderline reviews. While the reviewers acknowledged novel loss function, good number of experiments and good numeral results that demonstrated the proposed method outperforms the existing state of the art, there are shared concerns: the experimental setup is toy setting and is not scalable to a more practical one with a large number of tasks, the training process is expensive, and its performance is not as good as those based on pre-trained transformer models. In addition, it's unclear how to apply the proposed method to other tasks such as language modeling and multimodal tasks besides image classification ones. Overall, the reviewers were not convinced after the rebuttal.
 I think this paper is not ready for publication at the current stage.

Reviews: review: paper_summary: paper_summary: The paper proposes a novel technique to automatically discover in-context continual learning dynamics for image classification task sequences through meta-learning. In order to achieve this purpose, the approach relies on 2 main novelties: 
* Using self referential weight matrices on top of an image encoder - SRWM, as self-modifying that adapts itself to the stream of inputs, is an natural model for continual learning. 
* Encoding continual learning desiderata in the meta-objective, i.e. backward and forward transfer. 

The authors first apply the approach in a classic two-task setting (Split-MNIST) that allows them to showcase and analyse the emergence of in-context catastrophic forgetting phenomena, and to show that using their ACL loss can help reduce it. They further evaluate their method and compare them to replay-free baselines from the CL and meta-CL literature, showing an advantage of their approach in scenarios with up to 3 tasks. 

The authors further test the limits of their approach by comparing it to more recent learning to prompt techniques for continual learning, leveraging the power of pretrained large models. This scenario show a limitation of the technique in more complex scenarios with more tasks, more diverse and complex data.
strength_weakness: strength_weakness: * In my opinion, the main limitation of the approach is its practicality. From the experiments reported in Table 4, it seems that the approach requires to met-train on a sequence of similar length and/or complexity to provide its potential. This is not possible to know in advance in practice. Moreover, one limitation that the authors have not mentioned is that the meta-objective seems to require keeping in memory a number of copies of the model that is equal to the number of tasks. This can quickly become cumbersome for real applications that can require more complex models and very long sequences of tasks.  

* While the authors focus on classic benchmarks for continual and meta-learning, these benchmarks are artificial, relatively simple and lack of diversity. Different works highlight the limits of these benchmarks, I invite the authors to look at "Meta-Album: Multi-domain Meta-Dataset for Few-Shot Image Classification" Ullah et al. 2023, and "NEVIS'22: A Stream of 100 Tasks Sampled from 30 Years of Computer Vision Research" Bornschein et al. 2023 for examples of more realistic benchmarks. 

* It would be interesting to add a discussion of the cost of the approach (computation, memory, ...). Even is it gives a substantial boost in many cases, it would be interesting for practitioners to compare what they gain to what they pay.
questions: questions: * The approach is focused on the task aware scenario, and rooted in a notions of tasks. In many practical scenarios, the distribution shift occurs in a softer way, with no clear notion of task boundaries. Can the authors comment on the possibility of extend their approach to the task-agnostic scenario?
limitations: limitations: The authors provide a detailed discussion of the work limitations, both in the experiments and the discussion sections. Some other limitations are highlighted in the Weaknesses paragraph above.
paper_summary: paper_summary: The paper studies the problem of catastrophic forgetting (CF) by formulating continual learning (CL) as learning from a sequence of demonstrations of tasks. The paper proposes a meta-learning objective function that includes backward transfer terms. These terms compute the error of the predictor on previous tasks after receiving demonstrations of the current task.
strength_weakness: strength_weakness: - The paper is difficult to follow. Many definitions and the algorithm are not very well explained.
    - The motivation of formulating (continual learning) CL as meta-learning is not well presented.
    - Some details of the architecture are mentioned in the background section only (e.g. replacing self-attention with SRWN and the multi-head version.)
    - The details of the training and inference process are not well presented.
- The training process can be very costly and poorly scaled with the number of tasks and the number of examples per task. In each step over a sequence of demonstrations, the method needs to compute and store a new weight matrix in order to perform back-propagation. It might require more memory during training and at inference.
- Even being a meta-learning approach, the model still needs fine-tuning when given a new task to adapt to a new number of tasks.
questions: questions: - Can the authors explain more on the following claim “The stability-plasticity dilemma are automatically discovered and handled by the gradient-based program search process.“ (line 52)?
- What are the advantages of this method compared to previous approaches?
- How do the number of examples per task and order of tasks during the training affect the performance at inference time?
- How does the method scale with the number of tasks in terms of performance and computation?
- It’s unclear how to calculate the loss function in a batch fashion since each training point requires a different sequence of inputs (depending on the position of the task in the sequence) and loss components.
limitations: limitations: There are no negative social impacts. My suggestions have been listed in the previous sections.
paper_summary: paper_summary: The paper describes a method for in-context continual learning (CL) by using a type of meta-learning neural architecture based on ‘self-referential weight matrices’ (SRWM). Proposed in prior work, these models learn to modify weight matrices iteratively as they process more and more inputs. In this work, they are given few-shot examples from different tasks and iteratively update the weight matrices as the examples are processed. This update process is referred to as “in-context” learning in this work. The key innovation is to define the loss function of SRWM training to optimise for both forward (improving performance of subsequent CL tasks) and backward (improving performance of previous CL tasks) transfer while achieving good performance on the current task. Experiments are conducted on commonimage classification meta-learning benchmarks such as Split-MNIST and Mini-ImageNet. Results show the proposed method prevents catastrophic forgetting (without using replay), outperforming existing meta-learning baselines on the evaluated benchmarks.
strength_weakness: strength_weakness: One weakness of the proposed method is that the number of loss function terms increases with the number of CL tasks, as pointed out by the authors in Appendix A.5. This prevents this method from being scaled to more practically relevant settings where a large number (much more than 2 or 3 that this paper has mostly focused the experiments on) of tasks are considered in a CL setting. Method of reducing the loss terms would strengthen the paper.

Another weakness, which is also noted by the authors in Table 4 and Section 4.3, is that the performance of the proposed model and method is poor compared with those based on pre-trained transformer models, even on an easier evaluation task. The authors in Section 5 also discuss a potential connection between LLM transformer training as an implicit version of the proposed model and method. Given these existing strong and more widely adopted methods, it is unclear how much value the proposed method adds. SRWMs are not widely used and LLMs training can scale to a massive number of tasks with a single loss [1] (albeit not CL). A more detailed explanation of the application of the findings of this paper beyond those interested in SRWMs would be helpful.

Another weakness of this paper is its focus on image classification meta-learning tasks only. It is helpful to know the generality of this method, for example on language modelling tasks or multimodal tasks. An experiment demonstrating the method in CL language tasks would be helpful.

[1] Finetuned language models are zero-shot learners. Wei et al. ICLR 2022.
questions: questions: None
limitations: limitations: Limitations have been adequately addressed.
paper_summary: paper_summary: The paper focuses on Automated Continual Learning which is different than handcrafted continual learning. It uses self referential neural networks to meta learn their own in-context continual learning algorithm. First, the paper shows the emergence of in-context catastrophic forgetting. Second, the paper analyze the performance of proposed method (ACL) and finally the paper discuss the limitation of the proposed method.
strength_weakness: strength_weakness: - The paper claims to do in-context continual learning but the concept of in-context learning is not clearly explained.
- The paper mainly focus on two task and five task settings but it would be more helpful to see the more different settings such as three task or four task
- How is the size of SRWM affects the maximum sequence length that can be train?
questions: questions: - Why only consider the two-task setting?
- Why ACL was not compared with replay buffer based methods?
- What is the architecture of SRWM?
limitations: limitations: Authors have addresses the limitation of the work.

Comments: title: Response to Authors' Rebuttal
Author-reviewer discussion ending imminently
Thank you
Answer to rebuttal
Thank you for your response
Comment
Friendly request
Explanation requested
Update
comment: Thank you for your detailed rebuttal. It really helps clarify my concerns. I agree that the formulation of CL as an ICL problem is interesting and the fact that the trained model can generalize is even more impressive. I am raising the score to 5 and am willing to increase it if more information/discussion is provided.

1. Can you discuss the importance/advantage/limitation of the self-referential weight matrix in facilitating in-context learning? Self-referential is a powerful idea but it is not very clear why you choose that algorithm for CL. 
2. At first glance, the update rule for $W_t$ is quite simple, is it general enough to learn all kinds of algorithms? This question might be out of the scope of this paper but I just want to have a conversation. 

Also, I still maintain that the presentation of this paper could be improved.
While we have been respecting the rule of not sending individual reminders to the reviewers ourselves, please allow us to post this single/final message as the end of author-reviewer discussion period is imminent.

We would like to thank the reviewer one more time for their valuable time spent on reviewing our work.

As all the other reviewers have already responded, we would appreciate it a lot if the reviewer could communicate their perception and rating of this work that take into account our rebuttal.

Thank you very much.
We acknowledge the increased score and appreciate your consideration. Thank you very much.
We really do not intend to bother the reviewer further, but we wanted to let you know that Reviewer d5XY has increased the score after considering our rebuttal... If you think their original review had influenced your score change, please do not hesitate to take a look at their response.
Thank you very much for your reply and the increased score. 

We genuinely thank the reviewer for the effort put into checking our rebuttal.
I thank the authors for their clarifications: 

* "up to 3 tasks" in my original review is a typo - I meant 5 tasks, and I thank the authors for pointing out the additional results in the appendix. This however doesn't alleviate my main concern here: These validation benchmarks are still of limited length and diversity. 

* I thank the authors for the clarifications and pointers regarding the memory constraint. This answers my concern. 

In general, I find that the authors answered most of my questions. Despite the limitation in the evaluation benchmarks mentioned above, I am raising my score.
Thank you very much for your response.

We agree that the paper will be much stronger if we could solve all these problems.

However, it is unreasonable to assume that one can solve all the problems in this single paper. 

It is not as if our paper lacks content; **we have a full-length paper with plenty of novel results.**

We are deeply disappointed that the reviewers seem to consider our results/contributions to be trivial, and only evaluate our methods through their limitations (when many of them can be solved directly by better-resourced teams).
I thank the authors for the detailed rebuttal. It appears that a majority of the concerns mentioned in the review has been considered by the authors but as future work (for example whether subsampling the loss terms for more tasks can work, or expanding beyond the image domain). The contributions of this paper would be much stronger for these results to be included in this paper. Given the existing results, the generality of this method remains quite unclear. I maintain my original recommendation.
Dear Reviewers,

While we have been respecting the rule of not sending individual reminders to the reviewers ourselves, we would still like to share our honest feelings that we have found the author-discussion period very unfair so far.

Given the borderline ratings by three reviewers, we wrote a thorough rebuttal to resolve the concerns raised by all the reviewers. 

In particular, we draw your attention to a few factual misunderstandings. For example,

Reviewer d5XY did not see the correct number of tasks used in the experiment ("showing an advantage of their approach in scenarios with up to 3 tasks"),

Reviewer HvJH overlooked comparisons to prior meta-learning methods,

and most critically, both Reviewers d5XY and HvJH raised concerns about a memory/storage problem of our training algorithms but any practitioner of linear Transformers knows that no such problem exists in practical implementations (if there were such an issue, it would have been impossible to run any experiments we ran): We find it unfair if we get rejected because of a problem that does not exist.

Influenced by these original reviews, Reviewer vTAU reduced the score from 6 to 4, which, in our view, lacks justifications as this decision ignores our rebuttal clarifications.

We completely understand that no single reviewer is expected to be fully familiar with all these technical aspects of this work, as we combine methods from various sub-fields of machine learning (continual learning, in-context/meta-learning, linear Transformers; and we believe this is the very strength of this work). However, we find that the current review/discussion status is unfair.

Regarding Reviewer x6Jn who seems to have thorough understandings of our work, we also addressed their remaining concern (regarding the impact of this work beyond our scope focused on SRWM) in our rebuttal.

We are respectfully waiting for justifications of their scores that take into account our rebuttal.

Thank you very much once again.

[We posted this message multiple times as OpenReview was experiencing some technical issues with notifications today. We deleted the old copies below.]
Dear Reviewer vTAU,

Thank you for your response.

However, given that you changed your score from 6 to 4, we would like to know more elaborated explanations/justifications, especially considering your high/influential confidence score of 4.

You wrote:

> I have missed major points raised by other reviewers

What "major points" are you referring to? As we explained in our rebuttal, we tried to resolve many concerns raised by the reviewers, including corrections of certain factual misunderstandings. They have not responded yet, but we hope they will.

We would like to express our concern that for now, our rebuttal has not been considered. This is very unfair.

If you found that our response did not convincingly address these points, please explain the reasons.
Thank you for the detailed explanation. I have missed major points raised by other reviewers and feel like I should change my original review. I believe that the paper presents very original idea and need just final finishing in terms of clarity.