Reawakening knowledge: Anticipatory recovery from catastrophic interference via structured training

Authors: yanlai  yang
matt  jones
michael c mozer
mengye  ren
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 04:51:29

Abstract: We explore the training dynamics of neural networks in a structured non-IID setting where documents are presented cyclically in a fixed, repeated sequence. Typically, networks suffer from catastrophic interference when training on a sequence of documents; however, we discover a curious and remarkable property of LLMs finetuned sequentially in this setting: they exhibit *anticipatory* behavior, recovering from the forgetting on documents *before* seeing them again. The behavior emerges and becomes more robust as the architecture scales up its number of parameters. Through comprehensive experiments and visualizations, we uncover new insights into training over-parameterized networks in structured environments.

Main paper: 
 introduction: 
 Large language models (LLMs) [1, 2, 3, 4] have demonstrated remarkable general capabilities in a wide range of natural language tasks. During the training of LLMs, documents are typically uniformly sampled at random. Due to the large scale of the training set-in contrast to many other domains-LLM training typically occurs in an online fashion: each document is used only once for just one update step without further repetition [5, 6, 7] . Such a training style is in stark contrast with how real world agents like humans acquire new knowledge. In naturalistic settings, the material we are exposed to is structured in time and often repeats in predictable, quasi-cyclic patterns (e.g., a person's everyday morning routine consists of first taking a shower, then eating breakfast, and finally dressing up). Hence it is important to understand how existing deep learning methods and architectures perform in this setting.Toward the goal of investigating more naturalistic training setups, we study a simplistic setting involving structured training of LLMs: documents are presented cyclically in a fixed sequence and repeated multiple times, just as we humans go through our daily routines. Moreover, to account for the cost of switching among documents (analogous to the mental switching cost between different environments and the waiting cost of obtaining new data), we allow the network to take multiple gradient steps for each document. Compared to standard task-incremental and class-incremental continual learning settings [8] which experience each task only once, our cyclic training setting better approximates the quasi-cyclic temporal structure of real-world environments.Typically, networks exhibit catastrophic interference (also known as catastrophic forgetting) [9] when training on a sequence of tasks: the loss on a given document increases as the training advances to other documents. Curiously, we discover that in a structured training environment, LLMs exhibit a remarkable anticipatory recovery behavior: they recover from the forgetting of 1 : Average online loss across epochs 2 to 5 for cyclic fine-tuning and random shuffled fine-tuning.one document before seeing it again, multiple steps in the sequence prior to the recurrence of the document (see Figure 1 (a) and 1(b)). It is analogous to a person anticipating to eat breakfast while taking a morning shower, but leaving the thought aside for the rest of the day. Critically, we never present two documents together in context, so the model cannot directly learn sequential relationships between them. Thus, our finding is surprising as there is no explicit memory in LLMs that stores sequential knowledge across context windows, and there is no systematic overlap of content across documents-the behavior emerges from a random document sequence after repeated exposure to that sequence. Furthermore, we demonstrate that, as a result of anticipatory recovery, training with fixed ordering achieves superior performance than random shuffling in the prequential evaluation [10] setting (see Figure 1 (c) and Table 1 ). For an agent that continuously acts and learns in the real world, the performance on the upcoming task is what matters, and prequential evaluation measures such performance. This result hints at the practical benefits of structured training.Through extensive experiments, we study how different factors in model architecture and training contribute to the anticipatory recovery phenomenon (Section 3.3). We show that only large-scale networks exhibit this reawakening of knowledge, and smaller ones exhibit no such behavior (Section 3.2). We also show that this phenomenon is not unique to LLMs; some vision models with sufficient width and depth also demonstrate a similar behavior, but LLMs on language modeling tasks exhibit the strongest recovery (Section 3.4). We offer insights on the training dynamics in sequentially and cyclically structured input data and propose hypotheses for the causes of the behavior (Section 4). 
data and experiment setup: 
 In this section, we describe the models, datasets, and training setups that we use in the subsequent experiments. Additional details are presented in Appendix A.Models. For the LLM experiments, we use Pythia [11] , a suite of decoder-only autoregressive language models pre-trained on the Pile dataset [12, 13] . We use pre-trained Pythia models ranging from 160M to 2.8B parameters. For the vision experiments, we use pre-trained Image GPT [14] models for causal image modeling and pre-trained vision transformer (ViT) [15] and VGG-19 [16] models for image classification.Datasets. For the LLM experiments, we use the CNN/Daily Mail news dataset [17] . We repurpose it for causal language modeling by discarding the summaries and only using the articles. Importantly, the CNN/Daily Mail dataset is not part of the Pile dataset and hence it is a new domain for the Pythia pre-trained models. We use the same documents for both training and evaluation. Our goal here is not to determine whether a trained model generalizes to new documents, but rather to study the memory for a particular document as a function of position within the training history. For the vision experiments, we use images sampled from CIFAR-10 [18] and ImageNet [19] . Training Setup. We randomly sample T documents from the dataset. In pre-processing, we truncate each document to the first C tokens (we refer to C as "context length" in subsequent text). We then fine-tune the LLM on each pre-processed sample for M gradient steps (i.e., using a batch size of 1). We refer to the multiple gradient updates of each document as an "episode". After each episode we evaluate the loss on all T documents. We repeat the training process for E epochs, where an epoch consists of one episode of each document in a fixed sequence. We use a vanilla gradient descent optimizer. Unless otherwise stated, the default hyperparameters in the subsequent experiments are T = 25, C = 256, M = 10, E = 5. We use the average cross entropy loss (average negative log-likelihood for each token) as our training and evaluation metric. 
emergent anticipatory recovery: 
 In this section, we present our experimental results that reveal the anticipatory recovery phenomenon in cyclic fine-tuning of large language models. We then demonstrate that anticipatory recovery is an emergent behavior that appears only for models with sufficient capacity.In this first experiment, we have T = 100 documents, and we do cyclic fine-tuning of a pre-trained Pythia-1B model [11] on the documents for E = 5 epochs in the same ordering. Both the documents and the ordering are sampled at random beforehand, but kept fixed during the sequential fine-tuning process. We refer to these T documents as x 1 , • • • , x T . At the start, we fine-tune on x 1 for M = 10 gradient steps, leading to a significant decrease in the model's loss on x 1 . As we move away from x 1 and fine-tune on other documents, we naturally observe catastrophic interference: the model's loss on x 1 gradually increases until we finish fine-tuning on all other documents and return to x 1 . As we iterate through the same document sequence for a second time, we would normally expect the loss on x 1 to increase monotonically after the initial decrease. However, Figure 1 (a) shows that the loss on x 1 peaks around x 60 and then starts to decrease. Before we return to x 1 , the model has recovered more than half of its initial forgetting during the second epoch. We refer to this counterintuitive decrease in loss as the anticipatory recovery phenomenon. In Figure 1 (b), we plot the losses for all the documents and re-align them so that 0 on the x-axis refers to the loss on each document t immediately before training on it for the first time. The figure confirms that the anticipatory recovery phenomenon exists for not only x 1 but all documents. On the other hand, when we randomly shuffle the document order within each epoch (except x 1 is always the first document), we do not observe such anticipatory recovery behavior, and the loss on x 1 keeps increasing before we return to it every time.To quantify the strength of the anticipatory recovery phenomenon, we define the recovery score as the proportion of the initial forgetting during the current epoch that the model recovers before returning to the same document. Mathematically, let the mean (over t) of the maximum loss on each document x t between the n th and (n + 1) th time we train on that document be l max (n), right before the (n + 1) th time we train on it be l before (n), and right after the (n + 1) th time we train on it be l after (n). Then we define the recovery score (RS) for epoch n to be1 RS(n) = lmax(n)-lbefore(n) lmax(n)-lafter(n-1) . In the following subsections we compute the recovery scores for different model sizes and training hyperparameters (Figures 2 and 5 ) to investigate their effects on the anticipatory recovery phenomenon.To study how the model size affects the amount of anticipatory recovery, we repeat this experiment with pre-trained Pythia models [11] of sizes 160M, 410M, 1.4B, and 2.8B. We plot the average loss curves as well as the recovery score for epoch 4 in Figure 2 (a) . We observe that larger models clearly demonstrate stronger anticipatory recovery. The sharp increase of average recovery score from the 160M model to the 410M model indicates that anticipatory recovery is an emergent behavior.Anticipatory Recovery in Randomly Initialized Models. To study whether anticipatory recovery is a result of pre-training, we repeat the experiments on randomly initialized models of different sizes, and plot the loss curves and average recovery scores in Figure 2(b) . We follow the model initialization recipe of [11] . From the loss curves for the 410M and 1B models, especially in the last epoch, we see that the anticipation phenomenon also exists in randomly initialized LLMs. We observe that the anticipation effect is not as strong as in the pre-trained models. The effect of model size still holds: larger models clearly demonstrate stronger recovery.Effects of Model Width and Depth. To further study the effect of model width and depth on the anticipatory recovery phenomenon beyond the model hyperparameters in the Pythia suite, we take a Pythia-1B model and vary the width (size of token embedding) and depth (number of transformer blocks) of the model and plot the average loss curves for cyclic training from random initializations in Figure 3 . The original Pythia-1B model has token embedding of size 2048 and 16 transformer blocks.We observe that the model needs sufficient width (at least 512) and depth (at least 8 transformer blocks) to exhibit noticeable recovery, confirming that it is an emergent behavior contingent on model size.In this section we discuss the effect of other training hyperparameters on the anticipatory recovery phenomenon. We also include additional experiment details in Appendix A. This observation is consistent with section 3.2 and confirm that the model needs sufficient depth to exhibit anticipatory recovery even with a frozen pre-trained deep representation.Optimizer. In addition to the gradient descent optimizer, we experimented with the Adam optimizer [20] . Loss curves are plotted in Figure 6 . We reset the optimizer state for each document. Results show that Adam, which is a stronger optimizer, further facilitates anticipatory recovery for both randomly initialized and pre-trained models.Data Randomness. In realistic sequential learning setting the data points might be slightly different for each repetition (e.g. different descriptions of the same concept, different perspectives of the same object), leading to stochasticity in the optimization process. To explore sequential cyclic training with data randomness, we design the following two training settings: (1) we randomly mask a subset of the tokens in the input, and (2) we randomly shift the "window" of C tokens used for training. The resulting loss curves are plotted in Figure 4 . We observe that, while anticipatory recovery is generally weaker when there is more variation in each data point, the recovery still clearly exists.Summary. The experiment results in this subsection suggest that the model's ability to fit on each task is crucial for the strength of anticipatory recovery. With a larger number of gradient steps, shorter context length, more learnable layers, and a better optimizer, the model is more capable of fitting to the focal task, and those factors also correlate with larger recovery score. We also confirmed that anticipatory recovery exists for long task sequences and slightly augmented data points within each episode, and again these factors that make learning harder also reduce anticipatory recovery.To examine the generality of the anticipatory recovery phenomenon, in this subsection we explore the sequential cyclic training setting on two tasks in computer vision: causal image modeling and image classification. More detailed experiment setup is available in Appendix A.Causal Image Modeling. Similar to the LLM experiments, we fine-tune a pre-trained Image GPT model [14] on each sampled image from CIFAR-10 [18] for M gradient steps, and repeat E epochs with a fixed order of the images. The resulting loss curves are shown in Figure 7 (a). The results show that the anticipatory recovery phenomenon also exists for sequential cyclic training of image modeling in addition to language modeling. Image Classification. For each experiment, we randomly sample 800 images from Imagenet [19] and divide them into 25 batches of 32 images each. We fine-tune a pre-trained vision transformer (ViT) [15] and VGG-19 [16] model on each batch for M gradient steps and repeat E epochs with a fixed order of the batches. The resulting loss curves are plotted in By these experiments we confirm that anticipatory recovery occurs not only for LLMs but also for at least some of the widespread image classification models and non-transformer architectures.We compare the performance of training with fixed ordering and random shuffling of each epoch in the prequential evaluation setting with T = 100 documents. Prequential evaluation [10] measures online performance by evaluating the model on the document it is about to be trained on, which is equivalent to evaluating the training loss of each batch. For the random shuffling condition, document 1 is always trained first in each epoch, and the other 99 documents are presented in a different random order in each epoch. In this experiment we set C = 512, M = 10, E = 5 and the error bars are based on 10 different seeds.The resulting loss curves are plotted in Figure 1 (c), and the average loss throughout each run is summarized in Table 1 . We exclude epoch 1 when computing the average loss since all documents are new to the model for both the cyclic training and random shuffling condition. We observe that training with fixed ordering is superior to random shuffling in the prequential evaluation setting across all 4 pre-trained Pythia models of different sizes, due to the structure in the data stream. The results suggest practical potential of structured training. 
understanding cyclic training dynamics: 
 An important general question about anticipatory recovery is whether it is due to some causal mechanism relating the dynamics of model parameters to the training sequence, or whether it is more correlational in that adjacent tasks come to be represented more similarly by the model. We found initial evidence for the latter hypothesis in experiments locally reversing the task sequence (e.g., showing that x t+1 primes x t nearly as much as vice versa). To further test this learned similarity hypothesis, we explore the relationships between tasks and the model's loss gradients, weights, and activations across training history. The results enable us to better understand the dynamics of cyclic training. Unless otherwise stated, all visualizations in this section use the 410M model and default hyperparameters.We first explore the similarities of gradient information between documents during the training process. Our goal is to test the hypothesis that anticipatory recovery is mediated by increased similarity between gradients of proximal documents in our training sequence.We do cyclic training for 4 epochs and compute the gradient of each document at the attention layer of transformer block 12 at the conclusion of training. In Figure 8 (a), we plot the cosine similarities between these gradient vectors of each document. Results show that the gradients have mostly positive cosine similarities (except for the last document, on which the model has just been trained). To our surprise, the gradient similarities are highest near the center of the heat map rather than peaking along the diagonal. That is, the gradient similarity between documents x t-1 and x t depends on where we are in the cycle. This result suggests an additional layer to the anticipatory recovery phenomenon: Recovery for document x t is greatest from training on document x t-1 , but the strength of the potential facilitation between x t-1 and x t is actually greatest after we train for another b documents (for some small number b). We verify this by computing the pairwise recovery: we take the model checkpoint after 4 epochs of cyclic training, and then for each pair of documents (x i , x j ), we do M gradient updates on x i and compute the difference in the loss of x j before and after these gradient updates. We plot these pairwise loss recoveries in Figure 8(b) . Results confirm that the amount of recovery on document x j is highest when the model checkpoint is taken from roughly b documents before or after document x j in cyclic training and then fine-tuned on a proximal document x i in the sequence. The fact that this pairwise loss recovery matrix is roughly symmetric also suggests that the anticipatory recovery phenomenon approximately exhibits task symmetry: gradient updates on document x t decrease the loss for document x t+k for small integers k, and vice versa. We provide additional visualizations for T = 50, 100, 200 in Appendix C and a more detailed description for this phenomenon.We explore the structure of model weights along the optimization trajectory of cyclic training. We flatten and concatenate the model weight vectors after fine-tuning on each document. However, the cosine similarities between these raw model weight vectors are all very close to 1 without obvious structure, due to the proximity of model weights along the same optimization trajectory and numerical instability in huge weight vectors. To resolve these issues, we instead explore the structure of "weight residuals." We compute the weight residuals by subtracting the average of weights in a window of length T centered at the current document from the current weight, i.e. w res (t) = w(t) -1 T t+T /2 n=t-T /2 w(n). This removes the shared components along the optimization trajectory and allows us to focus on the model weight updates for each document. Figure 8(c ) visualizes a heat map of the cosine similarity between each pair of weight residuals from the second epoch to the fourth epoch. The visualization shows a cyclic structure in the weight residuals, as equidistant bright stripes that align with the training epochs. Furthermore, each stripe spans several documents, suggesting the similarity of weight residuals for proximal documents.In addition to cosine similarities between weights, we explore visualizing the weights in a lowerdimensional space with Principle Component Analysis (PCA). We compute the top three PCs of the flattened last-layer weight vector (the output word embedding layer) for the Pythia-1B model, and plot its trajectory in Figure 9 . The plot exhibits a clear helical structure that gradually converges. We believe this is highly relevant to anticipatory recovery: right before revisiting a task, the projected model weights in the helix move closer to the point corresponding to the previous appearance of that task, leading to anticipatory recovery on the loss of that task. As we go on with cyclic training, the model also exhibits less forgetting and gradually converges to a solution that achieves low loss on all tasks. It is important to note that the helical structure of the weight trajectory is not an obviously necessary consequence of the cyclical training. Cyclical training could be expected to yield a repeating pattern, but the facts that the tasks come to be organized in a circle that respects their ordering and that the trajectory goes through one full revolution per epoch (rather than some other arc length) are nontrivial and seem to be essential for anticipatory recovery.In addition to gradients and weights, we visualize the trajectory of activations on a single document during the course of cyclic training. We do cyclic training for three epochs and save model checkpoints after fine-tuning on each document. We then compute the model activations before the output word embedding layer for document x 1 on each model checkpoint, and plot the cosine similarities between the flattened activation vectors in Figure 8(d) . From the plot we can clearly observe the blocked pattern wherein the similarity between the activations become progressively higher across each epoch of cyclic training. This pattern suggests that every time we train on document x i , the internal representation of x i in the model is more resistant to gradient updates on other documents x j .To further understand the essential mechanism that yields anticipatory recovery, we design a minimalist "toy" simulation experiment. In this toy simulation, each task (formerly, document)i ∈ {1, • • • , T } is described by a single data point, x 1 , • • • , x T ∈ R N .We assume a learnable linear embedding P ∈ R M ×N that projects each x i into an M -dimensional embedding space. We also assume a learnable vector w and task-specific mappings f i , where f i (w) is the target for task i in the same embedding space. We require each f i to be invertible as a simplifying assumption.We define the loss for task i asℓ i (P , w) = 1 2 ∥P x i -f i (w)∥ 2 2 .Just as when training a deep net, we assume here that representation learning occurs slowly, and that one training step for task i involves a single gradient update of P with step size α:EQUATIONIn contrast, at each training step, w, analogous to the fast-adapting weights in a neural network, can be rapidly tuned to solve for task i, yielding the loss minimizer conditional on P :EQUATIONAs in our main experiments, we sequentially optimize each ℓ i as we iterate through the sequence of tasks. In each training step, we first update P and then solve for w given the update to P . Updating P approximately reduces the distance between P x i+1 and f i+1 (f -1 i (P x i )), which entails reducing the upper bound of ||f -1 i+1 (P x i+1 ) -f -1 i (P x i )||, assuming that each f -1 i is Lipschitz continuous. As a result of this optimization objective, the model will evolve along the optimization trajectory such that the f -1 i (P x i ) for all tasks i gradually form a circular pattern. This gives an intuitive explanation on the anticipatory recovery phenomenon, since updaing w according to equation 2 will also bring it closer to f -1 i+1 (P x i+1 ), thus reducing the loss on task i + 1 and exhibits anticipatory recovery. We experimented with two very simple choices of f i : f i (w) = w and f i (w) = y i -w for some task-dependent targets y i . We follow the same order over tasks-1, • • • , T -for multiple epochs of training. The resulting loss curves are shown in Figure 10 , which exhibits very similar anticipatory recovery trajectory as the full-blown LLM experiment. Visualizations of the 2-dimensional PCA embeddings for f -1 i (P x i ) in the second experiment are shown in Figure 11 , which confirms our analysis that they gradually self-organize into a cyclic structure.There are two potential reasons large overparameterized networks might produce the anticipatory recovery in a way analogous to the toy simulation. First, for larger networks, it is more likely that the network can develop task-specific parameters that quickly adapt to and memorize new input data, corresponding to Equation 2. And when the fast memorization is achieved, the gradient descent dynamics of the slow weights push the representations of the two adjacent tasks (P x i and P x i+1 ) closer when f is an identity function, according to Equation 1. This effect can be seen in earlier LLM experiments (Figure 2 ), where larger models achieve significantly lower losses within a few gradient update steps. Second, larger networks have more learning capacity to map the features of two adjacent tasks closer. In our linear projection model, anticipatory recovery keeps growing over many epochs, whereas the anticipatory effect is already at the strongest within 2 or 3 epochs in LLM experiments. Moreover, all data points are randomly generated in the toy model, which makes it easier to separate and map their representations according to a temporal structure than real-world data. In contrast, real-world data could require more representation capacity since data points are noisy and correlated.In this section, we visualized model weight dynamics with heatmaps and we showed model activations and gradients during cyclic training. We discussed the special temporal structure that is exhibited in these heat maps. We also plotted the pairwise degree of recovery for fine-tuning on document i and evaluating on document j, as well as the change of distance between fine-tuned model weights on different tasks. The results suggest that after we train on a document, the model's representation of that document becomes less sensitive to gradient updates on other documents. Finally, we showed a simple toy experiment that demonstrates a similar anticipatory recovery phenomenon in its loss curve, and discuss its connections to neural network training dynamics through the lens of task-specific and task-general parameters. Overall, these results shed some light on the dynamics of cyclic training. 
related work: 
 In this section we discuss the most relevant prior works to this paper. Please refer to Appendix D for additional related work.Cyclic and Structured Training. Prior theoretical works have studied convergence rates, under various assumptions, for the training setup where the data points are shuffled only once and that order is reused for all epochs [21, 22, 23, 24] . On the empirical side, [25] found that shuffling the data only once in the beginning can achieve a convergence rate comparable to shuffling every epoch. The training setup is equivalent to our cyclic training setup, but our research examines the loss on each task throughout the training cycle and discovers the anticipatory recovery effect. We also extend it to multiple gradient update steps on each data point.Online Learning. Online learning deals with the setting where the tasks come from an online sequential stream. One of the simplest algorithms in online learning is follow-the-leader [26] , which stores all previous data from the stream and minimizes the total loss. It has strong performance guarantees but is computationally very expensive, and it also might not be feasible to store all the past data. Many subsequent works have developed cheaper algorithms under different assumptions [27, 28, 29] . Many recent works also explore the connection between online learning and meta-learning or continual learning [30, 31, 32, 33, 34, 35, 36] . The cyclic training setting that we explore in this research can be considered as a special case of the online learning setting where the data stream has a cyclic repetition structure. We employ multiple steps of online gradient descent [37] on each document from the stream and study the training dynamics of over-parameterized neural networks.Catastrophic Interference. When transitioning between tasks sequentially, neural networks often experience "catastrophic interference" [9] , marked by a significant drop in performance on previously learned tasks. Numerous algorithms have been proposed to mitigate catastrophic forgetting, focusing on general approaches including parameter regularization [38, 39, 40] , data replay [41, 42, 43] , knowledge distillation [44, 45, 46, 47] , and architectural isolation and expansion [48, 49, 50, 51] . Our work extends interleaved training [52] to a larger number of tasks, specifically investigating the emergent anticipatory recovery phenomenon in cyclic training. This finding adds to the above literature by demonstrating a new mechanism by which large networks can avoid or recover from catastrophic interference.Continual Learning Continual learning [8, 53, 54, 55] addresses a simplified setup where a model sequentially learns a set of tasks without revision. Recently, there have been debates over the practicality of continual learning setups. Studies like [56] have shown that as networks learn more tasks, they improve in learning speed and reduce forgetting. In large models, studies suggest that pre-trained vision classifiers can undertake continual learning with ease, by either freezing or fine-tuning representations [57, 58, 59 ]. In the language domain, research also suggests that LLMs exhibit emerging continual learning capabilities [60, 61] . Nevertheless, it is uncommon in real environments for tasks to occur only once yet for an agent to need to retain them. Unlike prior literature on continual learning, our research uniquely focuses on sequential learning environments with cyclic repetition.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We provide the code and instructions for reproducing main experimental results in the supplementary material.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
discussion and limitations: 
 In this work, we explored the training dynamics of overparametrized neural networks, especially LLMs, in sequential cyclic fine-tuning, where a finite set of documents are presented in the same order within each epoch. We demonstrated the remarkable phenomenon of anticipatory recovery-networks recover from the initial forgetting before seeing the same document again. The effect holds across many different network instances and training hyperparameters. This phenomenon is a sharp contrast with the well known phenomenon of catastrophic interference, where forgetting increases monotonically as a network is trained on a sequence of different documents.We showed that anticipatory recovery occurs only when the network has sufficient width and depth and when it is well fitted to each document before moving to the next. Visualizations of model weights, model activations, and gradients exhibit clear temporal structure, which provide insights on the underlying mechanisms of anticipatory recovery.Our research indicates that there is value in exploring naturalistic task sequences within continual learning, where tasks interleave in statistically regular patterns. This approach could expand the field's current focus on learning and retaining new tasks to also consider how effectively previously encountered tasks are re-learned when they reappear. With the anticipatory recovery phenomenon, we discovered a mechanism in which ML models can do surprisingly better than expected on prequential evaluation. By analyzing the different factors of model pre-training and fine-tuning that moderate this phenomenon, our experiments provide a promising first step toward leveraging structured training with agents in realistic environments.Limitations. The cyclic training setup investigated in this work is distinct from the IID training setting assumed in the vast majority of the machine learning literature. It accounts for task repetition and task switching costs, which are critical components of the learning experience of humans and other real world agents. However, our current setup is still highly simplified. Future research could investigate the emerging training dynamics of neural networks in different types of structured environments, such as multiscale temporal dynamics [62] , from both theoretical and empirical perspectives.The mathematical foundation of anticipatory recovery also requires further investigation. Although our computational toy model reproduces the anticipatory recovery phenomenon, it does not explain why the effect is stronger in LLMs and autoregressive tasks than in other types of architecture or learning objectives. In terms of the empirical experiments, the ablation studies are only run in a single setting. Future research could run the experiments in more settings to reach more conclusive results and investigate possible alternative theoretical explanations to the anticipatory recovery phenomenon. Throughout the paper we have been focusing on the setting where the document ordering is sampled once and stay fixed for all epochs. What if we only fix the first and the last document, and shuffle the documents in between? We experimented with shuffling the documents from document x 2 through x N for N ∈ {8, 16, 24} every epoch. In Figure 12 we plot the loss curves for document x 1 . From the loss curves we can observe that even when N = 24 we can still observe some anticipatory recovery, suggesting that the order of the tasks between two consecutive repetitions of the x 25 and x 1 can be arbitrary for us to observe recovery on x 1 .In Figure 5 (b) we observe that there is no anticipation when we take only one gradient descent step on each document with learning rate 0.001. We experimented with one-step gradient descent using a higher learning rate, 0.01. We plot the resulting average loss curves under the same training setup in Figure 13 . We observe that, with a larger learning rate, slight anticipation is still observed for 1 gradient step.In addition to varying the model width and model depth in Figure 3 , we also experimented with varying the number of attention heads h ∈ {2, 4, 8, 16} while keeping model width to be 2048 and model depth to be 16. Loss curves on document x 1 are shown in Figure 14 . The results suggest that the number of attention heads does not have a big effect on cyclic training in our setting.Here we compare the performance of the initialization scheme used by [11] (also used for all randomly initialized models in the main text) and a simple initialization scheme that samples the weight matrices from an isotropic Gaussian distribution with σ = 0.02. The loss curves for document 1 under these two initializations of the Pythia-1B model are plotted in Figure 15 . We observe that Pythia's initialization scheme achieves much better average loss and also exhibits stronger anticipatory recovery. This demonstrates the importance of LLM initializations. The result is consistent with our observations in section 3.3 that the model's ability to fit on each task is correlated with the amount of anticipatory recovery.In addition to comparing pre-trained models and randomly initialized models in Figure 2 , we further study the effect of model pre-training by examining model checkpoints with different numbers of pretraining steps. We took Pythia models pre-trained for 6K, 12K, 24K, 48K, and 96K steps and plot the shift-averaged loss curves for cyclic fine-tuning in Figure 16 . We found that more pre-training does give rise to higher anticipatory recovery. As we summarize at the end of section 3.3, we hypothesize this result fits a broader pattern in which the strength of the anticipatory recovery effect is related to how well the model can fit each successive training task. Models with more pre-training steps are more capable of fitting each successive training task, and therefore exhibit higher anticipatory recovery.For experiments in the main paper we used a constant learning rate during the fine-tuning process. To study whether anticipatory recovery occurs in typical LLM optimization schemes, we experimented with cosine learning rate scheduling on the Pythia-1B model with 10 epochs (minimum learning rate = 0, maximum number of epochs = 10), and plot the results in Figure 17 . We show that the model also exhibits the anticipatory recovery effect in other learning rate schedules.To examine how the anticipatory recovery effect may generalize to other forms of structured training, we experimented with a new setting where only the first 20 documents are kept fixed in each epoch, and a random number of other documents (between 20 and 100) are inserted after the first 20. These random padding documents appear only once in the entire sequence. This new setting generalizes cyclic training in that (1) rather than having the same documents in every epoch, we insert random other documents between every repetition (2) epochs can have different lengths. The resulting loss curve is plotted in Figure 18 . We still observe anticipatory recovery for documents 2 through 20 in this setting, suggesting that anticipatory recovery exists as long as there is a repeating sub-sequence in the data stream. In Figure 19a we experimented with inversely scaling the learning rate with the number of gradient steps. We use a learning rate of 0.01 for M = 1, learning rate 0.002 for M = 5, learning rate 0.001 for M = 10, and learning rate 0.0005 for M = 20. The results suggest that the anticipation effect is stronger when the same total update is divided among more gradient steps.In Figure 19b we experimented with different number of gradient steps M ∈ {10, 20, 40} for context length 1024. The results confirm that longer context length is not a fundamental limitation to anticipatory recovery, and we can achieve the same recovery score as a smaller context length with more gradient steps. To evaluate how the anticipatory recovery phenomenon generalizes across different LLM architectures, we experimented with GPT-2 architecture [64] , specifically the GPT2-large pre-trained model (812M parameters) on the CNN/Daily Mail dataset. The loss curve for document 1 is plotted in Figure 20 . The model consistently observed anticipatory recovery. Note that GPT-2 is the predecessor of many modern LLMs and therefore the results further suggest that the anticipatory recovery phenomenon is prevalent among more recent LLM architectures.To evaluate how the anticipatory recovery phenomenon generalizes across different natural language datasets, we experimented with the wikitext-103 dataset [65] , which contains over 100 million tokens from articles on Wikipedia. Since Wikipedia data is part of the pre-training dataset of Pythia, we only experiment with randomly initialized models. The loss curve for document 1 is plotted in Figure 21 . The result suggest that the anticipatory recovery phenomenon is generalizable to different data sources.We plot the magnitude of the difference between the x 1 activations of model checkpoints saved at consecutive episodes throughout four epochs of cyclic training of a Pythia-410M model in Figure 21a , and observe a clear stepwise pattern. In contrast, the magnitude of model weight updates (Figure 21b ) decreases monotonically over the training episodes and do not exhibit this stepwise pattern. This result is consistent with the pattern we observe in section 4.3. Similar to Figure 8 (b), we plot the pairwise loss recoveries for each pair of documents (x i , x j ) in longer document sequences, where T = 50, 100, 200 respectively, in Figure 23 . We use the 1B model and default hyperparameters. We observe that, as we increase the length of the document sequence, the highlight area near the center of the matrix is separated into two blobs, one in the top-left corner and the other in the bottom-right corner. We also observe a "boundary" on the sides of the matrix where there is little or no recovery. The width of this "boundary" stays relatively constant across different lengths of document sequences and is around 10 to 15 documents. This confirms our observation in the main text that the recovery on document x j when fine-tuning on a proximal document x i is highest when the model checkpoint is taken from document x j±b where b is a small number relative to the length of the document sequence.Learning in Structured Environments. Our research also relates to the more general topic of learning in structured environments. [62] studied regression and classification tasks with multi-scale temporal structure in the environment characterized by 1/f dynamics. While the cyclic training setting that we study is a more simplified setup than that of [62] , we aim at unveiling more insights on applying standard SGD on over-parameterized networks. A potential direction for future work would be to study anticipatory recovery in regimes with richer, hierarchical sequence structure.LLM Emergent Capabilities. Recent advancements in large-scale Transformer networks [66, 1] have demonstrated exceptional ability to model long sequence language data. Beyond basic language modeling and downstream task performance, these models have shown emergent behaviors [67] that appear to manifest only beyond a certain model scale [2, 68, 69, 70] . Related to our research, recent studies reveal that LLMs possess remarkable memorization skills, enabling them to recall news sentences after just a few exposures [11, 71, 72] . However, the sequential learning dynamics behind such memorization have not been thoroughly examined. Our work comprehensively explore the sequential learning setting with cyclic task repetition and demonstrates task anticipation, a new emergent capability of large models.This research deepens our understanding of catastrophic interference in naturalistic training setups. This understanding could lead to the design of better training algorithms of LLMs and other large neural networks that are more similar to human learning. These algorithms may give rise to more powerful and embodied AI systems with online adaptive learning capability, which may have many potential societal consequences.Recovery from catastrophic interference might be undesirable in scenarios where some documents are intended to be forgotten. Our research provide more understanding into the anticipatory recovery phenomenon. Future research into the unlearning problem as well as a deeper understanding of the training dynamics of different types of structured environments can help further address the privacy issue.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: The experimental settings and details are provided in Section 2 and Appendix A.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: All experiment result figures and tables in Section 3 of the paper are accompanied by error bars or confidence intervals. Some error bars might not be visible since they are too small.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: This paper does not release new data or models. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: The original papers for all the code, data and models used in the main experiments of the paper are cited. All these existing assets involved are publicly available, under the Apache 2.0 License (Pythia, huggingface transformers) or MIT License (cnn/dailymail version 3.0.0). Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: This paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.In some cases, a randomly initialized model will produce loss curves that decrease throughout the epoch, because its knowledge is so poor that it enjoys positive generalization among all documents. This yields a misleadingly large recovery score under this definition. We do not include such cases in our experiments so do not bother with more nuanced recovery scores. 
 
 Appendix:  


Decision: This paper studies a specific kind of structured training dubbed cyclic training, where documents are presented cyclically in a fixed, repeated sequence. This reveals that LLMs exhibit anticipatory recovery behavior, i.e. they begin to recover from forgetting a document before encountering it again in the sequence. Some insights into the training dynamics in the cyclic training setup are offered, emphasizing that this anticipatory behavior is not exclusive to LLMs but can be observed in other large-scale networks as well. Reasons for this anticipatory recovery are investigated: finding initial evidence that suggests adjacent documents in the training sequence are represented more similarly by the model.

Scores include strong accept (8), accept (7),  borderline accept (5), and weak reject (4). Despite this range of scores, the reviewers fundamentally agree on:
- Strength: what makes this problem exciting is  that  it “identifies a surprising phenomenon where models exhibit anticipatory recovery, which is counter-intuitive given the understanding of how LLMs handle sequential knowledge without explicit memory mechanisms” (reviewer dpPf who gave a 8)
- Weaknesses: there are serious “significance” issues (some rate it in the contribution score). As stated by reviewer 5Cpf who gave a 4: “The main weakness of the paper is its unclear connection with common practices in training neural networks, therefore, it can be a little bit difficult to appreciate the significance of the findings”

To dig out further in the issue of the significance of this work:
- Could it lead to improved "learning curricula that achieve learning efficiency “?
 However, several reviewers do not see much future in the fixed order training required by the method “training with fixed example order usually achieves worse generalization performance than random-shuffling”.
- Could it help our theoretical understanding of the training dynamics of neural networks? However, this paper only brings a little light to this understanding.

In summary, the reviewer disagreement is about how critical the paper should be to the ML practitioner, which represents nowadays a large fraction NeuRIPS attendance. As a long time NeuRIPS attendee, I don't think it should be limited to the practitioner, and even a paper whose main contribution is to challenge our understanding of the learning process is worth publishing, so I am inclined towards accept.

Reviews: review: paper_summary: paper_summary: This paper takes a deeper look at the training dynamics of neural networks such as LLMs in a particular kind of non-IID learning setting when tasks or data is iterated through cyclically in a fixed repeated sequence. The authors find evidence in this setting of an emergent behavior that they call "anticipatory recovery" where the model improves its loss on data before it actually arises seemingly in anticipation of the upcoming data. This behavior is surprising because the neural network does not contain an explicit memory mechanism for this purpose. The authors run a number of experiments to provide further clarity about the nature of this phenomenon and present a simple toy setting to help explain how this may happen.
strength_weakness: strength_weakness: The writing of this paper could be improved somewhat. Reading through the paper can feel like going along for a journey with the authors with very little idea of where it is headed ahead of time. Typically papers summarize the main findings upfront to a larger extent. A lot of the time it feels like a large set of results are presented with little explanation of their connections and then the authors add in a summary section to recap how everything ties together. It would be better to provide the summary up front so readers know where it is going. 

The authors could also do a better job of contextualizing the potential implications of their work (see questions below). Moreover, the empirical results are somewhat limited by the choice to prioritize breadth in terms of the number of questions asked rather than depth in terms of coming to conclusive answers to these questions by running them over a number of datasets and architectures.
questions: questions: Q1:  The authors seem to indicate that they always use a vanilla gradient descent optimizer. While it is obviously interesting to consider this case, I wonder whether this analysis is less relevant for typical methods of training LLMs i.e. with Adam optimization and cosine decay learning rate schedules. Do you have any thoughts on how this may impact recovery? 

Q2: Do you believe that changing the total amount of data may change the nature of anticipatory recovery? I am also trying to understand the potential relevance to the one pass very large data setting. I guess due to shuffling, recovery is not expected despite redundancy? 

Q3: I wonder if the authors see any actionable insights to follow up from this study. Are you actually advocating the cyclic training? Do you see any potential implications for continual learning or curriculum design?
limitations: limitations: The authors perform a number of experiments in the paper, which they use to draw various conclusions, but the authors could do a better job of noting the limitations in these experiments i.e. individual ablations only being run in single settings (which is understandable but still limits how much we could read into the results) and possible alternative explanations. Conceptually, the authors made the choice to consider a variety of experiments trying to answer a variety of questions rather than asking fewer questions and coming to more conclusive results about the answers to these questions.
paper_summary: paper_summary: The paper studies a specific kind of structured training dubbed cyclic training, where documents (or tasks) are presented cyclically in a fixed, repeated sequence. The paper reveals that LLMs exhibit anticipatory recovery behavior, i.e. they begin to recover from forgetting a document before encountering it again in the sequence. The paper offers some insights into the training dynamics in the cyclic training setup. emphasizing that this anticipatory behavior is not exclusive to LLMs but can be observed in other large-scale networks as well. The paper investigates the reasons for this anticipatory recovery, finding initial evidence that suggests adjacent tasks in the training sequence are represented more similarly by the model.
strength_weakness: strength_weakness: 1. While the anticipatory recovery phenomenon is intriguing and counterintuitive, its practical value remains unclear. The paper investigates implications on Online Loss Evaluation, but the experiments are limited and do not convincingly demonstrate why cyclic training would perform better than random training. It is also uncertain if the observed benefits are directly tied to anticipatory recovery or influenced by other experimental confounders.
2. The motivation for using cyclic training is not particularly convincing. Cyclic training appears less natural compared to continual learning, and it is unclear if understanding training dynamics in this setup provides deep insights into the training of large models. This connection is weak and requires further development to establish its significance.
3. Although the experiments are thorough, they are somewhat narrow in scope. The study primarily focuses on Pythia pretrained models and evaluates them using the CNN/Daily Mail dataset. It would have been beneficial to include a broader range of models and evaluation datasets. Additionally, the distinction between context length in the training setup and the models' context length is not clearly explained, leaving ambiguity about its impact on loss evaluation.
4. While lesser priority, the paper lacks citations for a substantial body of work on continual learning in LLMs, such as AdapterCL by Madotto et al. (2020), Lfpt5 by Qin & Joty (2021), and ProgPrompt by Razdaibiedina et al. (2023).
questions: questions: Can you please address each of the weaknesses presented above?
limitations: limitations: NA
paper_summary: paper_summary: The authors identify a phenomenon termed anticipatory recovery, where if the model is trained on a fixed shuffled data sequence for multiple epochs, the loss of a training sample will first increase and then decrease again right before it is met again. Extensive experiments have been conducted to show the phenomenon happens across different model size, optimization, initialization states, and several hyperparameters etc. The authors also show the existence of certain cyclic structure in data gradient, attention, and activations. Finally, a simple toy example is demonstrated for intuition.
strength_weakness: strength_weakness: In terms of presentation, several settings and contributions can be improved by providing more discussion. The analysis of the computational toy model can be improved.  See **questions** for more detail.
questions: questions: 1. In Figure 1(b), how exactly is the loss computed? Say when you plot the figure for the 0.5th epoch, how is the number obtained over all documents?
2. Figure 5 needs legends.
3. Can you give more context on the prequential evaluation? What I had in mind about prequential coding is the same as looking at the training loss of each batch (see [1]).
4. For figure 8(a, b), as a sanity check, if you reshuffle the data sequence (and fix that sequence for cyclic training), the same structure will show, is that correct?
5. Around line 273-275, is the argument theoretically justified?
6. From figure 6(a), we see that the anticipatory recovery happens with Adam but not quite with GD if the model is randomly initialized. Intuitively, Adam keeps a memory of the data sequence, but not GD. However, when the model is pretrained, then both GD and Adam show anticipatory recovery. This seems to suggest that pretraining is one of the most important factors. I would like to hear more from the authors.
7. Can you explain more about the significance of Figure 8(c)? What is the $T$ used for this figure? The residual is the averaged weight residual over $T$ steps, so for each $t$, the residual will have $T-1$ overlaps, doesn't the overlap contribute significantly to the cyclic structure?

[1] The Description Length of Deep Learning Models
limitations: limitations: Limitations have been discussed.
paper_summary: paper_summary: This paper explores the training dynamics of neural networks, particularly language models in a structured training setting where examples are presented cyclically. The paper discovered a phenomenon called "anticipatory recovery," where models in each training epoch recover from forgetting of an example right before encountering the example. The paper show that the behavior is due to the temporal structure of gradients and model weights in cyclic training, and provided a theoretical characterization of the phenomenon using a toy model.
strength_weakness: strength_weakness: The main weakness of the paper is its unclear connection with common practices in training neural networks, therefore, it can be a little bit difficult to appreciate the significance of the findings. 

  - The paper contrasts the anticipatory recovery phenomenon with catastrophic forgetting, but this is an unfair comparison as anticipatory recovery happens only when the model is trained on the same data repeatedly, while catastrophic forgetting mostly happens when the model is trained on new data and never again have access to the old data. There is typically no catastrophic forgetting in multi-epoch training setting, and anticipatory recovery cannot happen if the old data is never seen again.

  - The anticipatory recovery phenomenon may have limited practical implications, for 1) training with fixed example order usually achieves worse generalization performance than random-shuffling due to lack of mini-batch gradient diversity. 2) although anticipatory recovery relieves forgetting, it only reduces the forgetting of the next few examples in the coming sequence, rather than reducing forgetting of all previous examples as is the goal in continual learning. 3) the model weight seems to be forming a cyclic structure under cyclic training, which pushes the model weight to become closer to its corresponding state in previous epochs. Would this also mean that the model is farther apart from its state in the other half of the epoch (compared to random-shuffle training)? If so, less forgetting of the next few examples may come at the cost of more forgetting of the other examples in the dataset.

  - The author proposes to examine the potential of structured training, but it is not clear how the anticipatory recovery phenomenon can benefit structured training. Why is recovery from forgetting right before training useful, when the model is soon to be trained anyway? Also, the anticipatory recovery phenomenon seems only exist in the cyclic training setting, which is a very special form of structured training. It is not clear how the findings can be generalized to other forms of structured training.
questions: questions: Can you explain why the prequential evaluation setting is relevant, and why lower loss in prequential evaluation implies potential of structured training?

Can you explain why the particular toy model is chosen in the theoretical analysis? Is there an intuitive explanation for the loss function in the toy model?
limitations: limitations: The authors adequately addressed the limitations.

Comments: title: Increasing my score
Followup questions
comment: We thank the reviewer for the additional valuable feedback. We will address these comments in the revision.
I trust that you will address these limitations in your next revision and will increase my score accordingly.
I took some time to go through response to my review and the reviews by the other reviewers. I really appreciate the thorough response to my concerns and particularly value the new experiments analyzing the cosine decay learning rate schedule. I feel that this addition has improved the paper and made me even more confident about its relevance to the community. I thought a lot about it and decided to increase my score to an 8. Looking through the rating descriptions, the novel ideas aspect stood out to me.
We thank the reviewer for their response. We fully agree that cyclic training is still artificial and that it will be important to study the anticipatory recovery phenomenon in naturalistic task sequences that repeat only imperfectly or stochastically. We have made initial steps in that direction in this submission, including data randomization (Figure 4, lines 148-154), partial random shuffling (Figure 12, lines 602-608), and rebuttal experiment E3, which demonstrate that the anticipatory recovery phenomenon indeed generalizes to partially or stochastically repeating task sequences. The main focus of this paper, aside from identifying this surprising phenomenon, has been to understand its underlying mechanisms and what properties of models and learning algorithms it depends on. We hope you agree we have made good progress in this direction and that our findings will be valuable in guiding the next steps with naturalistic task sequences.

The two questions at the end of your comment nicely point to two messages that were missing from our submission but that we will add to the revision.
> Could this phenomenon offer deeper insights into model generalization, particularly in scenarios with subtle cyclic patterns?

Our work suggests that the field of continual learning should pay more attention to naturalistic task sequences in which tasks interleave in statistically regular patterns. 
> How might your findings be leveraged when working with true agents in dynamic environments?

The field might move beyond its focus on learning and retention of novel tasks to examine re-learning performance when old tasks recur. With the anticipatory recovery phenomenon, we identified an important way in which ML models can do surprisingly better than expected on this measure. By identifying the properties of models and learning algorithms that moderate this phenomenon, our experiments provide an important first step toward leveraging it with true agents in realistic dynamic environments.
Thanks for the detailed response and my questions are addressed. I have some minor comments (that do not affect my ratings)
1. I suggest the authors explain the figure 1(b) loss calculation in the revision as well (could be in the appendix).
2. In figure 8, personally I would put the recovery score to the right of the bar, and have a separate legend, this is not crucial.
3. There also exists some work that relates generalization and compression (via prequential coding), which can be mentioned in the related works section, and they are also good references to why prequential coding is relevant, see [1, 2].

[1] Sequential Learning of Neural Networks for Prequential MDL
[2] Continual Learning from the Perspective of Compression
Thank you for your detailed and thoughtful response. I understand the context you provided regarding cyclic training and its relevance to real-world scenarios where tasks repeat in predictable patterns. Your argument about the naturalistic sequence of tasks in day-to-day life is compelling and adds an interesting dimension to the discussion.

However, drawing from my experience with deploying models in environments characterized by continual distribution shifts, I find that cyclic training, as described, may not be widely applicable in practical machine learning setups. While the phenomenon of anticipatory recovery is intriguing, I am still left questioning the broader implications of your findings.

Specifically, what tangible contributions can a machine learning researcher or practitioner derive from this discovery? While the paper successfully highlights an interesting phenomenon in a particular training setup, it seems to stop short of demonstrating how this understanding can be applied in a meaningful way.

I certainly appreciate the value of scientific discovery and advancing our understanding of model behavior, but I would argue that this understanding needs to translate into practical utility, especially when derived from a specialized setup like cyclic training. Can you provide more clarity on how anticipatory behavior could be beneficial in real-world applications? For example, could this phenomenon offer deeper insights into model generalization, particularly in scenarios with subtle cyclic patterns? How might your findings be leveraged when working with true agents in dynamic environments?
Thank you for your response to my questions. While I appreciate the authors' effort put into the work and the response which provides more discussion around the limitations of the work, these fundamental limitation still exists. Therefore, I prefer to keep the current score.