Reawakening knowledge: Anticipatory recovery from catastrophic interference via structured training
Authors: Yanlai Yang
Matt Jones
Michael Curtis Mozer
Mengye Ren
Venue: NeurIPS 2024 poster (2024)
Keywords: large language models
LLMs
catastrophic interference
online learning
continual learning
anticipatory recovery
cyclic training
structured training sequences
Primary area: online_learning
TL;DR: When we fine-tune LLMs cyclically in a fixed repeated sequence of documents the model can recover from catastrophic interference before seeing the same document again.

Abstract: We explore the training dynamics of neural networks in a structured non-IID setting where documents are presented cyclically in a fixed, repeated sequence. Typically, networks suffer from catastrophic interference when training on a sequence of documents; however, we discover a curious and remarkable property of LLMs finetuned sequentially in this setting: they exhibit *anticipatory* behavior, recovering from the forgetting on documents *before* seeing them again. The behavior emerges and becomes more robust as the architecture scales up its number of parameters. Through comprehensive experiments and visualizations, we uncover new insights into training over-parameterized networks in structured environments.

Reawakening knowledge: Anticipatory recovery from catastrophic interference via structured training
Yanlai Yang', Matt Jones?, Michael Mozer and Mengye Ren'
New York University, UUniversity Colorado, Boulder, 'Google Deep.Mind meygeee mengye)@nyu.......m edu, mcj@colorado. edu, mcmozer@google. com
Abstract
We explore the training dynamics of neural networks in structured non-IID setting where documents are presented cyclically in a fixed, repeated sequence. Typically, networks suffer from catastrophic interference when training on sequence of documents; however, we discover a curious and remarkable property of LLMs finetuned sequentially in this setting: they exhibit anticipatory behavior, recov- ering from the forgetting on documents before encountering them again. This behavior occurs even though the documents are never presented in context together. The behavior emerges and becomes more robust as the architecture scales up its number of parameters. Through comprehensive experiments and visualizations, we demonstrate new mechanism by which over- per-paramettried neural networks can recover from catastrophic interference and uncover new insights into training over- peeemaarmettei networks in cyclically structured environments.
Introduction
Large language models (LLMs) 22 4 have demonstrated remarkable general capabilities in wide range natural language tasks. During the training of L.Mss, documents are typically uniformly sampled at random. Due to the large scale of the training set set-in contrast to many other domains-LLL LLM training typically occurs in an online fashion: each document is used only once for just one update step without further repetition [5 6 77.
Such a training style is in stark contrast with how real world agents like humans acquire new knowledge. In naturalistic settings, the material we are exposed to is structured in time and often repeats in predictable, quasi-cyclic patterns (e.g., person's everyday morning routine consists first taking a shower, then eating breakfast, and finally dressing up). Hence it is important to understand how existing deep learning methods and architectures perform in this setting.
Toward the goal of investigating more naturalistic training setups, we study a simplistic setting involving structured training of LLMs: documents presented cyclically in fixed sequence and repeated multiple times, just as we humans go through our daily routines. Moreover, to account for the cost of switching among documents (analogous the mental switching cost between different environments and the waiting cost of obtaining new data), we allow the network to take multiple gradient steps for each document. Compared standard fask-incremental and class-incremental continual learning settings [8] which experience each task only once, our cyclic training setting better approximates the quasi-cyclic temporal structure of real-world environments.
Typically, networks exhibit catastrophic interference (also known as catastrophic forgetting) 9 when training sequence of tasks: the loss given document increases the training advances to other documents. Curiously, we discover that in a structured training environment, LLMs exhibit remarkable anticipatory recovery behavior: they recover from the forgetting of
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a)
(b)
(c)
Shuffl
Shuff Cycli
Cyclc
3.
Crline Loss
N
Training Epoch
Training Epocn
Training Epoch
Figure 1: (a) Loss curves on document for cyclic and random shuffled fine-tuning on a pre-trained Pythia- model. The black circles indicate points just prior to training on the focal document. The inverted-U loss curves within each epoch demonstrate the anticipatory recovery phenomenon. (b) Shift-averaged loss curve for cyclic fine-tuning (c) Online loss curves for cyclic and random shuffled fine-tuning with prequential evaluation.
Model Size
410M
TB
1.4B
2.8B
Cyclic 1.09
0.03
1.03 1. 0.03
1.14 1 0.06
1.34
0.06
Shuffle
1.34 + 0.02
1.51 + 0.04
1.51
+ 0.04
1.79 + 0.06
Table 1: Average online loss across epochs 2 to for cyclic line-tuning and random shuffled fine-tuning.
one document before secing it again, multiple steps in the sequence prior to the recurrence of the document (see Figure [aa and and [b)) It is analogous taking a morning shower, but leaving the thought aside for the rest the day. Critically, we never present two documents together in context, so the model cannot directly learn sequential relationships between them. Thus, our finding is surprising as there is no explicit memory in LLMs that stores sequential knowledge across context windows, and there is no systematic overlap of content across documents- the behavior emerges from a random document sequence after repeated exposure to that sequence. Furthermore, fixed ordering achieves superior performance than random shuffling in the prequential evaluation [IO] setting (see Figure Te) and Ta.le For agent that continuously acts and learns in the real world, the performance on the upcoming task is what matters, and prequential evaluation measures such performance. This result hints at the practical benefits of structured training.
person anticipating to eat breakfast while
demonstrate that, result of anticipatory recovery, training with
Through extensive experiments, we study how different factors in model architecture and training contribute to the anticipatory recovery phenomenon (Section 3.3). We show that only large-scale networks exhibit this reawakening of knowledge, and smaller ones exhibit no such behavior (Sec- tion 3.2) We also show that this phenomenon is not unique to LLMs; some vision models with sufficient width and depth also demonstrate a similar behavior, but LLMs on language modeling tasks exhibit the strongest recovery (Section 3.4). We offer insights the training dynamics in sequentially and eyclically structured input data and propose hypotheses for the causes of the behavior (Section 4h.
Data and Experiment Setup
In this section, we describe the models, datasets, and training setups that we use in the subsequent experiments. Additional details are presented in Appendix A
Models. For the LLM experiments, we use Pythia [1. suite of decoder-only autoregressive
language models pre-trained on the Pile dataset [12 1. We use pre-trained Pythia models ranging
from 160M to 2.8B parameters. For the vision experiments, we use pre-trained Image GPT models for causal image modeling and pre-trained vision transformer (ViT) [5 and VGG-19 [6 models for image classification.
Datasets. For the LLM experiments, we use the CNN/Daily Mail news dataset 17 We repurpose for causal language modeling by discarding the summaries and only using the articles. Importantly, the CNN/Daily Mail dataset is not part of the Pile dataset and hence it is new domain for the Pythia pre-trained models. We use the same documents for both training and evaluation. Our goal here is not to determine whether a trained model generalizes to new documents, but rather to study the memory for a particular document as a function of position within the training history. For the vision experiments, images sampled from CIFAR-10 [18 and ImageNet [19]

---

(8)
(b)
150M
1100
Loos
11.0
18
40
2.440
+2 am
0.00
0:35
0.50
0:75
1.00
0.0
01
0.2
0.3
Traning Epoch
Recovery Score
Training Epoch
Recovery Score
Figure 2:
Effect of model size for (a) pre-trained models and (b) random initializations. each subfigure, the left shows shift-averaged loss curves and the right shows the recovery score as function of model size.
Training Setup. We randomly sample T documents from the dataset. In pre- precpesccesing.. we truncate each document to the first C' tokens (we refer to C' as "context length" in subsequent text). We then fine-tune the LLM on each pre-processed sample for M gradient steps (i.e., using a batch size of l). We refer to the multiple gradient updates of each document as "episode". After each episode we evaluate the loss on all T documents. We repeat the training process for epochs, where epoch consists of one episode of each document in a fixed sequence. We use vanilla gradient descent optimizer. Unless otherwise stated, the default hyperparameters in the subsequent experiments
25, C 256, M 10, F 5. We use the average cross entropy loss (average negative log-like ligliikl for each token) as our training and evaluation metric.
3 Emergent Anticipatory Recovery
In this section, we present our experimental results that reveal the anticipatory recovery phenomenon in eyclic fine-tuning of large language models. We then demonstrate that anticipatory recovery is an emergent behavior that appears only for models with sufficient capacity.
3.1 The Anticipatory Recovery Phenomenon
In this first experiment, we have T 100 documents, and we do cyclic fine- fine-tuning of pre-trained Pythia- IB model [TT] on the documents for E' = 5 epochs in the same ordering. Both the documents and the ordering process. We refer to these T documents as æ1, ... xT. At the start, we fine-tune on ø1 for M = 10 gradient steps, leading to significant decrease in the model's 's loss ø1. As and fine-tune on other documents, we naturally observe catastrophic interference: the model's loss on 21 gradually increases until finish fine-tuning all other documents and return to 21. As iterate through the same document sequence for second time, we would normally expect the loss on 21 to increase monotonically after the initial decrease. However, Figure T a) shows that the loss 1 peaks around Zoo and then starts to decrease. Before we return to 11, the model has recovered more than half of its initial forgetting during the second epoch. We refer this counterintuitive decrease in loss as the anticipatory recovery phenomenon. In Figure [bb, we plot the losses for all the documents and re-align them so that on the x-axis refers the loss on each document immediately before training on it for the first time. The figure confirms that the anticipatory recovery phenomenon exists for not only In but all documents. On the other hand, when we randomly shuffle the document order within each epoch (except is always the first document), we do not observe such anticipatory recovery behavior, and the loss on æ; keeps increasing before we return to it every time.
sampled random beforehand, but kept fixed during the sequential fine-tuning
away from æ1
To quantify the strength the anticipatory recovery phenomenon, we define the recovery score as the proportion of the initial forgetting during the current epoch that the model recovers before returning to the same document. Mathematically, let the mean (over t) of the maximum loss on each document æ, between the th and (n 1)th time train on that document be 4max (n). right before the (n + +1)th time we train on it be Isetore (n), and right after the (n + 1)th time we train on it be (n)) Then define the recovery score (RS) for epoch be RS(n)
In the following Tras ( -o
!nsx (n) (efue(()
'In some cases, a randomly initialized model will produce loss curves that decrease throughout epoch, because its knowledge is so poor that it enjoys positive generalization among all documents. This yields a misleadingly large recovery score under this definition. We do not include such cases in our experiments so do not bother with more nuanced recovery scores.
3

---

(a]
(b)
(8)
(b)
Loss
Los
Loss
Training Epoch
Training Epoch
Training Epoch
Training Epoch
Figure 3: Models trained from scratch with (a) dif-
Figure 4: Effect of data randomization strength. (a) Random masking with probability up to 0.3; (b) Ran- dom shift of context window up 128 tokens.
ferent width (token embedding size) and (b) different depth (number of transformer blocks).
(a)
(b)
100
10
200
20
Training Epoch
Recovery Score
Training Epoch
Recovery Score
(c)
(d)
120
236
Loss
512
J12
000
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Training Epoch
Recovery Score
Training Epoch
Recovery Score
Figure 5: Effects of (a) number of documents (b) number of gradient steps (c) context length and (d) number of frozen blocks.
subsections we compute the recovery scores for different model sizes and training hyperparameters (Figures) 5) to investigate their effects the anticipatory recovery phenomenon.
3.2 Anticipatory Recovery is an Emergent Behavior
To study how the model size affects the amount of anticipatory recovery, we repeat this experiment with pre-trained Pythia models [I]] of sizes 160M, 410M, 1 1.4B, and 2.8B. We plot the average loss curves as well as the recovery score for epoch
Figure We observe that larger models clearly demonstrate stronger anticipatory recovery. The sharp increase of average recovery score from the 160M model to the 410M model indicates that anticipatory recovery is an emergent behavior.
Anticipatory Recovery in Randomly Initialized Models. To study whether anticipatory recovery a result of pre-training, and plot the loss curves and average recovery scores in Figure b)) We follow the model initialization recipe of [1T. From the loss curves for the 410M and IB models, especially in the last epoch, see that the anticipation phenomenon also exists randomly initialized LLMs. We observe that the anticipation effect is not as strong in the pre-trained models. The effect of model size still holds: larger models clearly demonstrate stronger recovery.
repeat the experiments randomly initialized models of different sizes,
Effects of Model Width and Depth. To further study the effect of model width and depth on the anticipatory recovery phenomenon beyond the model hyperparameters the Pythia suite, we take Pythia- 1B model and vary the width (size of token embedding) and depth (number of transformer blocks) of the model and plot the average loss curves cyclic training from random initializations in Figure The original Pythia-1B 1B model has token embedding of size 2048 and 16 transformer blocks. We observe that the model needs sufficient width (at least 512) and depth (at least transformer blocks) to exhibit noticeable recovery, confirming that is emergent behavior contingent model size.

---

3.3 Other Influential Factors
In this section we discuss the effect of other training hyperparameters on the anticipatory recovery phenomenon. We also include additional experiment details in Appendix A. and additional results in Appendix B
plots the loss curves for different number of documents (T' {10, 25, 50, 100, 2003). We observe clear recovery for all the curves, suggesting that the model can "memorize" a certain task transition even after training on 200 other tasks.
Number of Tasks. Figure 5 a(a)
Number of Gradient Steps. Figure (b)) plots training curves with different numbers of gradient steps taken on each document (M E {1.5, 5. 10, 20}). More gradient steps in general leads to a higher recovery score, although in Appendix B.2 and B.8 we show that slight anticipation is still observed for gradient step if we use a larger learning rate, and that the anticipation effect is stronger when the same total update is divided among more gradient steps by scaling the learning rate inversely with M.
Context Length. Figure Sic) plots the loss curves for different context lengths (C E {128, 256, 512, 1024)). Documents are padded to the same length with padding tokens if they are shorter than the specified context length. With the same number of gradient steps, larger context length is correlated with lower recovery score. This suggests that sufficient training on each task is necessary, and for longer input context it takes more gradient descent steps to memorize the task.
Number of Frozen Blocks. We experimented with freezing the first B E [4, 6, 8, 10, 12) trans- former blocks in the pre-trained Pythia- IB model and tune only the last 16 B blocks. Loss curves are plotted in Figure dd) More frozen transformer blocks is correlated with lower recovery score. This observation is consistent with section 3.2 and confirm that the model needs sufficient depth to exhibit anticipatory recovery even with frozen pre-trained deep representation.
Optimizer. In addition to the gradient descent optimizer, we experimented with the Adam opti- mizer (2)] Loss curves are plotted in Figure [ We reset the optimizer state for each document. Results show that Adam, which stronger optimizer, further facilitates anticipatory recovery for both randomly initialized and pre-trained models.
Data Randomness. In realistic sequential learning setting the data points might be slightly different for each repetition (e.g. different descriptions of the same concept, different perspectives of the same object), Icading to stochasticity in the optimization process. To explore sequential cyclic training with data randomness, we design the following two training settings: (1) we randomly mask a subset of the tokens in the input, and (2) we randomly shift the "window" of C tokens used for training. The resulting loss curves are plotted in Figure 4 We observe that, while anticipatory recovery is generally weaker when there more variation in each data point, the recovery still clearly exists.
Summary. The experiment results in this subsection suggest that the model's ability to fit on each task is crucial for the strength of anticipatory recovery. With larger number of gradient steps, shorter context length, more learnable layers, and a better optimizer, the model is more capable of fitting to the focal task, and those factors also correlate with larger recovery score. We also confirmed that anticipatory recovery exists for long task sequences and slightly augmented data points within each episode, and again these factors that make learning harder also reduce anticipatory recovery.
3.4 Anticipatory Recovery in Vision Models
To examine the generality of the anticipatory recovery phenomenon, in this subsection we explore the sequential cyclic training setting on two tasks in computer vision: causal image modeling and image classification. More detailed experiment setup is available in Appendix A
Causal Image Modeling. Similar to the LLM experiments, we fine-tune pre-trained Image GPT model [14] on each sampled image from CIFAR-I0 18] for M gradient steps, and repeat epochs with a fixed order of the images. The resulting loss curves are shown in Figure a The results show that the anticipatory recovery phenomenon also exists for sequential cyclic training of image modeling in addition to language modeling.
5

---

(a)
(b)
(a)
(b)
VGGG-2
Losa
Lets
Taining Epoch
Training Epoch
Training Epoch
Training Epoch
Figure 6: Comparison between Adam and vanilla gra- dient descent on (a) randomly initialized and (h) pre- trained Pythia-IB models with cyclic training.
Figure 7: Results for cyclic training (a) Causal image modeling with Image GPT. (b) image classifi- cation with vision transformers and VGG networks.
(0)
(b)
(c)
(d)
10
4.1
92
40
40
00
4.0
20
20-
08
0.
-25
Figure Heat map visualizations (a) cosine similarities between gradient vectors the attention layer transformer block 12 of model for each task; (b) loss recoveries training task a:; (y-axis) and evaluating on task æy (x-axis); (c) cosine similarities between the flattened model weight residuals at each point in training: (d) cosine similarities between the last layer activations for document at each point in training.
Image Classification. For each experiment, we randomly sample 800 images from Imagenet 19 and divide them into 25 batches of 32 images each. We fine-tune pre-trained vision transformer (ViT) [15] and VGG-19 [16 model on each batch for M gradient steps and repeat E epochs with fixed order of the batches. The resulting loss curves plotted in Figure (b). Results show that both the transformer ViT and the convolutional VGG exhibit anticipatory recovery in cyclic training.
By these experiments confirm that anticipatory recovery occurs not only for LLMs but also for at least some of the widespread image classification models and con-transformer architectures.
3.5 Online Loss Evaluation
We compare the performance of training with fixed ordering and random shuffling of each epoch in the prequential evaluation setting with T= 100 documents. Prequential evaluation [10] measures online performance by evaluating the model equivalent to evaluating the training loss of each batch. For the random shuffling condition, document 1 is always trained first in each epoch, and the other 99 documents
the document it is about be trained on, which is
presented in different random order in each epoch. In this experiment we set C 512, M = 10, E and the error bars are based on 10 different seeds.
The resulting loss curves are plotted in Figure c), and the average loss throughout each run is summarized in Table We exclude epoch 1 when computing the average loss since all documents are new to the model for both the cyclic training and random shuffling condition. We observe that training with fixed ordering is superior random shuffling in the prequential evaluation setting across all 4 pre-trained Pythia models of different sizes, due to the structure in the data stream. The results suggest practical potential of structured training.
4 Understanding Cyclic Training Dynamics
important general question about anticipatory recovery is whether it is due some causal mechanism relating the dynamics of model parameters to the training sequence, whether it is more correlational that adjacent tasks come to represented more similarly by the model. We found ini- tial evidence for the latter hypothesis in experiments locally reversing the task sequence (e.g., showing that # primes æ, nearly as much as vice versa). further test this learned similarity hypothesis, we explore the relationships between tasks and the model's loss gradients, weights, and activations across

---

epoch epoch
epoch
Loss for Tosk
1 18
15
Training Episode
Training Episode
(a) f((wn) = w
(b) f: (w) = w
Figure 9: Top three PCA compo- nents of last layer weights in the first. three epochs.
Figure 10: Loss curve for task I in computational toy model, with
different fi- More experiment details in Appendix A.5
-30 -25
20 23
3.0
0.0 2.5
3.0
300
-2.
05 3.
3.0
0.0 25
30
30 -23 23
30
Figure 11: Visualization of PCA embeddings of the projected data points (f, (Pa;), where fi(w) y/ in the toy model throughout training. Epoch 0 refers to the model before any training.
w)
training history. The results enable us better understand the dynamics of cyclic training. Unless otherwise stated, all visualizations in this section use the 410M model and default hyperparameters.
4.1 Temporal Structure of Gradients
We first explore the similarities of gradient information between documents during the training process. Our goal is to test the hypothesis that anticipatory recovery is mediated by increased similarity between gradients of proximal documents in our training sequence.
We do cyclic training for 4 epochs and compute the gradient of each document the attention layer of transformer block 12 at the conclusion of training. In Figure B[a) ()), we plot the cosine similarities between these gradient vectors of each document. Results show that the gradients have mostly positive cosine similarities (except for the last document, on which the model has just been trained). To our sur- prise, the gradient similarities are highest near the center of the heat map rather than peaking along the diagonal. That is, the gradient similarity between documents -;-1 and 2; depends on where we are in the cycle. This result suggests an additional layer to the anticipatory recovery phenomenon: Recovery for document æ; is greatest from training on document 2r-1, but the strength of the potential facilita- tion between - and æ, actually greatest after we train for another documents (for some small number b). We verify this by computing the pairwise recovery: we take the model checkpoint after 4 epochs of cyclic training, and then for each pair documents (z,, xyy) we do M gradient updates on 2; and compute the difference in the loss of σ; before and after these gradient updates. We plot these pairwise loss recoveries in Figure 8 b). Results confirm that the amount of recovery on document æ; highest when the model checkpoint is taken from roughly b documents before after document a, in cyclic training and then fine-tuned on a proximal document æ; in the sequence. The fact that this pairwise loss recovery matrix is roughly symmetric also suggests that the anticipatory recovery phenomenon approximately exhibits task symmetry: gradient updates on document æ; decrease the loss for document Ør+& for small integers k, and vice versa. We provide additional visualizations for T = 50, 100, 200 in Appendix C and a more detailed description for this phenomenon.
4.2 Temporal Structure of Model Weights
We explore the structure of model weights along the optimization trajectory of cyclic training. We flatten and concatenate the model weight vectors after fine-tuning on each document. However, the cosine similarities between these raw model weight vectors are all very close to without obvious structure, due to the proximity of model weights along the same optimization trajectory and numerical instability in huge weight vectors. To resolve these issues, instead explore the
7

---

structure of "weight residuals." We compute the weight residuals by subtracting the average of weights in a window of length T centered at the current document from the current weight, i.e. (te(() - w(t) - mn= w(n). This removes the shared components along the optimization trajectory and allows us to focus on the model weight updates for each document. Figure ( visualizes heat map of the cosine similarity between each pair of weight residuals from the second epoch to the fourth epoch. The visualization shows cyclic structure in the weight residuals, as equidistant bright stripes that align with the training epochs. Furthermore, each stripe spans several documents, suggesting the similarity of weight residuals for proximal documents.
In addition to cosine similarities between weights, we explore visualizing the weights in a lower- dimensional space with Principle Component Analysis (PCA). We compute the top three PCs of the flattened last-layer weight vector (the output word embedding layer) for the Pythia- IB model, and The plot exhibits a clear helical structure that gradually converges. We believe this is highly relevant to anticipatory recovery: right before revisiting a task, the projected model weights in the helix move closer to the point corresponding to the previous appearance of that task, leading to anticipatory recovery on the loss of that task. As we go on with cyclic training. the model also exhibits less forgetting and gradually converges to solution that achieves low loss on all tasks. It is important to note that the helical structure of the weight trajectory is not an obviously necessary consequence of the cyclical training. Cyclical training could be expected to yield a repeating pattern, but the facts that the tasks come to be organized a circle that respects their ordering and that the trajectory goes through one full revolution per epoch (rather than some other are length) are nontrivial and seem to be essential for anticipatory recovery.
plot its trajectory in Figure ?
4.3 Temporal Structure of Activations
In addition gradients and weights, we visualize the trajectory of activations a single document during the course of cyclic training. We do cyclic training for three epochs and save model checkpoints after fine-tuning on each document. We then compute the model activations before the output word embedding layer for document on each model checkpoint, and plot the cosine similarities between the flattened activation vectors in Figure d(.. From the plot we can clearly observe the blocked pattern wherein the similarity between the activations become progressively higher across each epoch of cyclic training. This pattern suggests that every time we train on document æ, the internal representation of x; in the model is more resistant to gradient updates on other documents Z.
4.4 Computational Toy Model
To further understand the essential mechanism that yields anticipatory recovery, we design a minimalist "toy" simulation experiment. In this toy simulation, each task (formerly, document) i E {L.... ,T} is described by a single data point, æι, TT E IR,. We assume a learnable linear embedding P E MxN that projects each æ; into an M -ddimensional embedding space. We also assume a learnable vector w and task musk-specific mappings fú, where f.(w) is the target for task i in the same embedding space. We require each f. to be invertible
simplifying assumption.
We define the loss for task as (P, w) = 1 || Px. - [i(w)) I|2. Just as when training a deep net, we assume here that representation learning occurs slowly, and that one training step for task involves single gradient update of P with step size ar:
P-aaa(PPy - [Z)
In contrast, at each training step, w, analogous to the fast-adapting weights in neural network, can be rapidly tuned solve for task yielding the loss minimizer conditional on P:
w + f f
(2)
As in our main experiments, sequentially optimize each l, as we iterate through the sequence of tasks. In each training step, we first update P and then solve for w given the update to P. Updating P approximately reduces the distance between Рx4+1 and ff+ 1 (Pxx)). which entails reducing the upper bound of [ (Px,41) - f: (Pxx))] assuming that each f. is Lipschitz continuous. As result of this optimization objective, the model will evolve along the optimization trajectory such that the f. f (Pxx)) for all tasks gradually form circular pattern. This gives intuitive explanation
8

---

on the anticipatory recovery phenomenon, since updaing w according to equation 2 will also bring it closer to f (P thus reducing the loss on task i + 1 and exhibits anticipatory recovery.
We experimented with two very simple choices of f:: f((w) = w and f((w) = y: w for some desk-dependent targets yt- We follow the same order over tasks masks-1,..... , -T-for multiple epochs of training. The resulting loss curves are shown in Figure [0, which exhibits very similar anticipatory recovery trajectory as the full-blown LLM experiment. Visualizations of the 2-dimensional PCA embeddings for f; (Px;) in the second experiment are shown in Figure 11 which confirms our analysis that they gradually self-organize into a cyclic structure.
There are two potential reasons large overparameterized networks might produce the anticipatory recovery in way analogous to the toy simulation. First, for larger networks, it is more likely that the network can develop lask-specific parameters that quickly adapt to and memorize new input data, corresponding
Equation 2 And when the fast memorization achieved, the gradient descent dynamics of the slow weights push the representations of the two adjacent tasks (Pz; and P0.+1) closer when is identity function, according Equation [ This effect be seen in earlier LLM experiments (Figure 2). where larger models achieve significantly lower losses within a few gradient update steps. Second, larger networks have more learning capacity to map the features of two adjacent tasks closer. In our linear projection model, anticipatory recovery keeps growing over many epochs, whereas the anticipatory effect is already at the strongest within 2 or 3 epochs in LLM experiments. Moreover, all data points are randomly generated in the toy model, which makes it easier to separate and map their representations according to temporal structure than real-world data. In contrast, real-world data could require more representation capacity since data points are noisy and correlated.
4.5 Summary
In this section, we visualized model weight dynamics with heatmaps and we showed model activations and gradients during cyclic training. We discussed the special temporal structure that is exhibited in these heat maps. We also plotted the pairwise degree of recovery for fine-tuning on document and evaluating on document j. as well as the change of distance between fine-tuned model weights on different tasks. The results suggest that after we train on a document, the model's representation of that document becomes less sensitive to gradient updates on other documents. Finally, we showed simple toy experiment that demonstrates a similar anticipatory recovery phenomenon its loss curve, and discuss its connections to neural network training dynamics through the lens of task-specific and task-general parameters. Overall, these results shed some light on the dynamics of cyclic training.
5 Related Work
In this section we discuss the most relevant prior works to this paper. Please refer to Appendix D for additional related work.
Cyclic and Structured Training. Prior theoretical works have studied convergence rates, under various assumptions, for the training setup where the data points are shuffled only once and that order is reused for all epochs [21] 22 23 24). On the empirical side, (25) found that shuffling the data only once in the beginning can achieve convergence rate comparable to shuffling every epoch. The training setup is equivalent to our cyclic training setup, but our research examines the loss on each task throughout the training cycle and discovers the anticipatory recovery effect. We also extend it to multiple gradient update steps on each data point.
Online Learning. Online learning deals with the setting where the tasks come from an online sequential stream. One of the simplest algorithms in online learning is follow-the-laade [26). which stores all previous data from the stream and minimizes the total loss. It has strong performance guarantees but is computationally very expensive, and it also might not be feasible to store all the past data. Many subsequent works have developed cheaper algorithms under different assumptions 27 28 29.. Many recent works also explore the connection between online learning and meta- meta-learning or continual learning 30 31
36 The cyclic training setting that we explore this research can be considered as special case of the online learning setting where the data stream has a cyclic repetition structure. We employ multiple steps of online gradient descent [77 on each document from the stream and study the training dynamics of over- over--vraaenn peee-parattterieed neural networks.

---

Catastrophic Interference. When transitioning between tasks sequentially. neural networks often experience "catastrophic interference" [9) marked by significant drop performance on previously learned tasks. Numerous algorithms have been proposed to mitigate catastrophic forgetting. focusing on general approaches including parameter regularization 38 39 40, data replay knowledge distillation 44
42 43
45
46, 47), and architectural isolation and expansion [48, 49 50, 51].
Our work extends interleaved training [52] to a larger number of tasks, specifically investigating the emergent anticipatory recovery phenomenon in cyclic training. This finding adds to the above literature by demonstrating a new mechanism by which large networks can avoid or recover from catastrophic interference.
Continual Learning Continual learning 8 53 54 55] addresses a simplified setup where a model sequentially learns a set of tasks without revision. Recently, there have been debates over the practicality of continual learning setups. Studies like (56) have shown that as networks learn more tasks, they improve in learning speed and reduce forgetting. In large models, studies suggest that pre-trained vision classifiers can undertake continual learning with ease, by either freezing or fine-tuning representations [87] 59). In the language domain, research also suggests that LLMs exhibit emerging continual learning capabilities [60] 61]. Nevertheless, it is uncommon in real environments for tasks to occur only once yet for agent need to retain them. Unlike prior literature on continual learning, our research uniquely focuses on sequential learning environments with cyclic repetition.
6 Discussion and Limitations
In this work, explored the training dynamics of overparametrized neural networks, especially LLMs, in sequential cyclic fine-tuning, where a finite set of documents are presented in the same order within each epoch. We demonstrated the remarkable phenomenon of anticipatory nettors recover from the initial forgetting before seeing the same document again. The effect holds across many different network instances and training hyperparameters. This phenomenon is a sharp contrast with the well known phenomenon of catastrophic interference, where forgetting increases monotonically as a network trained on a sequence of different documents.
We showed that anticipatory recovery occurs only when the network has sufficient width and depth and when it is well fitted to each document before moving the next. Visualizations of model weights, model activations, and gradients exhibit clear temporal structure, which provide insights on the underlying mechanisms of anticipatory recovery.
Our research indicates that there is value in exploring naturalistic task sequences within continual learning, where tasks interleave in statistically regular patterns. This approach could expand the field's s current focus on learning and retaining new tasks to also consider how effectively previously encountered tasks re-learned when they reappear. With the anticipatory recovery phenomenon, discovered mechanism in which ML models can do surprisingly better than expected on prequential evaluation. By analyzing the different factors of model pre-training and fine-tuning that moderate this phenomenon, our experiments provide promising first step toward leveraging structured training with agents in realistic environments.
Limitations. The cyclic training setup investigated in this work is distinct from the IID training set- ting assumed in the vast majority of the machine learning literature. It accounts for task repetition and task switching costs, which are critical components of the learning experience of humans and other real world agents. However, our current setup is still highly simplified. Future research could investi- gate the emerging training dynamics of neural networks in different types of structured environments, such multiscale temporal dynamics [62]. from both theoretical and empirical perspectives.
The mathematical foundation of anticipatory recovery also requires further investigation. Although computational toy model reproduces the anticipatory recovery phenomenon, it does not explain why the effect is stronger in LLMs and autoregressive tasks than in other types of architecture or learning objectives. In terms of the empirical experiments, the ablation studies only in single setting. Future research could run the experiments in more settings to reach more conclusive results and investigate possible alternative theoretical explanations the anticipatory recovery phenomenon.
10

---

Acknowledgment
We thank the Microsoft Accelerating Foundation Models Research program for providing Azure cloud compute credits. We thank members of the NYU Agentie Learning Al Lab for helpful discussions. The compute was supported by the NYU High Performance Computing resources, services, and staff expertise. MJ was supported by NSF grant 2020-906.
References
[1] Jacob Devlin, Ming-Wes Weig-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Conference of the North Ameri- can Chapter of the Association for Computational Linguisties: Human Language Technologies. 2019.
[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33: 33:1877-1901, 2020.
[3] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint 3rX11 2023.
[4] OpenAL. GPT-4 technical report, 2023.
[5] Jordan Hoffmann. Scbastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 2022.
[6] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1- 113, 2023.
[7] Fuzhao Xue, Yao Fu, Wangchunshu Zhou, Zangwei Zheng, and Yang You. To repeat not to repeat: Insights from scaling Ilm under token-crisis. Advances in Neural Information Processing Systems, 36, 2024.
[8] Zhiyuan Chen and Bing Liu. Lifelong Machine Learning, volume 1. Springer, 2018
[9] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of Learning and Motivation, volume 24, pages 109-165. Elsevier, 1989.
[10] Zhipeng Cai, Ozan Sener, and Vladlen Koltun. Online continual learning with natural distribu- tion shifts: An empirical study with visual data. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8281-8290, 2021.
[11] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference Machine Learning. pages 2397-2433 PMLR, 2023.
[12] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 2020.
[13] Stella Biderman, Kieran Bicheno, and Leo Gao. Datasheet for the Pile. arXiv preprint arXiv:2001.03311 2022.
[14] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, pages 1691-1703. PMLR, 2020.

---

[15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
[16] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
[17] Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Gulçehre, and Bing Xiang. Abstrac- rnss and beyond. In Proceedings of the
tive text summarization using 20th SIGNLL Conference on Computational Natural Language Learning, pages 280 290, 2016.
[18] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. University of Toronto, 2009.
[19] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on Computer Vision and Pattern Recognition, pages 248-255. IEEE, 2009.
[20] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations, 2015.
[21] Kwangjun Ahn, Chulhee Yun, and Suvrit Sra. SGD with shuffling: Optimal rates without component convexity and large epoch requirements. Advances Neural Information Processing 2020.
Systems, 33:11566-::
[22] M Gurbuzbalaban, Asu Ozdaglar, and Pablo Parrilo. Convergence rate of incremental gradient and incremental newton methods. SIAM Journal on Optimization, 29(4):2542-2566 2019.
[23] Konstantin Mishchenko, Ahmed Khaled, and Peter Richtárik. Random reshuffling: Sim- ple analysis with vast improvements. Advances in Neural Information Processing Systems, 33:17309- 17320, 2020.
[24] Itay Safran and Ohad Shamir. How good is SGD with random shuffling? In Conference on Learning Theory, pages 3250-3284. PMLR, 2020.
[25] Lijie Xu, Shuang Qiu, Binhang Yuan, Jiawei Jiang, Cedric Renggli, Shaoduo Gan, Kaan Kara, Guoliang Li, Ji Liu, Wentao Wu, et al. Stochastic gradient descent without full data shuffle. arXiv preprint arXiv:2206.05880 2022.
[26] James Hannan. Approximation to Bayes risk in repeated play. Contributions to the Theory of Games, 3:97-139, 1957.
[27] Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In International Conference on Machine Learning, pages 928-936 2003.
[28] Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games. Cambridge University Press, 2006.
[29] Shai Shalev-Shwartz al. Online learning and online convex optimization. Foundations and Trends@ in Machine Learning, 4(2):107-194, 2012.
[30] Giulia Denevi, Carlo Ciliberto, Riccardo Grazzi, and Massimiliano Pontil. Gearning-to-lean stochastic gradient descent with biased regularization. In International Conference on Machine Leurning, pages 1566-1575. PMLR, 2019.
[31] Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta-learning. In International Conference on Machine Learning, pages 1920-19300 PMLR, 2019.
[32] Giulia Denevi, Dimitris Stamos, Carlo Ciliberto, and Massimiliano Pontil. Online-within-nnlin meta-learning. Advances in Neural Information Processing Systems, 32, 2019.
[33] Khurram Javed and Martha White. Meta-learning representations for continual learning. Ad- vances in Neural Information Processing Systems, 32, 2019.
12

---

[34] Enrico Fini, Stéphane Lathuiliere, Enver Sangineto. Moin Nabi, and Elisa Ricci. Online contin- ual learning under extreme memory constraints. In Proceedings of the European Conference on Computer Vision, pages 720-735. Springer, 2020.
[35] Mengye Ren, Michael Louis luzzolino, Michael Curtis Mozer, and Richard S. Zemel. Wandering within world: Online contextualized few-shot learning. In International Conference on Learning Representationss 2021.
[36] Jianren Wang, Xin Wang, Yue Bhang-GGua, and Abhinav Gupta. Wanderlust: Online continual object detection in the real world. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10829-10838, 2021.
[37] Michael Biehl and Holm Schwarze. Learning by on-line gradient descent. Journal of Physics A: Mathematical and general, 28(3):643, 1995.
[38] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521-33
et al.
2017.
[39] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, pages 3987-3995. PMLR, 2017.
[40] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuyte- laars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference Computer Vision, pages 154, 2018
[41] Sylvestre-Alviss Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. iCaRL: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001-2010, 2017.
[42] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap. and Gregory Wayne. Experi- ence replay for continual learning. Advances in Neural Information Processing Systems, 32, 2019.
[43] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puncet K Dokania, Philip HS Torr, and Marc' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv:
2019.
[44] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprini arXiv: 1,
[45] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions Pattern Analysis and Machine Intelligence, 40(12):2935- 2947, 2017.
[46] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. Advances in Neural Information Processing Systems, 33: 13:10090-55930 2020.
[47] Divyam Madaan, Hongxu Yin, Wonmin Byeon, Jan Kautz, and Pavlo Molchanov. Heteroge- neous continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15985-159999 2023.
[48] Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In International Conference on Learning Representations, 2018.
[49] Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. International Conference on Machine Learning, pages 4548 4557. PMLR, 2018.
[50] Mustafa B Gurbuz and Constantine Dovrolis. NISPA: Neuro-inspired stability- adap- tation for continual learning sparse networks. International Conference on Machine Learning, pages 8157-8174, PMLR, 2022.
13

---

[51] Haeyong Kang, Rusty John Lloyd Mina, Sultan Rizky Hikmawan Madjid, Jachong Yoon, Mark Hasegawa-Johnson, Sung Ju Hwang, and Chang D Yoo. Forget-free continual learning with winning subnetworks. In International Conference on Machine Learning, pages 10734-10750. PMIR, 2022.
[52] David Mayo, Tyler Scott, Mengye Ren, Gamaleldin Elsayed, Katherine Hermann, Matt Jones, and Michael Mozer. Multitask learning via interleaving: A neural network investigation. In M. Goldwater, F. K. Anggoro, B. K. Hayes, and D. C. Ong, editors. Proceedings of the 45th Annual Conference of the Cognitive Science Society, volume 45, 2023.
[53] Andrea Madotto, Zhaojjang Lin, Zhenpeng Zhou, Seungwhan Moon, Paul Crook, Bing Liu, Zhou Yu, Eunjoon Cho, Pascale Fung, and Zhiguang Wang. Continual learning in task- oriented dialogue systems. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7452-7467, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[54] Chengwei Qin and Shafiq R. Joty. LFPT5: A unified framework for lifelong few-shot language learning based on prompt tuning of T5. In International Conference on Learning Representa- tions. OpenReview.net, 2022.
[55] Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, and Amjad Almahairi. Progressive prompts: Continual learning for language models. In International Conference on Learning Representations. OpenReview.net, 2023.
[56] Guy Davidson and Michael C Mozer. Sequential mastery of multiple visual tasks: Networks naturally learn to learn and forget to forget. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pages 9282-9293, 2020.
[57] Paul Janson, Wenxuan Zhang, Rahaf Aljundi, and Mohamed Elhoseiny. A simple baseline that questions the use of pretrained-models in continual learning. In NeurIPS 2022 Workshop Distribution Shifts: Connecting Methods and Applications, 2022.
[58] Kuan- Ying Lee, Yuanyi Zhong, and Yu-Xiong Wang. Do pre-trained models benefit equally in continual learning? In Proceedings of the IEEE/CVF Winter Conference on Applications of 2023.
Computer Vision, pages
[59] Enrico Fini, Victor G Turrisi Da Costa, Xavier Alameda-Pineda, Elisa Ricci, Karteek Alahari, and Julien Mairal. Self-supervised models are continual learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9621 19300, 2022.
[60] Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. Fine-tuned language models are continual learners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6107-6122, 2022.
[61] Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. Continual pre-training of language models. In International Conference on Learning Representations, 2022.
[62] Matt Jones, Tyler R. Scott, Mengye Ren, Gamaleldin Fathy Elsayed, Katherine L. Hermann, David Mayo, and Michael Curtis Mozer. Learning in temporally structured environments. In International Conference on Learning Representations, 2023.
[63] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: off art natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online, October 2020. Association for Computational Linguistics.
[64] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188)99 2019.
14

---

[65] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In Sth International Conference on Learning Representations, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
[66] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017.
[67] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022.
[68] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Advances Neural Information Processing Systems, 35:24824 24837, 2022.
prompting elicits reasoning in large language models.
[69] Deep Ganguli, Danny Hernandez, Liane Lovitt, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova Dassarma, Dawn Drain, Nelson Elhage, et al. Predictability and surprise in large generative models. Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1747-1764, 2022.
[70] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolatingg the capabilities of language models. arXiv preprini arXiv:2206.04615. 2022.
[71] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr, and Chiyuan Zhang. Quantifying memorization across neural language models. In International Conference on Learning Representations, 2023.
[72] A Emin Orhan. Recognition, recall, and retention of few-shot memories in large language models, arXiv preprint arXiv:2303. /7557, 2023.
15

---

A Additional Experiment Details
A.1 LLM Experiments
We use the Huggingface Transformers Library [65] for fine-tuning the LLMs. The learning rate 0.001 for vanilla gradient descent and 0.00001 for Adam. For all experiments different random seeds, except the results in Figure which are based on 20 seeds. The shaded area in the figures denotes standard deviation among trials and documents (for shift-averaged loss curves).
3 to 5 trials with
A.2 Causal Image Modeling Experiments
Models Image GPT [4] is a GPT-2-like model trained to predict the next pixel value in an image. is pre-trained on the Imagenet dataset [9] resized to 32x32. The Image GPT authors provide three pre-trained models of different sizes. In experiments, we use the Image GPT-small and Image GPT-medium models.
Datasets We use the CIFAR-10 18 dataset for fine-tuning. For tokenization, the pixel RGB values are categorized into 512 pre-determined clusters with the nearest-neighbor classifier, as in [14]. After pre-processing, each image is transformed sequence of length 1024, with code book of size 512.
Training Setup We did not manage sequentially fine-tune the model stably with the dropout layers, so the dropout layers are turned off during the Image GPT fine-tuning experiments. We use the Adam optimizer (20] with learning rate 0.001. The default hyperparameters in the experiments are T = 25 images, M = 10 gradient update steps, E epochs. Same as the LLM experiments, use the average cross-entropy loss evaluation metric.
A.3 Image Classification Experiments
The images are resized to 256x256 followed by center crop of 224x224. We use the Adam optimizer with learning rate 0.0001 for M = 10 gradient steps on each batch images.
A.4 Shift-averaged Loss Calculation
The shift-averaged loss curves plotted in Figure I calculated by replicating Figure a each document in the training sequence, re-aligning these curves so that 0 on the x-axis always represents the moment before the first occurrence of the focal document, and average them. For example, if the length of the sequence is 50, then for training epoch 0.5 on the x-axis, we take the loss of document after training document 25; the loss of document after training on document 26: ... the loss of document 50 after training on document 24 of the next epoch; and average these losses. Subsequent are plotted with the same approach.
figures (Figure 7 12 19
A.5 Computational Toy Model
For Figure 9al we pick fi(w) w. and each data point æ; and w is a vector of length N = M 1000. We have = 25 data points and use the vanilla gradient descent optimizer with learning rate 0.01. The projection matrix is initialized with every entry sampled independently from A/, 1/N2). Each entry of the data points 2; and w is sampled independently from Unif(-1, 1). For Figure 9b pick fi(w) y: w, N M 100, T = 25, and learning rate 0.01. Each entry of y; is also sampled independently from 1n1,(-
A.6 Compute Resources
Each experiment presented in the paper run with one NVIDIA A100 GPU, 2 CPUs, and 32GB of RAM. The training time highly depends on the hyperparameter choices, especially model size and number of gradient steps. The longest fine-tuning experiment with 20 gradient steps per episode on the Pythia-1B 1B model takes roughly 30 minutes under this setup. The minimal compute resource requirement needed to reproduce the experiments with Pythia- 1 model one GPU with 16GB of GPU memory.
16

---

B Additional Experiment Results
Pytha init Unioam Scale
Loss
Tranning Epoch
Training Epoch
Training Epach
Training Epoch
Figure 12: Effect of partial document shuffling.
Figure 13: Effect of learn- ing rate in 1-step GD.
Figure 14: Effect num-
Figure 15: Effect model initialization.
ber of attention heads.
24k
Lo33
90
0.00
0:25
0.50
0.75
1.00
1.00
200 300
Triining Epoch
Recovery
Training Epoch
Training Step
Figure 16: (Left) Effect of pre-training steps. The full Figure 17: Loss curve for
Figure 18: Effect insert- ing random documents after the repeating sequence.
pre-training process is 143k steps. (Right) Recovery cosine learning rate sched- scores for models with different pre-training steps.
ule.
B.1 Partial Random Shuffling
Throughout the paper we have been focusing on the setting where the document ordering is sampled once and stay fixed for all epochs. What if we only fix the first and the last document, and shuffle the documents in between? We experimented with shuffling the documents from document æ2 through NN for E {8, 16, 24) every epoch. In Figure we plot the loss curves for document æ1. From the loss curves we can observe that even when N 24 we can still observe some anticipatory recovery, suggesting that the order the tasks between two consecutive repetitions of the 2255 and æΙ can be arbitrary for to observe recovery æ1.
B.2 One-step Gradient Descent with Larger Learning Rate
In Figure b) we observe that there is no anticipation when we take only one gradient descent step each document with learning rate 0.001. We experimented with one-step gradient descent using higher learning rate, 0.01. We plot the resulting average loss curves under the same training setup in Figure [3 We observe that, with larger learning rate, slight anticipation is still observed for 1 gradient step.
B.3 Effect of Number of Attention Heads
also experimented with varying the number of attention heads E {2, 4, 8, 16] while keeping model width to be 2048 and model depth be 16. Loss curves on document æ1 are shown in Figure [4 The results suggest that the number of attention heads does not have big effect on cyclic training in our setting.
In addition varying the model width and model depth in Figure 3
B.4 Effect of LLM Model Initialization
Here we compare the performance of the initialization scheme used by (also used for all randomly initialized models in the main text) and simple initialization scheme that samples the weight matrices from an isotropic Gaussian distribution with a 0.02. The loss curves for document 1 under these two initializations of the Pythia-11 B model are plotted in Figure 15 We observe that Pythia's s initialization scheme achieves much better average loss and also exhibits stronger anticipatory recovery. This demonstrates the importance of LLM initializations. The result is consistent with
17

---

observations in section 3.3 that the model's ability to fit on each task is correlated with the amount of anticipatory recovery.
B.5 Effect of Pre-training Steps
In addition to comparing pre-trained models and randomly initialized models in Figure we further study the effect of model pre-training by examining model checkpoints with different numbers of pre- training steps. We took Pythia models pre-trained for 6K, 12K, 24K, 48K, and 96K steps and plot the shift-aver averaged loss curves for cyclic fine-tuning in Figure T6] We found that more pre-traiie pre-training does give rise to higher anticipatory recovery. As we summarize at the end of section 3.3] we hypothesize this result fits broader pattern which the strength of the anticipatory recovery effect related to how well the model can fit each successive training task. Models with more pre-training steps are more capable of fitting each successive training task, and therefore exhibit higher anticipatory recovery.
B.6 Effect of Cosine Learning Rate Schedule
For experiments in the main paper we used constant learning rate during the fine-tuning process. To study whether anticipatory recovery occurs in typical LLM optimization schemes, we experimented with cosine learning rate scheduling on the Pythia- IB model with 10 epochs (minimum learning rate = 0, maximum number of epochs 10), and plot the results in Figure 17 We show that the model also exhibits the anticipatory recovery effect other learning rate schedules.
B.7 Effect of Inserting Random Documents after the Repeating Sequence
To examine how the anticipatory recovery effect may generalize to other forms of structured training, setting where only the first 20 documents are kept fixed in each epoch, and random number of other documents (between 20 and 100) are inserted after the first 20. These random padding documents appear only once in the entire sequence. This new setting generalizes cyclic training in that (1) rather than having the same documents in every epoch, we insert random other documents between every repetition (2) epochs can have different lengths. The resulting loss curve is plotted in Figure 18, We still observe anticipatory recovery for documents 2 through 20 in this setting, suggesting that anticipatory recovery exists as long as there is repeating sequencee in the data stream.
experimented with
+10
55
Loes
20
100
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Training Epoch
Recovery Score
Training Epoch
Recovery Score
(a) Number of Gradient Steps with Inverse LR Scaling (b) Number of Gradient Steps for Context Length 1024
Figure 19: Effect of number of gradient steps (a) with inverse learning rate scaling and (b) for context length 1024.
B.8 Effect of Number of Gradient Steps with Inverse Learning Rate Scaling
In Figure 19a we experimented with inversely scaling the learning rate with the number of gradient steps. We use learning rate of 0.01 for M 1, learning rate 0.002 for M = 5, learning rate 0.001 for M = 10, and learning rate 0.0005 for M = 20. The results suggest that the anticipation effect is stronger when the same total update is divided among more gradient steps.
B.9 Effect of Number of Gradient Steps for Long Context Length
In Figure 19b we experimented with different number of gradient steps M E (10, 20, 40) context length 1024. The results confirm that longer context length is not fundamental limitation to
18

---

anticipatory recovery, and we can achieve the same recovery score smaller context length with more gradient steps.
1.00
Euclikan Disiace
1202
Euclideean a02
2.
Training Epoch
Training Epoch
Model Activations
(b) Model Weights
Figure 20: Experiments Figure 21: Experiments
with GPT2-large.
with wikitext- 103 dataset.
Figure 22: Magnitude of (a) model activation updates and (b) model weight updates through cyclic training.
B.10 Experiments with GPT-2
To evaluate how the anticipatory recovery phenomenon generalizes across different LLM architectures, we experimented with GPT-2 architecture [64], specifically the GPT2-large pre- pre-trained model (812M parameters) on the CNN/Daily Mail dataset. The loss curve for document The model consistently observed anticipatory recovery. Note that GPT-2 is the predecessor of many modern LLMs and therefore the results further suggest that the anticipatory recovery phenomenon prevalent among more recent LLM architectures.
plotted Figure 20
B.11 Experiments with Wikitext
To evaluate how the anticipatory recovery phenomenon generalizes across different natural language datasets, we experimented with the wikitext- 103 dataset (65), which contains over 100 million tokens from articles on Wikipedia. Since Wikipedia data is part of the pre-training dataset of Pythia, we only experiment with randomly initialized models. The loss curve for document plotted in Figure 21 The result suggest that the anticipatory recovery phenomenon is generalizable different data sources.
C Additional Visualizations
C.1 Magnitude of Changes Model Weights and Model Activations
We plot the magnitude of the difference between the 21 activations of model checkpoints saved at consecutive episodes throughout four epochs of cyclic training of Pythia-410M model in Figure 2 and observe a clear stepwise pattern. In contrast, the magnitude of model weight updates (Figure 21b decreases monotonically over the training episodes and do not exhibit this stepwise pattern. This result consistent with the pattern we observe in section
0.75
10
20
0.50
0.5
0.5
100
0.0
150
40
0.5
-05
10
50
100
150
(a) Documents
(b) 100 Documents
(c) 200 Documents
Figure 23: Loss recoveries for training on task æ, (y-axis) and evaluating on task (x-axis) for longer document sequences of different lengths.
19

---

C.2 Pairwise Recovery Matrices for Longer Document Sequences
Similar to Figure B.b,, we plot the pairwise loss recoveries for each pair of documents (æ, T;) in longer document sequences, where T = 50, 100, 200 respectively, in Figure 23 We use the 1B model and default hyperparameters. We observe that, as we increase the length of the document sequence, the highlight area near the center of the matrix is separated into two blobs, one in the top-left corner and the other in the bottom-right corner. We also observe "boundary" on the sides of the matrix where there is little or no recovery. The width of this "boundary" stays relatively constant across different lengths of document sequences and is around 10 documents. This confirms our observation in the main text that the recovery on document æ, when fine-tuning proximal document æ; is highest when the model checkpoint is taken from document a e where a small number relative to the length of the document sequence.
D Additional Related Work
Learning in Structured Environments. Our research also relates to the more general topic of learning in structured environments. [62] studied regression and classification tasks with multi-scale temporal structure in the environment characterized by 1/f dynamics. While the cyclic training setting that we study is a more simplified setup than that [62], we aim at unveiling more insights applying standard SGD wer-parrameterized networks. A potential direction for future work would be to study anticipatory recovery regimes with richer, hierarchical sequence structure.
LLM Emergent Capabilities. Recent advancements in large-scale Transformer networks 666 have demonstrated exceptional ability to model long sequence language data. Beyond basic language modeling and downstream task performance, these models have shown emergent behaviors [67] that appear to manifest only beyond a certain model scale [2 68 69 77]. Related to our research, recent studies reveal that LLMs possess remarkable memorization skills, enabling them to recall news sentences after just a few exposures [1 72). However, the sequential learning dynamics behind such memorization have not been thoroughly examined. Our work comprchensively explore the sequential learning setting with cyclic task repetition and demonstrates task anticipation, new emergent capability of large models.
E
Broader Impact
This research deepens our understanding of catastrophic interference in naturalistic training setups. This understanding could lead to the design of better training algorithms of LLMs and other large neural networks that are more similar human learning. These algorithms may give rise more powerful and embodied Al systems with online adaptive learning capability, which may have many potential societal consequences.
Recovery from catastrophic interference might be undesirable in scenarios where some documents are intended be forgotten. Our research provide more understanding into the anticipatory recovery phe- nomenon. Future research into the unlearning problem as well as deeper understanding of the train- ing dynamics of different types of structured environments can help further address the privacy issue.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The authors confirm that the claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of this work are discussed in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
21

---

Answer: [NA]
Justification: The paper does not include theoretical results. While we provide a well-defined computational model in section 4.4 are not trying prove new theoretical results and the purpose of section 4.4 is to provide more intuition on how the anticipatory recovery phenomenon might occur.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
we disclose all the information needed to reproduce the main experimental results of the paper. The pre-trained models and datasets we use are also publicly available. We also include the code for reproducing main experimental results in the supplementary material.
Justification: In Sections and Appendices
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility.
22

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the code and instructions for reproducing main experimental results in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
7 //nips. //nips.cc/publll wwtpc/ppppppg..d.....o.pppppppee htte:/guuuddee/oooSubbbieiseee www.eo/umiiiiiiooooooooo
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experimental settings and details are provided in Section and Ap- pendix
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All experiment result figures and tables in Section 3 of the paper accompa- nied by error bars or confidence intervals. Some error bars might not be visible since they are too small.
Guidelines:
23

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provided detailed information the compute resources needed to repro- duce the experiments in Section A.6
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htttpp////aarrrsss..c/pppl//hhhs
Answer: [Yes]
The authors confirm that this paper conforms with the NeurIPS Code of Ethics. We used publicly available, non-deprecated datasets and models, and conform with their licenses.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?

---

Answer: [Yes]
Justification:
The broader impacts of this work are discussed in Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper does not release new data or models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The original papers for all the code, data and models used in the main experi- ments of the paper are cited. All these existing assets involved are publicly available, under the Apache 2.0 License (Pythia, huggingface transformers) or MIT License (cnn/dailymail version 3.0.0).
Justification:
Guidelines:
The answer NA means that the paper does not existing assets.
25

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the datasets..paerssttt paperswi thcode. com/datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
26

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
27


Reviews:
[{"review_id": "sAZcCxkHJX", "reviewer": "Reviewer_dpPf", "summary": "This paper takes a deeper look at the training dynamics of neural networks such as LLMs in a particular kind of non-IID learning setting when tasks or data is iterated through cyclically in a fixed repeated sequence. The authors find evidence in this setting of an emergent behavior that they call \"anticipatory recovery\" where the model improves its loss on data before it actually arises seemingly in anticipation of the upcoming data. This behavior is surprising because the neural network does not contain an explicit memory mechanism for this purpose. The authors run a number of experiments to provide further clarity about the nature of this phenomenon and present a simple toy setting to help explain how this may happen.", "questions": "Q1:  The authors seem to indicate that they always use a vanilla gradient descent optimizer. While it is obviously interesting to consider this case, I wonder whether this analysis is less relevant for typical methods of training LLMs i.e. with Adam optimization and cosine decay learning rate schedules. Do you have any thoughts on how this may impact recovery? \n\nQ2: Do you believe that changing the total amount of data may change the nature of anticipatory recovery? I am also trying to understand the potential relevance to the one pass very large data setting. I guess due to shuffling, recovery is not expected despite redundancy? \n\nQ3: I wonder if the authors see any actionable insights to follow up from this study. Are you actually advocating the cyclic training? Do you see any potential implications for continual learning or curriculum design?", "limitations": "The authors perform a number of experiments in the paper, which they use to draw various conclusions, but the authors could do a better job of noting the limitations in these experiments i.e. individual ablations only being run in single settings (which is understandable but still limits how much we could read into the results) and possible alternative explanations. Conceptually, the authors made the choice to consider a variety of experiments trying to answer a variety of questions rather than asking fewer questions and coming to more conclusive results about the answers to these questions.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main insight of this paper that neural networks of a certain size exhibit anticipatory recovery as an emergent property is a very novel and interesting concept to me. I had never considered that this kind of thing may be happening. As a continual learning researcher, this is definitely something that adds to my mental picture of what may be happening during continual training. There are really a very significant number of experiments in this paper and many of the first questions that would come to mind were answered by the authors. Throughout much of the beginning of the paper I was curious about what the potential causes of this behavior could be, so I really appreciate the analysis in section 4.4. It still is not very clear, but in my mind identifying the problem is a worthwhile contribution for the paper, so I was satisfied with the effort given this context.", "weaknesses": "The writing of this paper could be improved somewhat. Reading through the paper can feel like going along for a journey with the authors with very little idea of where it is headed ahead of time. Typically papers summarize the main findings upfront to a larger extent. A lot of the time it feels like a large set of results are presented with little explanation of their connections and then the authors add in a summary section to recap how everything ties together. It would be better to provide the summary up front so readers know where it is going. \n\nThe authors could also do a better job of contextualizing the potential implications of their work (see questions below). Moreover, the empirical results are somewhat limited by the choice to prioritize breadth in terms of the number of questions asked rather than depth in terms of coming to conclusive answers to these questions by running them over a number of datasets and architectures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed feedback.\n\n**Re: Writing improvement and summary of main findings.**\n> We have a summary of main findings in the final paragraph of section 1 but we agree some elaboration will help readers anticipate what is coming. We thank the reviewer for the writing feedback, and will incorporate these improvements proposed by the reviewers in the next version of the paper.\n\n**Re: Q1: The analysis of Adam optimization and cosine decay learning rate schedules.**\n> We report an experiment comparing Adam to SGD which shows Adam produces a stronger anticipation effect in Figure 6 (lines 145-147) of the paper. We hypothesize this result fits a broader pattern in which the strength of the anticipatory recovery effect is related to how well the model can fit each successive training task (see Section 3.3). In rebuttal experiment E2 we experimented with the cosine learning rate schedule and verified that the model still exhibits strong anticipatory recovery.\n\n**Re: Q2: Total amount of data and the one-pass very large data setting.**\n> We report the effects of the total number of tasks in Figure 5a (lines 128-130) of the paper, and found that the anticipatory recovery effect is more significant with a higher number of documents in the sequence. The one-pass very large data setting should produce no anticipation, similar to what we find when the task order is fully reshuffled on each pass in figure 1a.\n\n**Re: Q3: Actionable insights to follow up from this study.**\n> Our paper challenges the conventional wisdom in the continual learning community on when catastrophic forgetting occurs: we suggest that for certain models, catastrophic forgetting can be largely mitigated when training on structured sequences. As argued in our general rebuttal, natural environments often have predictable transitions, and cyclic training is the most extreme version of this type of sequential structure. We therefore advocate for more research in structured training and rethinking classic continual learning algorithms in structured sequences, and using prequential evaluation as an important metric. We can also study the types of structures that achieve best learning performance and efficiency, which could contribute to designing good learning curricula and active learning methods.\n\n**Re: Limitations.**\n> We thank the reviewer for pointing out these limitations. We will mention these points in the limitations section in the next version of the paper.\nWe also want to point out that, while many of the experiments are conducted on Pythia models and the CNN/Daily Mail dataset to identify the influential factors of the anticipatory recovery effect, we carried out many experiments with other models and datasets as well, including vision models (Figure 7, Section 3.4), the GPT-2-Large model (Figure 17, Appendix B.7), and wikitext-103 dataset (Figure 18, Appendix B.8). With these experiments, we are confident that anticipatory recovery is a general phenomenon and not specific to the model or dataset we are using. We also want to point out that Pythia uses a very generic transformer architecture, and many experiments in the paper (Figure 2b, Figure 3, Figure 6a, Figure 14, Figure 15) use randomly initialized Pythia models and are therefore not particular to the specific pre-training recipe of Pythia.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mrX871n4QT", "reviewer": "Reviewer_4gi1", "summary": "The paper studies a specific kind of structured training dubbed cyclic training, where documents (or tasks) are presented cyclically in a fixed, repeated sequence. The paper reveals that LLMs exhibit anticipatory recovery behavior, i.e. they begin to recover from forgetting a document before encountering it again in the sequence. The paper offers some insights into the training dynamics in the cyclic training setup. emphasizing that this anticipatory behavior is not exclusive to LLMs but can be observed in other large-scale networks as well. The paper investigates the reasons for this anticipatory recovery, finding initial evidence that suggests adjacent tasks in the training sequence are represented more similarly by the model.", "questions": "Can you please address each of the weaknesses presented above?", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "1. The paper identifies a surprising phenomenon where models exhibit anticipatory recovery, which is counter-intuitive given the understanding of how LLMs handle sequential knowledge without explicit memory mechanisms\n2. It presents extensive experiments demonstrating that anticipatory recovery occurs not only in LLMs but also in other large-scale vision models. It shows that this behavior persists across several pretrained Pythia large models and, to a lesser extent, in randomly initialized models. The study further examines various factors, such as the number of tasks, training parameters, and data randomness, providing robust evidence for the phenomenon.\n3. The paper provides insights into the relationship between the anticipatory recovery phenomenon and the training dynamics.", "weaknesses": "1. While the anticipatory recovery phenomenon is intriguing and counterintuitive, its practical value remains unclear. The paper investigates implications on Online Loss Evaluation, but the experiments are limited and do not convincingly demonstrate why cyclic training would perform better than random training. It is also uncertain if the observed benefits are directly tied to anticipatory recovery or influenced by other experimental confounders.\n2. The motivation for using cyclic training is not particularly convincing. Cyclic training appears less natural compared to continual learning, and it is unclear if understanding training dynamics in this setup provides deep insights into the training of large models. This connection is weak and requires further development to establish its significance.\n3. Although the experiments are thorough, they are somewhat narrow in scope. The study primarily focuses on Pythia pretrained models and evaluates them using the CNN/Daily Mail dataset. It would have been beneficial to include a broader range of models and evaluation datasets. Additionally, the distinction between context length in the training setup and the models' context length is not clearly explained, leaving ambiguity about its impact on loss evaluation.\n4. While lesser priority, the paper lacks citations for a substantial body of work on continual learning in LLMs, such as AdapterCL by Madotto et al. (2020), Lfpt5 by Qin & Joty (2021), and ProgPrompt by Razdaibiedina et al. (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed feedback.\n\n**Re: “its practical value remains unclear.” and “cyclic training appears less natural compared to continual learning”.**\n> We would like to argue the opposite, that traditional continual learning is less natural than cyclic training. Most standard benchmarks and methods in continual learning (CL) assume IID task sampling, or every task trained only once; and the objective is to perform well on all tasks under these assumptions. Both the assumptions and the objective are unnatural: tasks naturally repeat in day-to-day life, with predictable transitions (e.g., a person might always eat breakfast after brushing teeth) and the main metric that an agent which acts and learns in the real world cares about is its performance on the tasks it actually encounters, at the times it encounters them (which justifies the prequential evaluation experiment in the paper, Table 1). As a result, cyclic training is closer to the challenges facing real-world agents than standard class- or task-incremental continual learning. \n\n> In this paper, we are not trying to argue that cyclic training performs better than random training in terms of average performance on all tasks. Instead, we argue that, when training machine learning models on naturalistic sequences (where task sequences tend to repeat), certain models automatically enjoy an anticipation benefit. The experiments in our paper aim at identifying what factors influence the anticipatory recovery effect, so that we can use the right models and optimizers to maximize this effect.\n\n**Re: “[the experiments] are somewhat narrow in scope”.**\n> While many of the experiments are conducted on Pythia models and the CNN/Daily Mail dataset to identify the influential factors of the anticipatory recovery effect, we also carried out many experiments with other models and datasets, including vision models (Figure 7, Section 3.4), the GPT-2-Large model (Figure 17, Appendix B.7), and wikitext-103 dataset (Figure 18, Appendix B.8).  With these experiments, we are confident that anticipatory recovery is a general phenomenon and not specific to the model or dataset we are using. We also want to point out that Pythia uses a very generic transformer architecture, and many experiments in the paper (Figure 2b, Figure 3, Figure 6a, Figure 14, Figure 15) uses randomly initialized Pythia models and is therefore not particular to the specific pre-training recipe of Pythia.\n\n**Re: “lacks citations for a substantial body of work on continual learning in LLMs”**\n> Thanks for pointing out these references. We will discuss these relevant works in the next version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cDrzuutLJJ", "reviewer": "Reviewer_gu9E", "summary": "The authors identify a phenomenon termed anticipatory recovery, where if the model is trained on a fixed shuffled data sequence for multiple epochs, the loss of a training sample will first increase and then decrease again right before it is met again. Extensive experiments have been conducted to show the phenomenon happens across different model size, optimization, initialization states, and several hyperparameters etc. The authors also show the existence of certain cyclic structure in data gradient, attention, and activations. Finally, a simple toy example is demonstrated for intuition.", "questions": "1. In Figure 1(b), how exactly is the loss computed? Say when you plot the figure for the 0.5th epoch, how is the number obtained over all documents?\n2. Figure 5 needs legends.\n3. Can you give more context on the prequential evaluation? What I had in mind about prequential coding is the same as looking at the training loss of each batch (see [1]).\n4. For figure 8(a, b), as a sanity check, if you reshuffle the data sequence (and fix that sequence for cyclic training), the same structure will show, is that correct?\n5. Around line 273-275, is the argument theoretically justified?\n6. From figure 6(a), we see that the anticipatory recovery happens with Adam but not quite with GD if the model is randomly initialized. Intuitively, Adam keeps a memory of the data sequence, but not GD. However, when the model is pretrained, then both GD and Adam show anticipatory recovery. This seems to suggest that pretraining is one of the most important factors. I would like to hear more from the authors.\n7. Can you explain more about the significance of Figure 8(c)? What is the $T$ used for this figure? The residual is the averaged weight residual over $T$ steps, so for each $t$, the residual will have $T-1$ overlaps, doesn't the overlap contribute significantly to the cyclic structure?\n\n[1] The Description Length of Deep Learning Models", "limitations": "Limitations have been discussed.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The phenomenon is very interesting, and the experiment settings are well considered and extensive.", "weaknesses": "In terms of presentation, several settings and contributions can be improved by providing more discussion. The analysis of the computational toy model can be improved.  See **questions** for more detail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed feedback.\n\n**Re: “In Figure 1(b), how exactly is the loss computed?”**\n> The loss in figure 1(b) is computed by replicating figure 1(a) on each document in the training sequence, and re-aligning these curves so that 0 on the x-axis always represents the moment before the first occurrence of the focal document. For example, if the length of the sequence is 50, then for the 0.5th epoch, we take the loss of document 1 after training on document 25; the loss of document 2 after training on document 26; …; the loss of document 50 after training on document 24 of the next epoch; and average these losses.\n\n**Re: “Figure 5 needs legends.”**\n> The legends in figure 5 are in the bar plots. For example, in figure 5(a), the light blue curve in the left subfigure corresponds to the light blue bar in the right subfigure, which shows the loss curve for 25 documents.\n\n**Re: “Can you give more context on the prequential evaluation?”**\n> Yes, prequential evaluation is the same as looking at the training loss of each batch. We argue that the performance on the upcoming task is what matters for an agent which continuously acts and learns in the real world, and prequential evaluation measures this performance. We will explain this better in the revision.\n\n**Re: “For figure 8(a, b), if you reshuffle the data sequence, the same structure will show.”**\n> Yes, the data sequence is randomly selected at the beginning and fixed for cyclic training. The same structure will show for any random data sequence, provided it is the same on every cycle.\n\n**Re: “Around line 273-275, is the argument theoretically justified?”**\n> Let $z_1 = Px_{i+1}$ and $z_2 = f_{i+1}(f_i^{-1}(Px_{i}))$, and assume that $f_{i+1}^{-1}$ is Lipschitz continuous. By the definition of Lipschitz continuity, we have  $f_{i+1}^{-1}(Px_{i+1}) - f_i^{-1}(Px_{i}) = |f(z_1)-f(z_2)| \\leq K|z_{1}-z_{2}|$ for some positive real constant K, so when we reduce $|z_{1}-z_{2}|$ we also reduce the upper bound for $f_{i+1}^{-1}(Px_{i+1}) - f_i^{-1}(Px_{i})$. Although this argument does not directly prove that  $f_{i+1}^{-1}(Px_{i+1}) - f_i^{-1}(Px_{i})$ will necessarily decrease, we think that proving the decrease of its upper bound serves as sufficient justification for this computational toy model.\n\n**Re: “This seems to suggest that pretraining is one of the most important factors.”**\n> Thanks for the good suggestion. In fact, GD exhibits anticipatory recovery without pre-training as well, especially in the later epochs. That said, more pre-training does make anticipatory recovery more significant. In rebuttal experiment E1, we took Pythia models pre-trained for a different number of steps (6K, 12K, 24K, 48K, 96K), and we found that more pre-training does give rise to higher anticipatory recovery (see Figure 1 of the rebuttal PDF).  We hypothesize this result fits a broader pattern in which the strength of the anticipatory recovery effect is related to how well the model can fit each successive training task (see section 3.3). Models with more pre-training steps are more capable of fitting each successive training task, and therefore exhibit higher anticipatory recovery. We plan to further study the effect of pre-training in future work.\n\n**Re: “Can you explain more about the significance of Figure 8(c)?”**\n> For this figure we used $T=25$. We want to note that, as seen from the equidistant stripes with interval $T$ in figure 8(c), the weight residuals at step $i$ have high similarity with those of $i+T$ and $i+2T$, while there is no overlap in their averaging window. For example, the weight residuals from step 25 and step 50 are highly correlated (with cosine similarity close to 1), but their averaging windows have no overlap at all; on the other hand, the averaging windows of step 25 and step 35 have roughly 50% overlap but the cosine similarity of the weight residuals at step 25 and step 35 is negative. Hence the cyclic structure is not due to overlaps in the averaging window. The significance of figure 8(c) is that, along with figure 9, it reveals the temporal structure of model weights in the cyclic training process.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "luNmFNhhLP", "reviewer": "Reviewer_5Cpf", "summary": "This paper explores the training dynamics of neural networks, particularly language models in a structured training setting where examples are presented cyclically. The paper discovered a phenomenon called \"anticipatory recovery,\" where models in each training epoch recover from forgetting of an example right before encountering the example. The paper show that the behavior is due to the temporal structure of gradients and model weights in cyclic training, and provided a theoretical characterization of the phenomenon using a toy model.", "questions": "Can you explain why the prequential evaluation setting is relevant, and why lower loss in prequential evaluation implies potential of structured training?\n\nCan you explain why the particular toy model is chosen in the theoretical analysis? Is there an intuitive explanation for the loss function in the toy model?", "limitations": "The authors adequately addressed the limitations.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The phenomenon of \"anticipatory recovery\" during cyclic training is a novel discovery and contrasts with the well-known issue of catastrophic forgetting.\n- The experiment is comprehensive, examining not just language models but also vision models and other architectures.\n- The analysis part is clear and well-structured, with detailed visualizations and theoretical analysis on a toy model to support and explain the empirical findings.", "weaknesses": "The main weakness of the paper is its unclear connection with common practices in training neural networks, therefore, it can be a little bit difficult to appreciate the significance of the findings. \n\n  - The paper contrasts the anticipatory recovery phenomenon with catastrophic forgetting, but this is an unfair comparison as anticipatory recovery happens only when the model is trained on the same data repeatedly, while catastrophic forgetting mostly happens when the model is trained on new data and never again have access to the old data. There is typically no catastrophic forgetting in multi-epoch training setting, and anticipatory recovery cannot happen if the old data is never seen again.\n\n  - The anticipatory recovery phenomenon may have limited practical implications, for 1) training with fixed example order usually achieves worse generalization performance than random-shuffling due to lack of mini-batch gradient diversity. 2) although anticipatory recovery relieves forgetting, it only reduces the forgetting of the next few examples in the coming sequence, rather than reducing forgetting of all previous examples as is the goal in continual learning. 3) the model weight seems to be forming a cyclic structure under cyclic training, which pushes the model weight to become closer to its corresponding state in previous epochs. Would this also mean that the model is farther apart from its state in the other half of the epoch (compared to random-shuffle training)? If so, less forgetting of the next few examples may come at the cost of more forgetting of the other examples in the dataset.\n\n  - The author proposes to examine the potential of structured training, but it is not clear how the anticipatory recovery phenomenon can benefit structured training. Why is recovery from forgetting right before training useful, when the model is soon to be trained anyway? Also, the anticipatory recovery phenomenon seems only exist in the cyclic training setting, which is a very special form of structured training. It is not clear how the findings can be generalized to other forms of structured training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed feedback.\n\n**Re: Unfair comparison for catastrophic forgetting and anticipatory recovery.**\n> We agree with the reviewer that there is no catastrophic forgetting if all tasks are trained in every epoch, or if training passes over all tasks many times. However, in our work we show that the forgetting and recovery can be observed within each training cycle, before the same document is encountered again. This is a different timescale than the reduction of forgetting in typical multi-epoch training, and is a surprising phenomenon. Prior to our paper, the accepted wisdom in the continual learning community was that catastrophic forgetting would arise within each cycle as we train on other tasks before training on the focal task again. We show an important case where the catastrophic forgetting effect is largely mitigated due to the anticipatory recovery towards the end of the training cycle, which challenges the accepted wisdom.\n\n**Re: “The anticipatory recovery phenomenon may have limited practical implications.”**\n> We agree with the reviewer that for cyclic training, compared to random shuffled training, less forgetting of the next few examples comes at the cost of more forgetting of the other examples in the dataset. However, we argue that what real-world agents care about is the performance on tasks they actually encounter at the times they encounter them. If task order is (approximately) repeating, then a bias toward good performance on upcoming tasks at the cost of other tasks is a good thing, as shown in figure 1c of the paper.\n\n**Re: “Why the prequential evaluation setting is relevant, and why lower loss in prequential evaluation implies potential of structured training.”**\n> As we explain in the general rebuttal, we introduce cyclic training not as a method but as an objective. Because many natural environments are quasi-cyclic, it is important to evaluate existing methods in this setting, as opposed to traditional CL setups. For an agent that acts and learns in the real world, the most important metric it cares about is its performance on the tasks it actually encounters, at the times it encounters them. This justifies the relevance of prequential evaluation in building and evaluating such agents. According to the prequential view, training and testing are two sides of the same coin. While it is correct that the model is about to be trained on the next task, its loss on the next task is determined by how good its predictions are prior to the update. Lower loss in prequential evaluation suggests that certain models can automatically enjoy an anticipation benefit when trained on structured sequences (where task sequences tend to repeat) and catastrophic forgetting might be less of an issue for these models in these structured environments.\n\n**Re: “It is not clear how the findings can be generalized to other forms of structured training.”**\n> In addition to pure cyclic training, we also experimented with some variants in the paper, including data randomization (Figure 4, lines 148-154) and partial random shuffling (Figure 12, lines 602-608), and demonstrated anticipatory recovery in these cases. In rebuttal experiment E3, we experimented with another variant where only the first 20 documents are kept fixed in each epoch, and a random number of other documents are inserted in between. This new setting generalizes cyclic training in that (1) rather than having the same documents in every epoch, we insert random other documents between every repetition (2) epochs can have different lengths. We still observe anticipatory recovery for documents 2 through 20 in this setting, suggesting that anticipatory recovery exists as long as there is a repeating sub-sequence in the data stream.\n\n**Re: “Why is the particular toy model chosen in the theoretical analysis.”**\n> The particular toy model is chosen for theoretical analysis because it is a minimalist setting where we can reproduce the anticipatory recovery phenomenon and each component in the toy model has a natural and intuitive counterpart in the more complex LLM training procedure. The L2 loss function is chosen due to its wide usage in machine learning applications. We believe that similar analysis can be done for other common loss functions such as the cross entropy loss as well.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**The practicality of cyclic training and relevance of prequential evaluation:**

​​We appreciate the reviewers' recognition of the technical soundness and overall quality of our manuscript. While reviewers dpPf and gu9E have appreciated the contribution of our findings, we understand that reviewers 4gi1 and 5Cpf raised concerns about the practicality of the cyclic training approach and the relevance of prequential evaluation. Here we provide a more comprehensive explanation of the practical applications and relevance of our findings. We will incorporate these into the next version of the paper.

We believe cyclic training and prequential evaluation are more natural and relevant for real-world agents than standard continual learning. In natural environments, tasks recur and they often do so in predictable patterns (e.g., a person anticipating to eat breakfast after taking a morning shower). Our cyclic training approximates this structure, while standard continual learning settings (e.g. task- and class-incremental learning) of seeing each task only once are artificial and fail to consider this structure. Furthermore, the only metric a real agent cares about is its performance on the tasks it actually encounters, at the times it encounters them. This is prequential evaluation, which examines the quality of predictions at each time step conditioned on knowledge from earlier experience.

The aim of our paper is not to propose cyclic training as a method to improve performance on standard continual learning metrics. Instead we argue many natural environments are quasi-cyclic and so it is important to understand how existing deep learning methods perform in this setting. Anticipatory recovery is a surprising and significant discovery in this regard.

**New Experiment Results on pre-training, cosine learning rate schedule, and structured training:**

**Experiment E1:** In response to question 6 from reviewer gu9E, we took Pythia models pre-trained for a different number of steps (6K, 12K, 24K, 48K, 96K), and we found that more pre-training does give rise to higher anticipatory recovery.  As we summarize at the end of section 3.3, we hypothesize this result fits a broader pattern in which the strength of the anticipatory recovery effect is related to how well the model can fit each successive training task. Models with more pre-training steps are more capable of fitting each successive training task, and therefore exhibit higher anticipatory recovery. The experiment results can be found in Figure 1 of the rebuttal PDF.

**Experiment E2:** In response to question Q1 from reviewer dpPf, we experimented with cosine learning rate scheduling on the Pythia-1B model with 10 epochs. We show that the model also exhibits the anticipatory recovery effect in typical LLM optimization schemes. The experiment results can be found in Figure 2 of the rebuttal PDF.

**Experiment E3:** In response to a question from reviewer 5Cpf, we experimented with a new setting where only the first 20 documents are kept fixed in each epoch, and a random number of other documents are inserted after the first 20. This new setting generalizes cyclic training in that (1) rather than having the same documents in every epoch, we insert random other documents between every repetition (2) epochs can have different lengths. We still observe anticipatory recovery for documents 2 through 20 in this setting, suggesting that anticipatory recovery exists as long as there is a repeating sub-sequence in the data stream. The experiment results can be found in Figure 3 of the rebuttal PDF.

Meta-review: This paper studies a specific kind of structured training dubbed cyclic training, where documents are presented cyclically in a fixed, repeated sequence. This reveals that LLMs exhibit anticipatory recovery behavior, i.e. they begin to recover from forgetting a document before encountering it again in the sequence. Some insights into the training dynamics in the cyclic training setup are offered, emphasizing that this anticipatory behavior is not exclusive to LLMs but can be observed in other large-scale networks as well. Reasons for this anticipatory recovery are investigated: finding initial evidence that suggests adjacent documents in the training sequence are represented more similarly by the model.

Scores include strong accept (8), accept (7),  borderline accept (5), and weak reject (4). Despite this range of scores, the reviewers fundamentally agree on:
- Strength: what makes this problem exciting is  that  it “identifies a surprising phenomenon where models exhibit anticipatory recovery, which is counter-intuitive given the understanding of how LLMs handle sequential knowledge without explicit memory mechanisms” (reviewer dpPf who gave a 8)
- Weaknesses: there are serious “significance” issues (some rate it in the contribution score). As stated by reviewer 5Cpf who gave a 4: “The main weakness of the paper is its unclear connection with common practices in training neural networks, therefore, it can be a little bit difficult to appreciate the significance of the findings”

To dig out further in the issue of the significance of this work:
- Could it lead to improved "learning curricula that achieve learning efficiency “?
 However, several reviewers do not see much future in the fixed order training required by the method “training with fixed example order usually achieves worse generalization performance than random-shuffling”.
- Could it help our theoretical understanding of the training dynamics of neural networks? However, this paper only brings a little light to this understanding.

In summary, the reviewer disagreement is about how critical the paper should be to the ML practitioner, which represents nowadays a large fraction NeuRIPS attendance. As a long time NeuRIPS attendee, I don't think it should be limited to the practitioner, and even a paper whose main contribution is to challenge our understanding of the learning process is worth publishing, so I am inclined towards accept.
Decision: Accept (poster)