Emergent properties with repeated examples

Authors: Francois Charton, Julia Kempe
Primary area: foundation or frontier models, including LLMs
Keywords: transformers, learning on repeated examples, emergence

Abstract:
We study the performance of transformers as a function of the number of repetitions of training examples with algorithmically generated datasets. On three problems of mathematics: the greatest common divisor, modular multiplication, and matrix eigenvalues, we show that for a fixed number of training steps, models trained on smaller sets of repeated examples outperform models trained on larger sets of single-use examples. We also demonstrate that {\em two-set training} - repeated use of a small random subset of examples, along normal sampling on the rest of the training set - provides for faster learning and better performance. This highlights that the benefits of repetition can outweigh those of data diversity. These datasets and problems provide a controlled setting to shed light on the still poorly understood interplay between generalization and memorization in deep learning.

Decision: reject
Reviewer scores: 6, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730152433443

thread_type:
official_review

<thread_object>
creator: reviewer_bpnp

<content>
summary:
Modern language models are typically trained with only one or at most a few epochs. This paper studies the potential benefit of training for more than a few epochs. With experiments on three synthetic problems (GCD, modular multiplication and matrix eigenvalues), the paper shows that for certain compute budget (or "training budget" as termed in the paper), repeating the same data many times can actually be better than using completely fresh data (i.e. 1 epoch or "online" training). Based on this intuition, this paper further proposed a "two-set training" paradigm where only a subset of training examples is repeated, to get the benefit of repeated training while also mitigating the potential overfitting issue.

strengths:
The paper presented the idea clearly and conducted controlled experiments to study the question. It also proposed new training paradigms based on the observations that could potentially help improving model training.

weaknesses:
1. This paper is making very big claims (e.g. of "emergent properties" or "emergent learning") based on very synthetic settings. While the models used are transformer models, they are tasked with specialized tokenizers for numbers and trained to perform a single synthetic task (e.g. given two numbers, output the greatest common divisor of the two numbers). In other words, those "language models" do not have any language capabilities. I think the paper could be better either (A) frame this as a general machine learning problem and have comparison studies covering many different neural network architectures, or (B) focus on LM but have additional experiments on more realistic settings. For example, in terms of relevancy to LMs, I think a study of fine-tuning with a proper LM would be more "transferable" to our understanding of how LM learning works than this synthetic pre-training setting.

2. There was the classical decomposition of test accuracy to training accuracy + generalization gap. It seems especially relevant when the paper talks about training with or without repetitions on difficult math / arithmetic tasks. Many of the observations might be more intuitively explained by undertraining --- i.e. the training accuracy itself is already quite bad as the model struggle to fit the problem (with small data repetitions). Adding studies from this angle could potentially make the paper more clear, but in the meantime, the observations would be less surprising if it is mostly explained by such classical decomposition.

3. The paper proposed two-set training algorithm, but there is no comparison to any previous curriculum learning baseline algorithms.

4. It is not clear if the experiments are comprehensive enough to support some of the big claims. For example, the paper talks about "emergence", *a task inaccessible to models trained with large or unlimited DB is learned with small DB*. The conclusion may change depending on the model architecture or even model sizes, and if may even be unclear if each setting allows to choose their own optimal hyperparameters. This paper is making those conclusions based on experiments with a single transformer model with fixed number of parameters and a fixed set of hyperparameters. For example, with repeated examples, later training steps would have potentially smaller updates because the gradients are smaller on seen examples. If this is benefiting the learning, could a similar effects be achieved by a better learning rate decaying scheme?

questions:
Please see the "Weakness" section above.

---

**Post Rebuttal**: Thanks the authors for the response. I raised my rating, but I'm still not super convinced about the setting as well as concrete evidences in the experiments of this paper excluding the undertraining as a potential confounding factor.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730398302675

thread_type:
official_review

<thread_object>
creator: reviewer_uayc

<content>
summary:
This work investigates the effect of training with repeated examples (like training over the training data over multiple epochs) compared to training on very large datasets only once. The authors run experiments on three large-scale math problems, to compute GCD, modular multiplication, and computing eigenvalues using varying sizes of transformers and varying data and training budgets. The authors also propose a two-set training method in which they repeat one small subset of data throughout training and otherwise train on new data continually.

strengths:
The paper is written clearly and is easy to follow and understand. The problem posed is important and if answered can be helpful in guiding training methods, fine-tuning and data gathering for complex problems. The results for the two-set training method bring up interesting questions about the role of memorization that would be interesting for future research.  All experiments are very thorough, showing good evidence for their claims on the datasets chosen.

weaknesses:
From my point of view, this is not a particularly new idea. Training over many epochs over the training data is standard in most applications. This is perhaps not common in new applications such as in NLP as the authors cite for large language models. Given this, it would have been more interesting to see applications within NLP instead. The problems considered (algebraic problems) are very different, and the connection to other applications like NLP is not well-motivated. 

In terms of related work, it may be interesting to investigate the connection to continual learning and catastrophic forgetting. This is more geared towards learning new tasks, and not about revisiting old examples from the same distribution. However, some of the work done here may provide valuable insight and would be nice to have some discussion comparing the methods in this area with yours.

Moreover, your idea of two-set learning sounds similar to spaced repetition in human learning -- being shown old instances right before you are about to forget them. There is not much work in this direction for ML training applications, although I did find this paper [1] which seems to have similar ideas to yours, proposing an order and frequency to the examples being trained on. 


[1] Repeat before Forgetting: Spaced Repetition for Efficient and Effective Training of Neural Networks, Hadi Amiri, Timothy A. Miller, Guergana Savova.

questions:
1. Why did you choose these experiments? I understand it is easier to control the data and the distribution of repeated examples. However, it is hard to see why conclusions drawn on algebraic problems should extend to other more natural data and NLP problems. 

2. Can you try your ideas and run the experiments on non-synthetic datasets? It would be interesting to see if the two-set training works well for other data like CIFAR-10 or other natural datasets that you may find more appropriate.

3. Can you provide some discussion on continual learning etc. as I mentioned in the above section?

4. In Figure 5, the results are quite unstable as a function of your parameters, you need to be very careful in choosing the repeated set size etc. In practice, iterating over the possible set sizes and repeated set probability and training given each choice is very expensive. Do you have any intuition for how to choose these hyper-parameters in general? There does seem to be a general common "shape" where training works better.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730497624149

thread_type:
official_review

<thread_object>
creator: reviewer_knen

<content>
summary:
The present paper deals with the essential question of the role of repetition in the training data of machine learning models. More specifically, the work delves into the effect of introducing repeated examples when training transformers on three mathematical problems: finding the greatest common divisor between two numbers, performing modular multiplication, and computing the eigenvalues of (small) matrices. Through extensive numerical experiments, the authors show that repetition during training is very valuable for transformers to perform well on these tasks, and that might be necessary for learning to emerge in the case of modular multiplication. In light of these results, they propose a two-set training procedure, in which training examples may be drawn from either a large set of examples that will be seen only a handful of times or from a much smaller set of examples repeated many times during training. Consistent with their initial experiments, they find that this procedure improves the performance of trained networks for given data budgets, and in some cases enables learning. The authors finally consider variations of this procedure, notably the natural idea of curating the repeated set to further improve performance. Surprisingly, they find that curating does not provide further gains, or may be detrimental.

strengths:
Overall, this paper explores an important aspect of machine learning—understanding why algorithms are usually expected to suffer from repetitions whereas human and animal learning appear to be highly dependent on them for learning—and could therefore be a significant contribution to this puzzle as it provides counter-examples that deserve to be better understood. By relying on the transformer architecture, which has become ubiquitous in many practical implementations, the authors also maximize the chance that their findings, and the training protocol they propose, may be taken advantage of in practical settings. The study of mathematical tasks provides a well-controlled environment where memorization can be identified, while properly learning an algorithm is necessary to achieve good generalization. The paper is well-written, and the author’s conclusions are clearly stated.

weaknesses:
I believe the main limitation of this work lies in the limited number of tasks it considers, undermining the generality of its conclusions. In particular, the authors repeatedly make statements such as ‘models trained on smaller sets of repeated examples outperform models trained on larger sets of single-use examples’ (abstract); ‘smaller data budgets and more frequent repetition allow for faster learning, but also for much better performance’ (p. 5); ‘learning emerges through repetition’ (p. 5) etc. Based on the presented results however, it appears that these claims should be slightly tempered: there is a tradeoff between the repetition and the diversity that is necessary to avoid overfitting, as clearly illustrated in Fig. 2. While the authors show that repetitions at a given data budget may be beneficial, this relation is strongly non-monotonous, and in the greatest common divisor task one notably still requires a large number of unique samples to perform correctly (and as apparent in Fig. 4 the balance between repetition and diversity is hard to strike in this problem!). Clarifying this point and emphasizing that it is a tradeoff would make the paper more convincing, and would add to the relevance of the two-set procedure which precisely tries to provide the ideal tradeoff. 

It also remains unclear whether the two-set procedure is relevant in settings where the transformer must not necessarily learn a clear-cut algorithm. For instance, could such a procedure be effective in image classification and more generally vision-related tasks? It seems plausible that the need for repetition is related to the hardness of the task at hand, which is not discussed in the paper. 
In that respect, the difference in hardness of the three tasks exposed is not straightforward to understand: the authors notably state that the computation of eigenvalues is the hardest task they consider, yet the smallest model (in number of parameters) is used to tackle it. Therefore, it is not evident how the eigenvalue problem contributes to the clarity of the paper and its conclusion, also considering the relatively poor results that the authors find (4/30 as the maximum trained model success rate if I understand correctly) and the lack of clear plots dedicated to this third problem, despite it being expected to be the most difficult to tackle.

From a more technical standpoint, the accuracy metric chosen by the authors is not easy to interpret, notably in Fig. 2. While Appendix A provides some explanation as to how the chosen accuracy may stay fixed while the test loss explodes, the fact that it is on the eigenvalue computation and not the GCD problem for clear interpretation of Fig. 2 makes it still unsatisfactory. Besides, Fig. 6 does show some examples where the accuracy decreases as the model overfits.

Finally, the conclusion of the authors that the transformers should somehow be able to distinguish between already seen and unseen examples is puzzling. Could the role of repetition not be simply explained by the fact that there needs to be some form of symmetry breaking in the direction of the loss to follow and that repeating examples allows for a clear direction in the loss landscape to emerge? In that respect, I point the authors towards the papers: 
Dandi, Y., Troiani, E., Arnaboldi, L., Pesce, L., Zdeborová, L., & Krzakala, F. (2024). The benefits of reusing batches for gradient descent in two-layer networks: Breaking the curse of information and leap exponents. arXiv preprint arXiv:2402.03220. 
Arnaboldi, L., Dandi, Y., Krzakala, F., Pesce, L., & Stephan, L. (2024). Repetita iuvant: Data repetition allows sgd to learn high-dimensional multi-index functions. arXiv preprint arXiv:2405.15459

A more in-depth discussion of the different mechanisms that could explain the presented phenomenology, could significantly improve the paper.

questions:
List of questions (including some repetitions of the above):
- Do the authors believe that their findings hold in tasks that do not require learning an algorithm? For example in vision-related tasks where memorization could be more useful (and sufficient in some cases)?
- Do they know of an example where repetition or the two-step procedure is not beneficial? One could notably think of tasks where curriculum learning is decisively effective, then shouldn’t random repetitions at least fare worse than curated ones? In other words, can the two-step procedure work well when curriculum learning works well?
- Can the two-step procedure be performed by showing only the repeated samples in a first learning phase, and then moving on to the more diverse samples in a second phase? i.e. focus first on discovering some ‘rules’, before generalizing these rules? I believe that finding good results in such a procedure would go in the direction of a loss-landscape-based interpretation of the phenomenon, in the sense of the Dandi et al. paper mentioned above. This would also further clarify the difference between what the authors observe and grokking (in addition to the different timescales and sample sizes they mention in Sec. 2).
- Can the authors clarify how accuracy is never affected by overfitting in the GCD task? How general do they expect this behavior to be?
- Can the authors clarify what they mean by a transformer being able to identify deja vu and jamais vu examples? Why should the architecture ‘care’?

**AFTER DISCUSSION**

The authors have addressed the main concerns I presented and included different explorations that I believe clarify and strengthen their message.  I will raise my score to a 6, for the moment.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730558126253

thread_type:
official_review

<thread_object>
creator: reviewer_yjc8

<content>
summary:
This paper empirically shows that the repetition of training data can be beneficial in certain setups. The authors conduct experiments on three algorithmically generated datasets, and show that model trained with small data budgets outperforms model trained on larger data budgets. The authors also identify a two-phase training algorithm that gives faster training and better performance.

strengths:
1. The experiments are conducted on three algorithmically generated datasets of math tasks, which is an ideal setup of controlled experiments. The experiment setups are clearly described, and the results are well presented and explained. 

2. The empirical findings of the beneficial of repeated examples have decent practical meanings. The success of two set training is interesting and novel to me.

weaknesses:
1. My major concern is that the experiments are only conducted on algorithmically generated synthetic data, instead of real world datasets. This makes me not fully convinced that the experimental findings are universal and can be transferred to more realistic setups. 
Specifically, I am wondering whether the main findings, especially the success of two set training, also applies to real language datasets. Do you have empirical results to support that? Besides, the three dataset are all math problems. Do you think that this specific type of training data might have a positive influence on the performance of repeated data? 

2. Does the emergent property in the title mean that the observed phenomena only happens for training at scale? Do you have found any criteria that can indicate under what circumstance can repeated data or two-set training be beneficial?

questions:
See weaknesses part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732020106145

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response: Synthetic Data Setting, Relation to Continual/Curriculum Learning, Additional Experiments

comment:
1/2 
We thank the reviewers for taking time to read our paper and for their valuable comments and are happy to see the uniform interest our observed phenomena have elicited. We have responded to each reviewer individually and can include some of the desired discussion/references in the paper - we plan to upload a suggested new version after gathering initial feedback and will notify when we do.

**Synthetic Data Setting:**
Several reviewers have commented on the nature of our experiments on synthetic data only. We would like to address this point here:

We have made a deliberate choice to use synthetic data in this work: on "real world data" (i.e. text scraped off the Internet, or code from public repositories), repetition cannot be controlled, and it would not be possible to measure the effects observed here in a meaningful way. We believe two-set training is somehow "baked into" many natural language tasks, and may even explain some of the success of language models, because the training data is replete with repeated sentences and paragraphs (or statements and functions, in the case of code). The mathematical nature of the task is only visible to human observers: for the model, which minimizes a cross-entropy loss, it is not different from other "letter games", such as code refactoring, text summarizing, or various reasoning tasks. As such, we do not think it has an impact on our conclusion. 

We would like to add that the practice of studying phenomena in Deep Learning, and even discovering new ones, on *synthetic* data is widespread and established. Allow us to provide just three examples:
- the influential "grokking" paper [Power et al.22] uses small, synthetic, mathematical datasets (among them modular multiplication) to exhibit what is now well known as "grokking"
- in-context learning of transformers has gained its theoretical foundation in an early, highly cited paper [Garg et al. 22] (with many follow-ups). Here, the synthetic data comes from linear functions, but the insights on how "meta" learning works in-context are profound.
- another example is Zeyuan Allen-Zhu's impressive series of works (with various co-authors) on the "physics of deep learning" (cf. the excellent and very well attended NeurIPS'24 tutorial he gave on this subject). All the experiments performed there are on controlled, synthetic data, with interesting effects emerging - as in our case.

All of these papers hope and speculate that the insights they gain in these controlled settings carry over to explain "real" phenomena observed in larger models, at least to a certain extend, and it seems they have proven right, even though they have not directly performed "real world" studies. Note that there is increased visibility for this approach, as witnessed by the upcoming workshop at NeurIPS'24 on Scientific Methods for Understanding Deep Learning (ScifoDL), which precisely *promotes a complementary approach that is centered on the use of the scientific method, which forms hypotheses and designs controlled experiments to test them.*

Complementary to fostering understanding or discovery of new phenomena that carry over to larger realistic models, there is an additional important aspect of our work that pertains to AI4Math. There is increasing literature on using transformers for Math problems and success stories start to emerge (see e.g. [Alfarano et al. 24, Charton et al.24] and [New Scientist 24]), based on transformer models of the kind we utilize in our study. For this community, our insights are of immediate relevance, as they give prescriptive advice on how to utilize training data (iterate rather than one-pass, use two-set training). Indeed, one feature of our tasks - which is the case in most AI4Math settings - is that they are deterministic : there is only one correct solution. One might speculate how much this fact might impact our conclusions, which we believe is an excellent topic of study in future work. But even in this setting, we hope that our results will be of practical and immediate relevance.

[Power et al. 22] Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, Vedant Misra, "Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets", OpenAI, arXiv 2022

[Garg et al. 22] Shivam Garg, Dimitris Tsipras, Percy Liang, Gregory Valiant, "What Can Transformers Learn In-Context? A Case Study of Simple Function Classes", NeurIPS 2022

[Alfarano et al. 24] A Alfarano, F Charton, A Hayat, "Global Lyapunov functions: a long-standing open problem in mathematics, with symbolic transformers", arXiv 2410.08304

[New Scientist 24] New Scientist "Meta AI tackles maths problems that stumped humans for over a century" https://www.newscientist.com/article/2452780-meta-ai-tackles-maths-problems-that-stumped-humans-for-over-a-century/

[Charton et al. 24] F Charton, JS Ellenberg, AZ Wagner, G Williamson, "PatternBoost: Constructions in Mathematics with a Little Help from AI", arXiv 2411.00566
</content>
</thread_object>

</thread>
<thread>
creation date:
1732552489965

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Updated manuscript

comment:
In response to the reviewers' feedback and in particular the very helpful suggestions of reviewer knen, we have uploaded a modified version of our manuscript with changes highlighted in blue. 

Summary of changes and additions:

1. We have clarified and expanded the eigenvalue experiments and a new Table 3 (p. 8-9) and moved them into the main body of the paper (eliminated an Appendix)
2. We have added a new Appendix A with additional discussion and references. In particular, it adds additional clarifications and comparison to curriculum learning and continual learning, as well as comparison to a couple theoretical works and discussion on how they do or do not apply to our observed effects.
3. We have added a little more discussion in the eigenvalue paragraph in Section 3 discussing the choice of this problem. 
4. We have changed the final discussion (page 10) by being specific that our effects are observed for three deterministic reasoning tasks. We have also moved the more speculative 'deja-vu' paragraph to the Appendix A to make the discussion more crisp.
5. Some clarifications in Appendix D.1 and minor changes (marked in blue)

We hope the reviewers are satisfied with these changes and thank them for their constructive feedback.
</content>
</thread_object>

</thread>


Metareview: