Rethinking Memorization in LLMs: On Learning by Rote vs. with Understanding

Authors: Till Speicher, Bishwamittra Ghosh, Mohammad Aflah Khan, Qinyuan Wu, Vedant Nanda, Soumi Das, Krishna P. Gummadi, Evimaria Terzi
Primary area: foundation or frontier models, including LLMs
Keywords: language models, memorization, generalization

Abstract:
Understanding whether and to what extent token sequences generated by large language models (LLMs) are the result of regurgitating memorized training data or are based on meaningful learning of the training data's syntax and semantics has many important implications.
In order to cleanly measure and disentangle token recollection by rote (memorization) from generation with understanding, we create an experimental framework that is based on training LLMs over *sequences generated using formal grammars*. Our framework allows us to better understand the interplay between the two types of learning, namely, *by rote* vs. *with understanding*. Using our framework we make several striking observations that hold consistently across different open-source model families (Pythia, Llama, and Mistral): (a) we find that the learning types are at odds with each other during training, i.e., rote learning harms understanding and by developing understanding, models forget previously memorized sequences, (b) we find that *entropy of the training datasets* impacts the ease of learning, with lower entropy datasets being easier to learn with understanding and higher entropy datasets being easier to learn by rote, (c) we highlight the difficulty of determining the type of learning involved in a model based solely on recollecting a training data sequence. Our surprising results have significant downstream implications in the study and usage of LLMs.

Decision: reject
Reviewer scores: 3, 5, 3, 3

Reviews and discussion:
<thread>
creation date:
1730518053708

thread_type:
official_review

<thread_object>
creator: reviewer_dqyx

<content>
summary:
This paper studies learning curves of various language models on some synthetic languages (generated from a unigram model and a nonrecursive PCFG). They define memorization, or learning by rote, to begin at the point that a gap between the training and test loss begins (more precisely, when training loss > 1.05 * test loss), and generalization, or learning by understanding, to end when test loss is minimized.

-

strengths:
The paper finds that:
- memorization usually begins before generalization ends
- continued training on a different training set restarts generalization and causes forgetting of the first training set
- lower-entropy distributions are easier to generalize on; higher-entropy distributions are more amenable to memorization

weaknesses:
The paper sets for itself the goal of _how_ models memorize (l. 159), but to me this sounds like studying the mechanisms internal to the model that enable it to memorize, whereas this study is a study of learning curves only.

The terms of "learning by rote" and "learning with understanding" are central to the paper. The experiments all center around identifying the beginnings and ends of these two kinds of learning, and one of the aims of the paper is to propose these two definitions as a better alternative to the tests for memorization in previous work. Given the centrality of these two concepts, I would like to see (1) clearer definitions of them earlier in the paper (they are defined only briefly at lines 98-99, and more clearly not until page 6) and (2) some justification for why your definitions of these terms are the right ones.

I'd suggest that you use a single term for each concept, instead of using "learning by rote" and "memorization" as synonyms and "learning with understanding" and "generalization" as synonyms.

The experiments are on synthetic languages instead of naturally occurring data. The reason is to be able to control things like the vocabulary size or the entropy. But there's nothing about the experiments that depends on any details of the data distribution, and the findings are basically the same for the unigram model and the PCFG.

In the end, the findings are not very surprising. I feel that the insights that can be gained by looking at learning curves alone are limited.

questions:
How was the PCFG designed?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730643384636

thread_type:
official_review

<thread_object>
creator: reviewer_ojus

<content>
summary:
This paper attempts to explore the learning patterns of large language models, specifically whether they learn by rote or by understanding. The authors generate synthesis data via probabilistic context-free grammar(PCFG) as training data and compare loss curves of different LLMs such as Pythia and LLaMA. The main conclusion is that by observing the curves between training loss and test loss, it can infer the learning patterns of the language models. If the training loss continues to decrease while the test loss increases, it is considered to be learning by rote.

strengths:
1.  The topic under investigation is meaningful.

weaknesses:
1. The phenomenon of learning by rote discussed in this paper is similar to overfitting. However, their difference is never explained. If a small model of 128M is pre-trained on WikiText-103 for lots of epochs, a similar phenomenon occurs as well. Essentially, it is overfitting—when the model's capacity exceeds the information provided by the text.  It lacks some quantitative experimental data and rigorous argumentation. For example, the relationship between the amount of information in the pre-train corpus and the model parameters required to alleviate rote memorization, and how to estimate the amount of information. Thus current findings are difficult to constitute actual contributions. The conclusion also lacks practical significance. When the test loss diverges from the training loss, it indicates learning by rote, which is just another name for overfitting.

2. The synthetic data used is too trivial and lacks any distinguishing properties like other works(https://arxiv.org/abs/2305.13673). It simply generates strings based on PCFG, without any extra knowledge or inner logic. To determine whether the model is based on understanding or rote memorization, there needs to be at least some semantic-level properties. For example, executable formal languages. LLMs can understand complex logic, such as sorting and human-defined functions.  For example, making the model fit the results of executable formal languages may help us determine whether the model has achieved true understanding rather than learning by rote.

questions:
Does "training size n=8, n=64" refer to the number of items in the training dataset?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730704965724

thread_type:
official_review

<thread_object>
creator: reviewer_qwvw

<content>
summary:
The authors study the factor of memorization (rote) and generalization (understanding) during training. The experiments are conducted on training and testing data on synthetic data generated from context-free grammar. The authors propose a new way to quantify memorization, and find that rote learning harms understanding and by developing understanding.

strengths:
* the experiments consider various aspect of the generalization & memorization aspect.

* the new way to quantize memorization feels like a progress. And a revisiting of the traditional bias-variance trade-off seems to be interesting in that it makes the finer-grain distinction that there could be a phase of purely understanding before the traditional trade-off point ("end of learning by understanding"). 

* Line 418 - 420: this feels like an interesting observation.

weaknesses:
* missing citation and discussion. A more comprehensive study of generalization [2, 3] exists; and CFG data was used for conducting thorough experiments [1] before, and thus is not novel (but the author made them sound novel in abstract Line 16). Also, the second experiment (Figure 4) is closely connected to continual learning, but no discussion of relevant previous results are made.

* some of important point is not addressed in the paper.

Line 73-74: "start of learning by rote" --- Why is learning previous to this not by rote? "end of learning by understanding" --- When is the start of learning by understanding?

* the training set sizes and test sizes are too small. Despite this, the observation is still a bit confusing: train and test loss has small gap, while the author give the impression in Line 196-200 that the output space is large. I think this might be caused by simple CFG grammar. Would be nice to provide some insight for why.

* I disagree with some of the interpretation of results. Or the use of terms by the authors are not rigorous.

Line 422-424: First I think catastrophic forgetting happens in continual learning setting. I don't think the model forgets entirely ("destroy") about D_train,1 as the loss is still lower than D_test.

Line 426 "destroy the ability to generalize...": what's the observation for "destroy"?

Line 428-429 "differences between human and machine cognition": I think this spectulation is taken to far from what's in the paper.

Line 499-500 "Intuitively, .... to memorize": the phrasing seems in-accurate. It's harder to generalize with higher entropy distribution, so the model is **encouraged** to be for the model to be a lookup table. I don't know why it is "easier".

Line 521-522 "measures of memorization must ...": Why is it a "must"? 



[1]: Physics of Language Models: Part 1, Learning Hierarchical Language Structures
[2]: COGS: A Compositional Generalization Challenge Based on Semantic Interpretation
[3]: SLOG: A Structural Generalization Benchmark for Semantic Parsing

questions:
* does the syntax support testing hierarchical/compositional generalization?

* line 98: "memorization begins at epoch 6": modern LLM didn't go through data for more than 1 epoch sometimes, how would the author explain the big difference here?

* Line 317: what does it mean by "takes a very long time"? decoding time?

* Line 370-371 "one must generate test and trainign data...": I can't imagine scenario like this. What do the authors mean by "generate"?

* how would the author comment if the defined memorization term (Line 356-358) happens to be negative?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730776168771

thread_type:
official_review

<thread_object>
creator: reviewer_myzz

<content>
summary:
The paper looks at fine-tuning LLMs (Pythia, LLama, Mistral) on strings generated from PCFGs. They analyze the resulting training dynamics and find a generalization phase, and then a memorization phase. These phases are different for different LLMs, and also differ based on whether the training data came from a PCFG or is randomly sampled. Note however that the size of the training data is unusually small and always less than 64 strings.

Next, the paper tries to understand the interplay between memorization and generalization by fine-tuning sequentially on 2 samples $D_1$ and $D_2$,  drawn from the same distribution. They find that training on $D_1$ first causes loss on the test set to go down, and then go up with extended training. After training on $D_1$, training on $D_2$ causes loss to increase on $D_1$, while loss on the test set starts going down again. The authors conclude that they can trigger forgetting of previously memorized examples by memorizing new data from the same distribution, which further can also trigger a round of generalization (since the test set loss also starts going down).

strengths:
- This work looks at the challenging problem of identifying the interplay between generalization and memorization in LLMs, and designs some interesting experiments to study this.
- One interesting experiment is looking at what properties of the data cause memorization, where the authors find increasing entropy of the data distribution increases the model’s propensity to memorize, while lower entropy causes models to enter into the generalization mode.

weaknesses:
- I think the biggest weakness for the paper is that it re-frames many existing and well-known observations from standard machine learning. For instance the memorization / generalization plots are simply the U-shaped val loss plots seen in ML 101 classes. This is commonly referred to as over-fitting. I’m not sure if there’s any deeper insights to be gathered from just that one plot (which the paper heavily relies on).
- Similarly, the fact that loss will increase on a small train set of < 32 examples, after over-fitting to it and then starting to train on a second training sample, is also extremely well-known. 
- The size of the datasets used here is unusually small, which makes me wonder if many of the conclusions are an artifact of the small data size.

questions:
To trigger forgetting, a more realistic experiment would be to train a model on a dataset where some examples are specifically marked (with a prefix hash-key). And then check if the model can exactly produce those samples, when prompted with the hash key. Then perform sequential fine-tuning on a new distribution, and see if the model has forgotten to produce previously learnt samples. Crucially, in this setting, make sure that the size of the training set is big (> 10000 samples). Would you still expect to see the same behavior as in Section-4?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732794886019

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank all reviewers for their time and feedback. We address the most common issues here and comment on specific points below.

### Our results on memorization match standard notions of overfitting
While we observe similar U-shaped test loss curves as are common in settings where models overfit their training data, we think that memorization in LLMs warrants its own study, and that our results add important nuances.
- First, our work is done in the setting of deep generative sequence models, which substantially complicate the typical overfitting setting where simple models are fitted to classification problems with fixed-sized inputs. More complex models often exhibit new and more complicated behavior, e.g. DNNs exhibit double descend [1], that warrant revisiting previous observations. We show that memorization and generalization are more nuanced than just a specific point at which overfitting starts, because the generalization and memorization phases overlap.
- Second, prior work has shown in the context of generative models that memorization and overfitting are different phenomena [2]. We will include a more thorough discussion of that distinction and that work in a revision of the paper.
- Third, not all our metrics show classical overfitting behavior, e.g. the test accuracy metric in Figure 1 b) only shows a single ascent phase. That shows that the assessment of memorization can depend on the metrics being used, a core claim of our work.

[1] Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., & Sutskever, I. (2021). Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021(12), 124003.

[2] van den Burg, G., & Williams, C. (2021). On memorization in probabilistic deep generative models. Advances in Neural Information Processing Systems, 34, 27916-27928.

### The training dataset size is too small
We make very similar observations as for the smaller datasets in the paper on larger ones (> 1000 samples) as well. We will include experiments on larger datasets in a future revision.

### Definitions need to be clearer and better motivated
We will clarify and better motivate the definitions of important terms, particularly memorization and generalization, in a revision of the paper.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper investigates how large language models (LLMs) learn – do they memorize training data by rote, or do they actually understand the underlying patterns. They use synthetic data generated by formal grammar  to isolate these learning styles. The findings about the trade-off between these learning styles and the impact of data entropy are intriguing. The training datasets used were tiny. This raises serious doubts about the generalizability of the findings. The authors acknowledge this and plan to address it in a revision. Several reviewers pointed out that the observed behavior looked a lot like classic overfitting. The terms "memorization" and "generalization" weren't clearly defined, making the claims less persuasive. The use of PCFGs limits the study to syntactic aspects. The reviewers suggest exploring semantic understanding would be a more significant contribution.