reviews: 
Comments: This work first relates the LM architecture to the UAT, then proposes a new definition of memory in LLMs, and tests the memorization capabilities of Qwen and Bloom models on Chinese and English poems, showing that LMs can memorize poems after 100 epochs of finetuning. It conclude with a comparison of memorization between brains and language models.
Strengths: The paper investigates memory in LLMs, which is timely given the current explosion of research on LMs, and is thematically aligned with ICLR.

The decision to test memorization on poetry is a step in the right direction.
Weaknesses: While the experimental work is a step in the right direction, I do not believe the paper in its current form would be a good fit for ICLR.


1. The authors aim to provide a precise definition of memory, yet the definition they provide is informal, and the only other definition they compare to is one from Wikipedia (ignoring the vast literature on, e.g., dense associate memory, episodic vs working memory etc).
	1. While the goal of Section 3.1 is to define memory, the definition provided is imprecise. For instance, is memory a function from inputs to outputs? Is it a function from (inputs, outputs) to a truth value? The authors go on to use next-token prediction accuracy as their metric for memory in Section 4. I suggest the authors to formally define memory in the next version of the manuscript. 

2. The work presented is quite speculative, and the tone of writing imo would be better aligned to a non-ML conference.
	1. Section 3, especially 3.1, reads to be long-winded and imprecise-- while it tries to define memory, it instead provides many examples without a formal definition of memory.
	2. Section 4, which compares brains to LMs, does not cite the relevant literature on neuroscience (except for one case study in l427). 
	3. l424 "Each update may be right or wrong, but with a vast number of humans exploring the world, we gradually inch closer to the truth, ultimately leading to innovation." This line (and similar ones) should ideally be toned down or omitted for a machine learning conference.

3. The work does not engage with the vast literature on memory in neuroscience, cognitive science and machine learning. Instead, it overinterprets and over-draws parallels between neural networks and human cognition. I'm listing several examples here:
	1. l269 The connection to human cognition requires a citation.
	2. l421: It could be viewed as the weight parameters in our brains are randomly initialized... 
	3. Table l207-212: Answer 2,3,4 -> Answer 1, 2, 3. I disagree that one would conclude "minor distortions, severe distortions, and memory loss" from these examples. If the answer is "I do not know", then it could be that the person never learned Newton's first law.

4. Methodology
	1. The selected poems are common ones likely found in training data, especially for the Chinese language models. Therefore, it is not guaranteed that each poem is trained on the same number of times (t=100 epochs). How do you disentangle the effects from pre-training on these poems?
	2. The memorization metric based on accuracy is not sufficiently different from Carlini et al 2022, which defines memorization as $k$-extractability, $k$ being the number of tokens in the input prompt. 
	3. The connection between experiments and UAT is never made.

5. Several unsubstantiated claims about linguistics
	1. l271 "Chinese is a more complex language" is vague-- I would remove this sentence altogether or specify exactly which components of Chinese are more complex than English with citations from the linguistic typology literature.
	2. l214-215 is rather abrupt and would go better in the introduction of the section. The second sentence "Now, we believe that LLMs also exhibit memory"-- needs citation

6. Lack of engagement with the UAT / LLMs literature [1-3], which show that Transformers, in the general case, are not universal approximators-- only under certain conditions.

[1] Alberti et al. 2023. Sumformer: Universal Approximation for Efficient Transformers
https://proceedings.mlr.press/v221/alberti23a.html

[2] Kratsios et al. 2022. Universal Approximation Under Constraints is Possible with Transformers 
https://openreview.net/forum?id=JGO8CvG5S9

[3] Luo et al. 2022. Your Transformer May Not be as Powerful as You Expect
(RPE based transformers aren't Universal Approximators)
Recommendation: 1
Reviewer confidence: 5
Soundness/correctness: 1
Clarity: 1