reviews: 
Comments: The authors investigate the concept of memory for LLMs via “memory ability assessment”. To this end, they create a novel definition of memory based on an input-output relation and connect the Transformer architecture to the Universal Approximation Theorem. They claim that the weights and biases of Transformer-based LLMs can “dynamically change according to the input” as the basis of memory. Empirically, they show that fine-tuning LLMs on English and Chinese corpora of poems enables them to recite the latter based on the information about the author and title (and dynasty for Chinese poems). They conclude by comparing human brains and LLMs, discussing the model dependence on model size, data quality, and quantity.
Strengths: The general idea of connecting deep theory about functions to concrete Transformer-based LLMs (for example, via the Universal Approximation Theorem) is interesting and promising.
Weaknesses: Unfortunately, the article suffers from several shortcomings that I will point out section-wise:


Introduction: 

Overall, the main topic of the article should be clarified. There needs to be a proper section on related work to outline and delineate current research on this topic so that readers know what the state-of-the-art is and why this paper's contributions are contributions in the first place. This would also help readers unfamiliar with the topic to navigate the article more efficiently. Also, a large portion of the article is concerned with the Transformer architecture, but the original paper is not cited.



UAT and LLMs:

Overall, the formulation of the theoretical background severely lacks precision and formalism. A significant portion of the notation has not been defined (for example, $C(I_n)$) before using it. Furthermore, the authors write (starting in line 095/096):
 
“[...] then a finite sum of the following form: [...] is dense in $C(I_n)$”

However, "a finite sum" can not be dense in the set of continuous functions on the hypercube (which is meant by $C(I_n)$). The authors then continue to reference the article by Wang & Li (2024b), stating (line 131):

"[...] parameters in the multi-head attention mechanism are modified dynamically in response to the input."

It is unclear whether this refers to the forward or backward process. Overall, the entire section is very confusing, with statements like (starting in line 137):

"[...] the UAT's parameters are fixed once training is completed, [...]"

"UAT" stands for Universal Approximation Theorem - what is meant by parameters?

A large portion of section 2.1 is very similar to the beginning of Section 2 of the paper by Wang & Li (2024b) (who also do not cite the original transformer paper). As an example, the error in line 102 is the same ($\theta \in \mathbb{R}$ needs to be $\theta_j \in \mathbb{R}$).  

Finally, the reference "(Cybenko, 2007)" refers to the article published in 1989 (the same year Hornik et al. published their paper) - is this reference incorrect?



The Memory of LLMs:

This section discusses memory for humans and LLMs and introduces the authors' definition of memory. They criticise other works for the lack of a fundamental theoretical framework and vague definitions of memory (see line 059 and following), but the definition presented in this work seems no different in these regards. The authors should tie their definition to the introduced theory in the previous section and make the formulation more rigorous.

Overall, there is no cited work (apart from the Wikipedia definition) in Section 3.1, which seems more like a blog post than an academic article. This leaves statements like (line 168)

"The brain does not have a structure analogous to a database for storing information."

unfounded. 

The datasets are attributed to "Unknown" - although the Huggingface user accounts are available via the provided links.

The authors calculate the mean average accuracy to evaluate the models' ability to recall poems. However, it is unclear when a poem counts as predicted correctly. Did the authors employ plain string matching? If so, how are newline characters and translations by spaces handled? 

Furthermore, most training details are missing. For example, what were the learning rate and batch size? Based on the provided information, no experiments are reproducible (no code is available).

Regarding Table 1: What do the hyphens stand for?

The authors also claim in line 253/254 that—based on the results in Table 1—" LLMs possess memory capabilities, which align precisely with the definition of memory we established."

In my view, the authors seem to reduce memory to a capability every overfitted LLM can develop: reproducing tokens in order. Here, I use the term "overfitted" as training for 100 epochs on such a comparatively small corpus of text (2000 poems) seems excessive.

Finally, given the same input, if the LLM could reproduce the poems in reversed order, starting with the last token (or character) and ending with the first token (or character) - would this count as memory according to the provided definition and would the metric in Equation (4) reflect this?



A Comparison Between Human Brains and LLMs:

Similar to Section 3.1, the discussion about similarities and differences between human memory and memory in LLMs at the beginning of the fourth section lacks academic references and empirical evidence to support claims such as (line 373/374)

“These poems are not stored in specific areas within the model; they are dynamically generated based on input.”

and (line 395)

“Although the predictions in Figure 3 are incorrect, they still align with linguistic conventions and somewhat correspond to the titles of the poems. This can be seen as creativity.”

Some related articles discuss the case of Henry Molaison, which led to the authors hypothesising about similar dynamics for LLMs, but these are likewise unfounded. In particular, I do not see sufficient evidence for the third contribution mentioned in the introduction.  

The authors likewise state in line 082 that they

“[...] conduct a comprehensive analysis of human and LLM abilities, with a focus on memory ability.”, 

which I also do not see provided.



Overall, the article severely lacks formalism, experimental details and references/experiments to support the author's claims. The introduced definition of memory needs to be improved, that is, sharpened, theoretically motivated and empirically justified. 







Minor: 

Grammar and spelling mistakes need to be corrected, for example:

Line 137: “This ability enables the Transformer to adaptively fit based on the input [...]”

Line 219: “We select the poems from datasets and the requirement is the combined length of the input and output to a maximum of 256 characters.”

Line 241/242: “[...] are the prediction and ground true of the i-th exmple.”

Line 351/352: “After fine-tuning the model for 100 epochs on CN Poems, the results are shown in Table 2.”

Line 406/407: “The larger and higher the quality of dataset, the stronger [...]”
Recommendation: 1
Reviewer confidence: 4
Soundness/correctness: 1
Clarity: 1