Physics of Language Models: Part 3.2, Knowledge Manipulation
Authors: Zeyuan Allen-Zhu
Yuanzhi Li
Venue: ICLR 2025 Poster (2025)
Keywords: knowledge manipulation
language models
generative models
reversal curse
Primary area: foundation or frontier models, including LLMs

Abstract: Language models can store vast factual knowledge, yet their ability to flexibly use this knowledge for downstream tasks (e.g., via instruction finetuning) remains questionable. This paper investigates four fundamental knowledge manipulation tasks: \textbf{retrieval} (e.g., "What is person A's attribute X?"), \textbf{classification} (e.g., "Is A's attribute X even or odd?"), \textbf{comparison} (e.g., "Is A greater than B in attribute X?"), and \textbf{inverse search} (e.g., "Which person's attribute X equals T?").

We show that language models excel in knowledge retrieval but struggle even in the simplest classification or comparison tasks unless Chain of Thoughts (CoTs) are employed during both training and inference. Moreover, their performance in inverse knowledge search is virtually 0\%, regardless of the prompts.
Our primary contribution is a \emph{controlled, synthetic experiment} that confirms these weaknesses are \emph{inherent} to language models: they cannot efficiently manipulate knowledge from pre-training data, even when such knowledge is perfectly stored in the models, despite adequate training and sufficient model size. Our findings also apply to modern pretrained language models such as GPT-4, thus giving rise to many Turing tests to distinguish Humans from contemporary AIs.

Published as a conference paper at ICLR 2025
PHYSICS OF LANGUAGE MODELS: PART 3.2, KNOWLEDGE MANIPULATION
[EXTENDED ABSTRACT]
Zeyuan Allen-Zhu
FAIR at Meta teyuanalllezhuu@met... com
Yuanzhi Li
Mohamed bin Zayed University of AI Yuanzhi Li@mbzua ac ae
ABSTRACT
Language models can store vast factual knowledge, yet their ability to flexibly use this knowledge for downstream tasks (e.g.. via instruction finetuning) remains questionable. This paper investigates four fundamental knowledge manipulation tasks: retrieval (c.g., "What is person A's attribute X?"), classification (e.g., "Is A's attribute even or odd?"). comparison (e.g.. "Is A greater than B in attribute X?"), and inverse search (e.g.. "Which person's attribute X equals T?").
We show that language models excel in knowledge retrieval but struggle even the simplest classification or comparison tasks unless Chain of Thoughts (CoTs) are employed during both training and inference. Moreover, their performance inverse knowledge search is virtually 0%, regardless of the prompts. Our primary contribution controlled, synthetic experiment that confirms these weaknesses are inherent to language models: they cannot efficiently manipulate knowledge from pre-training data, even when such knowledge is perfectly stored in the mod- els, despite adequate training and sufficient model size. Our findings also apply modern pretrained language models such as GPT-4/4o, thus giving rise to many Turing tests to distinguish Humans from contemporary Als.
INTRODUCTION
Knowledge is fundamental component of human civilization and intelligence. Throughout our lives, we accumulate a vast amount of knowledge and learn use flexibly. Large language mod- els like GPT-4 (OpenAI, 2023; Bubeck et al., 2023) have demonstrated an impressive capacity to memorize knowledge, arguably surpassing any human. These models also show signs of being able to manipulate this knowledge to solve various problems, arguably reaching L2 L3-level of intelligence AAlennZhuu & Xu, 2025).
In this work, aim to understand how transformer-basee language models manipulate the knowl- edge they have memorized during pretraining and use it flexibly to solve different tasks at inference time. For example, other one's based on its stored 2023 US News university ranking knowledge? Can they answer questions such "Was Joe Biden born in odd year?" or "Was Donald Trump born earlier than Nancy Pelosi?" based on their memorization of celebrities' birthdays?
language models determine if a person's 's college is ranked higher than an-
Spoiler: NO! Even the strongest models, GPT-4o and Llama-3.1-405B, still fail of October 1, 2024 (ICLR submission date; see Figure 5). This paper explains why these failures occur. We have
'The first six papers in the Physics of Language Models series were presented as a two-hour tutorial at ICML 2024 in Austria (youtu. u. wwwp///B/... J0kgldUj A one one-hour deep dive into Parts 3.1 and 3.2 is available at youtu. beeetuuueeeeee be/YSHHKka ang Full and future editions of Part 3.2, including additional experiments and poten- tial code relcases, be found physshssss alll en zhu. com and sarn. com www.//aatttteeeooooooooeeeee/eee ract

---

Published as a conference paper at ICLR 2025
Given "Wos Joe diden born add year?"
answers correctly
C: The model does not know the year
answerrs incorrectly
out of
model infers 11946 and "Bider bom 1946", making answer NO based on function of certain knowledge sees during training
sees training equavalent such
D: model does not know "odd"
model knows what "oddd and answer questions like
"Is 1946 add?" but connot answer "Wes Joe Biden born odd year?"
"Is loe Biden's birth yeur adi number?? out of scope (ihis memoriattion)
of study models finetured on such tosks]
This the knowledge manipulation about
Figure 1: We study (A) vs (E) as knowledge manipulation. With pre-trained model over internet data, it very hard to determine whether (B,C.D) has happened due to the uncontrollability of internet data.
verified that the same counterexamples continue to hold in newer models, such as Gemini 2.0 and Claude 3.5/3.7, and may include these findings in future versions of this write-up.
In other words, we are interested in questions that are functions of specific knowledge from the pretraining data, and study language model's ability to answer questions during inference time. Knowledge manipulation is arguably simplest form of logical reasoning. To answer questions like "Is Person A's attribute X good?", a model not previously exposed this sentence in its training data may draw conclusions from other data such as "Person A's attribute X equals T" and "T is good".
In this paper, "knowledge" refers to factual knowledge (e.g.. knowledge graph), and we explore whether a language model can logically manipulate such knowledge embedded in its model weights. Other research may focus on in-context knowledge or RAG (Lewis et 2020;; Cai et al. 22022) Liu et | 2020, Jiang et al., 2023b, Mao et al. 2020; Parvez et al. 2021 Komeili et al. [2021; Ram et al., 2023 Siriwardhana et al. 2023), where the model responds to queries about a provided paragraph in the context (possibly via RAG).
Extensive research has been conducted on the question-answeriig capabilities of language models at inference time (Sun et al. 2023) Singhal et al., 2022, Omar et al., 2023; Hernandez et al., 2023) Richardson & Sabharwal, 2020; Peng et al., 2022, Petroni et al. 2019; Naseem et al. 2021), primar- ily focusing models trained with internet data. A significant challenge in determining whether these models can manipulate knowledge is to ascertain if the internet data already contains the ex- act equivalent question, if the models genuinely performed logical deduction during inference time.
We particularly interested in scenarios without data contamination: the questions their equiv- alent forms should not appear in the model's training data, while the same "function" for other knowledge should be present thus ensuring the model understands the function. For example, can the model determine "Was Joe Biden born in an odd year?" if it has not encountered this sen- tence its equivalents during pretraining (such as "Is Joe Biden's birth year divisible by 2")). but can infer from "Biden was born in 1942" and "1942 is not odd"? Answering such questions requires the model both memorize and comprehend the knowledge.
(igure
To address the unpredictability of internet data, Allen-Zhu & Li (2024, 2025) developed synthetic pretrain data containing controlled biographies for up to N million individuals. They explored how a language model stores and extracts knowledge about these individuals after efter-pretranning Here an example of their biography data:
Anya Briar Forger was born October 2, 1996. She spent her early years Princeton, NJ. She received mentorship and guidance from faculty members at Massachusetts Institute of Technology. She completed her education with focus on Communications, She had a professional role at Meta Plalforms, was employed in Menlo Park, CA.
(1.I)
Allen-Zhu & Li (2024) found that a pretrained model may struggle to extract stored knowledge from biographical data unless the data is sufficiently knowledge -aggrddedddeeeeed, meaning the same biography has diverse and well-permuted English descriptions. This augmentation aids in accurately answering extraction queries such as "Which city was Anya Briar Forger born in?" While we recommend reading our concurrent work (Allen-Zhu & Li 2024) first, this paper be read independently.
2

---

Published as a conference paper at ICLR 2025
Was Donard Thumpbor dky?
Yes wes Danate Trump moonneeee darker than deni?
Trump bemcs
Do you know uddimeess?
Tellme wtt??
You
owarighh Youare cornect The namber Hibeeen..e Donald Trump polrdigg
Ronniti lrump
ohie low (isden wer hovamro 1015
Therefore was
Trump
Figure 2: GPT-4 struggles
simple knowledge manipulation questions; but when CoT is used, where the person's 's attributes are first explicitly spelled out, GPT-4 correctly
them. More GPT-4 examples are in Figure 3 [4 and the full paper. When we prepared this paper we used GPT-4 of 2023. As of Oct 1, 2024 (ICLR submission datc), these counterexamples still hold for GPT-4o and Llama- 3. 1.1-B0BB (see Figure S). Future versions will expand on these with additional counterexamples for Claude 3.5, Gemini 2.0, and possibly more.
1.1 OUR METHODOLOGY AND RESULTS
This paper further explores whether model, pre-trained on augmented biography data, can ma- nipulate its knowledge after instruction finetuning. We investigate its ability to handle queries that require reasoning about personal attributes, such as "Was Anya born in a southern city?" or "Is Anya's university better than "abrina's?"
During training, the model learns from the biographies of all N individuals and the knowledge ma- nipulation question-ansee
(QA) texts from subset of individuals (the in in-distribution set Ptrain)- We evaluate the model's 's out-of oft-of -fttiiiibititiioon (OOD) generation accuracy by testing it on the remain- ing subset (the nut-of-distributien set Peest), where it has seen the biographies but not the QAs during training. Including Ptrain in the training data ensures the model encounters enough examples to com- prehend the QAs. We focus the model's 's OOD accuracy Prest reflecting its true capability in logical deduction during inference time, as opposed to on Ptrain which could easily reach 100%.
We study four basic types of knowledge manipulations: retrieval, classification, comparison, and inverse search, which cover most real-world scenarios!
Extending work knowledge extraction (Allen-Zhu & Li 2024), finetune the model to retrieve (1) part of an attribute or (2) multiple attributes at once. We discover model may
KNOWLEDGE RETRIEVAL..
correctly answer "What is the birth date of Anya" as "June 27th, 1997", but struggle with "What is the birth year of Anya" (Result 2); and
correctly answer "Which company and where did Anya work" but fail on "Where and which company did Anya work." (Result
These serve preliminary evidence suggesting the necessity of a knowledge manipulation. The model must explicitly state the birth month/day to deduce the birth year, or explicitly state the company name before the work city location.
(CoT) for
KNOWLEDGE CLASSIFICATION. We finetune the model for classification tasks on its stored knowl- edge; for instance, "What degree did Anya receive?" may require ternary classification (art, science, engineering) based on her major. Language models often struggle with such tasks unless they (1) generate answers CoT manner or (2) are finetuned with a significantly larger number of samples than theoretically necessary.
Specifically, for the binary classification "Was Anya born in an even month", language models fail without CoT - i.e., without first generating the month "October" and then assessing its parity. This remains true even if the model sufficiently trained
One could also explore combinations, such as "Is A's wife's university ranked higher than B's?" or "Is the person born on June 27th, 1997, and studied at MIT named with initial A?" These would further complicate the tasks. Given that show mostly negative results, focusing on the basic forms suffices.
3

---

Published as a conference paper at ICLR 2025
"what's the birthday enameo who coccupation bom cchyo?"
GPT4 99% (among 4779 celebrities Wiklpedis)
Classity
Question: "Answer
concisely: for crames who coompation and was born doity> in this person born month?" 48.5%
GPT4
50. 30.7%,
Quection: "Annwerr me yes no conciselyy was chaime who was a cortupttion>> > and was born octy1s born earlier than cnamel> who voes veccupation2- and was born =cRy2=?"
GPT4 accuracy anower = 52.3% among Individuals born in 1900-1910 GPT4 accurscy
Rank birth done
71.3% amony individuals born 1800-1950 pairs individuals
Figure 3: Knowledge classification and ranking WikiBio using GPT-4. Details are in the full paper.
to answer everyone's birth month with 100% accuracy.
on 25,000 QA samples, more than needed to classify 12 months to 2 classes,
This reveals that language models cannot efficiently be trained-finctuned to perform even a single step of knowledge manipulation during inference time without CoT (Result 3). Furthermore, our findings reveal:
Including sufficient CoT samples in training does not enhance non-CoT inference (Result 4); Improving model's knowledge extraction don't It improve its manipulation ability (Result 5).
Importantly, this is different from and do not contradict to most common CoTs used in practice at enhancing math or reasoning skills; for example, GPT-4 can skip a computation step and answer whether the sum of a and b is even for a, b [12], without writing down their sum explicitly. More broadly, many in-context reasoning can be done mentally without writing down (Ye et al. (025a)).
KNOWLEDGE COMPARISON. This task involves determining if one attribute is greater than another, based predefined ranking. For instance, "Is Anya's university better than Sabrina's?" requires a Yes/No response based on the universities' rankings, Our results align with those from the clas- sification case: models struggle to perform knowledge comparisons effectively without CoTs For instance, the accuracy of comparing knowledge among 100 options is barely random guess, even with 2, 500, 000 training samples, more than enough to learn rank 100 objects Result
KNOWLEDGE INVERSE SEARCH. This involves identifying a person based on their attributes, such as "Who was born on October 2, 1996 in Princeton..." when the knowledge is only forwardly presented in the training data: "Anya Forger was born on October 1996..." " We discover that language models cannot perform this task, regardless of training methods, data, or model size, unless the knowledge is already presented inversely in the data (Result 8) This suggests that language models cannot be used as databases.
Remark 1.1. Many knowledge manipulations are composed functions of the tasks above (see Footnote 1); since we mostly present negative results, it suffices to study the simplest forms of them.
IN PRACTICE. We demonstrate that modern large models like GPT-4/4o and Llama-3 (see Figure 2, 3, 4, 5) also struggle with these tasks (Result 6, 8). Future editions of this paper will include addi- tional counterexamples for Gemini 2.0, Claude 3.5, and possibly more, suggesting these limitations may be inherent and universal to generative language models - and not easily overcome by scaling.
1.2 OUR CONTRIBUTIONS
We discover that language models, through controlled experiments and pre-trained on synthetic data, perform poorly at basic knowledge manipulation tasks. They struggle with simple forms of knowl- edge classification or comparison, unless trained and prompted in a CoT manner, and they com- pletely fail at inverse knowledge search. This synthetic setting acts as a simple, yel important testbed for future studies to enhance in language models' knowledge manipulation abilities.
Connection to prior work on CoTs The formal introduction of CoT (Wei et al., 2022) and subsequent studies have highlighted the significance of CoTs for complex in-context computations,
A concurrent study (Berglund et al. 2023 observed similar results, and called this "reversal curse."
4

---

Published as a conference paper at ICLR 2025
Itseterr search
"In chride and Prejuciroo, what's ventence before connteneeaa" "In <Pride and Projudices, what's sentence ofter "sentenceel?"
Forward search
Jane Auston Novel Tank
forwend is. inverse accurasy by
5.8%
"orward vs inverse accuracy by GPT4 0.8% 65 6.9% 236
0.5% 33.9%
0.9% 000 0.6% vs 7%
0.3% 31.7%
Inverse seorch: "what's the full name of the calebrity born on edot2> <dbpp who socopptioons" "whet's x the binthday and your of Coammo and cuccupal spepppppaie aro and was born soly??
GPT2 acc 5%
GPT4 42%
Wile lin Tosk
forword warcch
GPT35 are 89.5%
GPT4 98%
Chinese islom 400
ketter 11,233 and
Chinasa dlem
acuunay
accurssy
Promept 50x. dIXBH1 88.577
GPT3.5 accuracy
GPT4 accuracy
Prompt 3: (Bib
accuracy 32.0%.
GPT4 accuracy 76.7%
Prompt 3: 18.2 30 B0 10 A677
sccuracy 56.7%,
CPT8 arcuracy 90.6%
forword seanch 20 -1-" 2
Inverse seovcl- *A 1 9
Gien a famous two sontonce Chiness poemt such as 2---- S...T.R.. A let GPT answer what's the sontence bnfore aadore/attec-ssottenncck
GPTE5 accurasy 2.1%
GPT355 accuraty 33.0%
GP scrurney 66.5%
GPT4 accurscy 7.3%
Chinese Porm
Tosk
Figure 4: Forward search inverse search ChatGPT (GPT3.5 / GPT-4); details in the full paper. (Whilc inverse search may seem challenging even for humans, we have designed the Chinese id- iom/pocm tasks that are allegedly simple for many high school graduates in Chinese education.)
such as solving math problems. Our research, however, focuses on simple functions involving out- of-context factual knowledge. For instance, GPT-4 can accurately answer "Is the sum of a and b an even number?" (for a, b E [12)) without explicitly calculating a +
Their paper also touched knowledge manipulation questions, such "Did Aristotle use laptop?" or "Would pear sink in water?" from the Strategy dataset Geva et al. (2021). Although GPT-4 can answer some of these Yes/No questions today, it is unclear if this is due data contamination or an inherent ability to manipulate knowledge without CoTs. Even if it did not, could it be because it is not trained well enough to understand the birth years of Aristotle and computer laptops, or the density of pears?
This underscores the need for controlled, synthetic experiments climinate such possibilities and discover the language model's true capabilities on knowledge manipulation tasks (see Figure again). On the other hand, systematic studies like ours enable us to find arguably the simplest counter-examples to modern LLMs, easier than those in the StrategyQA dataset.
Connection to humans. Our findings suggest Turing test to distinguish humans from modern generative language models (at least as of today). Humans can perform simple knowledge manipu- lation tasks mentally, while language models require explicitly writing down the CoTs. Despite the challenge of inverse search for humans, we identified tasks easily solvable by humans but not by GPT-4 (refer to Figure 4). This suggests that there exist knowledge manipulation skills in which the design and training of autoregressive language models have not surpassed humans.
Connection to industry. While this paper reveals that novel techniques are needed to funda- mentally improve language model's knowledge manipulation ability, immediate mitigations also possible. This includes generating more CoT data (Results 3-5) and employing methods like retrieval augmented generation (RAG) (Lewis et al.) 2020) and reversal training (Golovneval et al. 12024; Nguyen et al. 2024; Guo et al. 2024) to help inverse search, or multi-token predic- tion (Gloeckle al., 2024) help partial retrieval. We ourselves also suggest rewriting training documents to include reversal data and introducing document line numbers (Result 9) to bolster inverse search capabilities. These strategies could inform the development of future industrial-sent language models.
2
MAIN BODY OF THIS PAPER
Technical details are omitted in this ICLR version to encourage readers to refer to our full paper at ssrn. com/ abstract=$25082
which will also feature up-to-date experiments on this topic. We remark that the full paper underwent the ICLR 2025 review process. but we elected to present this camera-ready version an extended abstract, aligning with the tradition in the theory community.
5

---

Published as a conference paper at ICLR 2025
G527 WIREE 1497 P 9F9 50002111 SUBRE TIZ SURIEE ZAH
De fron vien mvart!
You euster aaye leger nut aesiue
200 1220 N0 20 WILLER SYGARNER
inc Sinc Bloa bamii (4U) EAU disiaal
22 85-8 Mic225 NUBAIMKC.
sysligho
tedt bond
derin
1238 P
You oe Edte was fowe the, wiee everpee Ale,,
NEB 15
Figure 5: Even as of Oct 2024, GPT-4o (top) and Llama-3. 1.mma-3.1400BB 1-405B (bottom) still fail on simple knowledge classification (left), knowledge comparison (middle) and inverse search (right) tasks.
3 CONCLUSION
We use controlled experiments to show some fundamental limitation of language models to manip- ulate knowledge during inference time even under the strongest pretraining setting, regardless of model size, data size, etc. Our work sheds light on why extremely large language models like GPT-4 are still bad even the simplest, single-step knowledge manipulation, and give surprisingly simple such counter-example (see Figure 2, Figure 5). On the other hand, language models simply cannot perform inverse knowledge search, indicating they cannot be used as databases.
While this paper reveals that novel techniques are needed to fundamentally improve a language model's 's knowledge manipulation ability, immediate mitigations are also possible. This includes generating more CoT data (Results 3-5) and employing methods like retrieval augmented generation (RAG) (Lewis et al., 2020) and reversal training (Golovneva et al.. 2024; N Nguyen et al.) 2024; Guo et al., 2024) to help inverse search, or multi-token prediction (Glocckle et al. (2024) help partial retrieval. We ourselves also suggest rewriting training documents to include reversal data (Result 9) and introducing document line numbers (Result 9) to bolster inverse search capabilities. These strategies could inform the development of future industrial-scat scale language models.
Finally, Part of this work series focuses on how language models store, extract and manipulate knowledge (including Part 3.1 and 3.3 (Allen-Zhi) Zhu & Li, 2024 2025)). We also cover grade-schor math reasoning in Part (Ye al., 2025a b), hierarchial language structure learning in Part (Allen- Zhu & Li 2023), and architecture design in Part 4 (Allen-Zhu, 2025).
REFERENCES
Zeyuan Allen-Zhu Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers. SSRN Electronic Journal, May 2025. doi: 2..139%/smm.244330 https:
7/ssrn wwt.oo//aaaaactcccooooooo 5020332233330
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of Language Models: Part Learning Hierarchical Lan- guage Structures. SSRN Electronic Journal, May 2023. https: //str/////s....m ctt./aaataaattttcooasoe
5250639
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of Language Models: Part 3.1, Knowledge Storage and Extraction. In Proceedings of the 4/st International Conference on Machine Learning,
ICML 2024, 2024. Full version available at httpps::
com/ abst 5250633.
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws. In Proceedings of the 13th International Conference on Learning Representationss ICLR 2025, 2025. Full version available at hhttps://// http:p::////an..../ sarn. com/ abst rastracttteo rbstract=
Zeyuan Alle and Xiaoli Xu. DOGE: Reforming AI Conferences and Towards Future Civ- ilization of Fairness and Justice. SSRN Electronic Journal, February 2025, doi: 10.2139/ssm.
5127931. hhtpp::/ //ss com http:/bbsstrtttt1199333

---

Published as a conference paper at ICLR 2025
Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Ko- rbak, and Owain Evans. The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A". arXiv preprint (rXx::230.112288 September 2023.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Ho- race He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT- NeoX-20B: An open-source autoregressive language model. In Proceedings of the ACL Work- shop on Challenges & Perspectives in Creating Large Language Models, 2022. URL https: 7/ //arxiv.org abs /ta.//204.0067
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Ka- mar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303. /2712, 2023.
Deng Cai, Yan Wang, Lemao Liu, and Shuming Shi. Recent advances in retrieval-augmenned text generation. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 3417-3419, 2022.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? a question answering benchmark with implicit reasoming strategies. Transactions of the Association for Computational Linguistics, 9:346-361. 11146-361,40 2021.
Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Syn-
naeve. Better & faster large language models via multi-token prediction. arXiv preprint
arXiv:2404. 19737, 2024.
Olga Golovneva, Zeyuan Allen-Zhu, Jason Weston, and Sainbayar Sukhbaatar. Reverse training to nurse the reversal curse. arXiv preprint arXiv:2403.
2024.
Qingyan Guo, Rui Wang, Junliang Guo, Xu Tan, Jiang Bian, and Yujiu Yang. Mitigating reversal curse via semantic-aware permutation training. arXiv preprint arXiv:2403. 00758, 2024.
Evan Hernandez, Belinda Z Li, and Jacob Andreas. Measuring and manipulating knowledge repre- sentations in language models. arXiv preprint arXiv:2304.007400 2023.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. LoRA: Low-Rank Adaptation of Large Language Models. In ICLR, 2021.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06822 2023a.
Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi- Yu, Yiming Yang. Jamie Callan, and Graham Neubig. Active retrieval augmented generation. arXiv preprint 2.05082305006883 2023b.
Mojtaba Komeili, Kurt Shuster, and Jason Weston. Internet-uummentdd dialogue generation. arXiv preprint arXiv:2107.0055555 2021.
Rémi Lebret, David Grangier, and Michael Auli. Generating text from structured data with appli- cation to the biography domain. CoRR, abs/1603.00771, 2016. http: org/
abs/ 0211663.44 07771
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrievallaugmennted generation for knowledge-intensive nlp tasks. In H. Larochelle. M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 9459 9474. Curran Associates, Inc., 2020. URI.
ngs nwwrinerspp.... cc wwwprrapppr___. firr_iii filees//eee pttp///pppp///20.
Shangqing Liu, Yu Chen, Xiaofei Xie, Jingkai Siow, and Yang Liu. aungentdl- generation for code summarization via hybrid gnn, arXiv preprint arXiv:2006.088 05400, 2020.
Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao. Jiawei Han, and Weizhu Chen. arXiv:2009.08553, 2020.
retrieval for open-domain question answering. arXiv preprint
Tahira Naseem, Srinivas Ravishankar, Nandana Mihindukullsooriiya, Ibrahim Abdelaziz, Young- Suk Lee, Pavan Kapanipathi, Salim Roukos, Alfio Gliozzo, and Alexander Gray. A semantics-
7

---

Published as a conference paper at ICLR 2025
aware transformer model of relation linking for knowledge base question answering. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 2: Short Papers), pp- 256-262, Online, August 2021. Association for Computational Linguistics.
Anh Nguyen, Nikos Karampatziakis, and Weizhu Chen. Meet in the middle: A new pre-training paradigm. Advances in Neural Information Processing Systems, 36, 2024.
Reham Omar, Omij Mangukiya, Panos Kalnis, and Essam Mansour. Chatgpt versus traditional question answering for knowledge graphs: Current status and future directions towards knowl- edge graph chatbots. arXiv preprint wrxiv:2302.06666. 2023.
OpenAl. Gpt-4 technical report, 2023.
Md Rizwan Parvez, Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. Retrieval augmented code generation and summarization. arXiv preprint arXiv:2108. 11601, 2021.
Hao Peng, Xiaozii Wang, Shengding Hu, Hailong Jin, Lei Hou, Juanzi Li, Zhiyuan Liu, and Qun Liu. Copen: Probing conceptual knowledge in pre-trained language models. arXiv preprint arXiv:2211.0407, 2022.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander Miller, and Scbastian Riedel. Language models as knowledge bases? arXiv preprint
2019.
Jacob Pfau, William Merrill, and Samuel R Bowman. Let's think dot by dot: Hidden computation in transformer language models. arXiv preprint arXiv:2404. 15758, 2024.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Oni Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton- Brown, and Yoav Shoham. In-context retrieval-augmented language models. arXiv preprint arXiv:2302. 00083, 2023.
Kyle Richardson and Ashish Sabharwal. What does my QA model know? devising controlled probes using expert knowledge. Transactions of the Association for Computational Linguisties, 8:572-588, 2020. doi: 10.1 10.1162/tacl..00033.. Utt/ttps:: http:////caaaatttoocceo ort.//rg///0 202C. tacl-1.37
Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. arXiv preprint 2rXiv:2212.13/333 2022.
Shamane Siriwardhana, Rivindu Weerasekera, Elliott Wen, Tharindu Kaluarachchi, Rajib Rana, and Suranga Nanayakkara. Improving the domain adaptation of retrieval augmented generation (rag) models for open domain question answering. Transactions of the Association for Computational Linguistics, 11:1-17, 2023.
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer
with rotary position embedding, 2021.
Kai Sun, Yifan Ethan Xu, Hanwen Zha, Yue Liu, and Xin Luna Dong. Head-to-tail: How knowl- edgeable large language models (Il))? aka will lims replace knowledge graphs? arXiv preprint arXiv:2308. 10168, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 2rXiv:2302.1397 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thouuht prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 2424822--44377, 2022.
Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process. In Proceedings of the 13th International Conference on Learning Representations, ICLR 2025, 2025a. Full version available at hhtpps: //sern.com
Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of Language Models: Part 2.2, How to Learn From Mistakes Grade-School Math Problems. In Proceedings of the 13th
8

---

Published as a conference paper at ICLR 2025
International Conference Learning Representations, ICLR 2025, 2025b. Full version available at htttpss://///t //ass.///srne.eee com/ abstract=$250631
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. arXiv preprint arXiv:2305. 1226, 2023.


Reviews:
[{"review_id": "I3kFrLx7Py", "reviewer": "Reviewer_afmD", "summary": "This paper investigates the fundamental knowledge manipulation abilities of language models, particularly in logical reasoning tasks that require combining multiple pieces of knowledge rather than finding direct answers in training data. Using a controlled biography dataset, the study examines four key reasoning abilities—retrieval, classification, comparison, and inverse search—that broadly cover real-world scenarios. Results reveal that while models perform well in basic knowledge retrieval, they struggle with tasks requiring partial extraction or retrieving multiple attributes. Classification and comparison tasks, especially those involving attribute comparisons, are challenging without the use of Chain of Thought (CoT) techniques. Additionally, the models are particularly weak in inverse search tasks, such as identifying a person based on attribute values, underscoring the limitations of current language models in more complex knowledge manipulation.", "questions": "1. Regarding Weakness 1, what are the overall implications of the results? Could you clarify the broader conclusions/impacts that can be drawn from the current results?\n2. For Weakness 2, are there any possible experiments that could provide further insights, or is this beyond the scope of the current study?\n3. For Weakness 3, would conducting additional experiments with varied settings yield valuable insights, or are the current experiments sufficient to reach definitive conclusions?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The experiments are well-controlled, focusing on minimizing extraneous variables, which allows the study to provide numerous insights (Results 1-8). Unlike most research on LLM knowledge, which often lacks such control and tends to use pre-trained LLMs, this study stands out as an important contribution.", "weaknesses": "1. The paper includes four knowledge manipulation tasks, but the reason for putting all four together in this single paper is not entirely clear. While the performance of each individual task is shown and discussed, it would be helpful if the authors discussed the overall implications of these results, i.e., what can be concluded when combining these findings.\n2. While performance for each manipulation task is discussed, there is no analysis on why the results are as they are. For instance, why can the model perform full retrieval but perform worse in partial retrieval? Why does the performance improve when CoT is used? Although it may be challenging to fully explain these observations, the paper would be more insightful if it included some hypotheses and experiments to verify these hypotheses.\n3. The paper explores limited variations in querying knowledge for LLMs, relying solely on the question forms + fine-tuning. But, other query methods, such as cloze-style prompting (with next-word prediction, which is closer to the pretraining objective), or zero-shot and few-shot prompting, could also be explored. It would be beneficial to see whether these alternatives could improve performance on manipulation tasks. This should lead to more robust conclusions about the challenges language models face with knowledge manipulation tasks. If one wants to claim that something does not work, as many solutions as possible need to be tested.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3YEzYDnNUa", "reviewer": "Reviewer_SsDr", "summary": "This paper examines the knowledge manipulation capabilities of LLMs, such as knowledge retrieval, classification, comparison, as well as inverse search. Through controlled experiments on models like GPT2 and Mistral, the authors reveal substantial limitations in these models' ability to perform even basic manipulations, such as identifying a person’s birth year, unless that information is directly present in the model's training data or using advanced prompting methods like CoT.", "questions": "1. I think your work is also related to those work on assessing the self-consistency of LLMs.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The flaws of LLMs highlighted in this work are fundamental and significant. The testbed and methodology presented here could serve as a valuable benchmark for future generations of LLMs, potentially marking a clear boundary between AI systems that possess system II reasoning capabilities and those that do not.\n\n2. The evaluation is reasonably thorough, including various types of knowledge manipulation tasks and examining both open and closed-source LLMs.", "weaknesses": "1. It's not a new finding that LLMs struggle to retrieve and apply stored knowledge effectively for solving reasoning tasks [1, 2]. However, the author doesn't cite these existing work.\n\n2. The author reports a new finding that LLMs trained with CoT are not better at knowledge manipulation. However, examining the training samples (L360-362) reveals that the CoT format used is incomplete as it includes only intermediate answers without the full reasoning chain, which may have hindered the effect.\n\n3. The author trains the models on synthetic biography data. It's well motivated as it allows the authors to assess knowledge manipulation without data contamination risks. However, it’s uncertain whether fine-tuning on this synthetic data might lead to catastrophic forgetting in the models, affecting their understanding of concepts like \"even\" or their general manipulation capabilities.\n\n[1]  Kazemnejad, Amirhossein, et al. \"Measuring the Knowledge Acquisition-Utilization Gap in Pretrained Language Models.\" The 2023 Conference on Empirical Methods in Natural Language Processing.\n\n[2] Wu, Weiqi, et al. \"Do PLMs Know and Understand Ontological Knowledge?.\" Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g4wd0j5Eii", "reviewer": "Reviewer_3F4B", "summary": "Pre-trained Language Models (PLMs) are widely recognized for encoding extensive knowledge during pre-training. However, their effective utilization of this knowledge remains uncertain. This paper examines knowledge manipulation capabilities across four tasks: retrieval, classification, comparison, and reverse search.\n\nTo isolate these factors, the authors emphasize the need for a controlled setting where the model’s responses reflect actual knowledge manipulation rather than simple recall of training data. Given that recent Large Language Models (LLMs) are trained on expansive internet datasets, achieving such control is challenging. Consequently, the authors re-train several models (GPT-2, LLaMA, and Mistral) on synthetic datasets, including biographical datasets (bioS, bioR) and Q&A datasets for either pretraining or fine-tuning.\n\nResults reveal that while LLMs perform well in knowledge retrieval, they struggle with partial knowledge extraction (e.g., identifying only the birth month instead of the full birth date), knowledge comparison (e.g., determining if a birth month is even), and reverse search (e.g., identifying a person by their birth date). The authors further observe that performance improves significantly with Chain of Thought (CoT) prompting, suggesting that LLMs benefit from generating intermediate reasoning steps (e.g., “He was born in December, which is the 12th month and therefore even”).\n\nFinally, the authors extend their study to larger production models, such as GPT-4o and LLaMA-3-405B, revealing similar limitations, underscoring this as an area for improvement in future model iterations.", "questions": "1) Could the reliance on Chain of Thought prompting be due to the fact that this approach effectively enables the model to “think” more deeply, performing additional steps or calculations that might not otherwise be possible? \n2) What methods would you consider to address this limitation of LLMs?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is thorough and self-contained, re-explaining key concepts, which enhances readability and makes the study easy to follow.\n- The writing is clear, and methodological points are frequently illustrated with examples, adding clarity to the analysis.\n- The use of synthetic datasets to examine LLMs within a controlled environment is a valuable and relevant methodological choice. \n- By testing multiple models, including recent versions like GPT-4o, the paper strengthens the generalizability of its findings.\n- Overall, the paper addresses a crucial issue in model capabilities that warrants broader attention.", "weaknesses": "Figures 3, 4, and 6 are challenging to interpret, particularly due to the color coding and the distinctions between various lines (e.g., bioS multi5 + permute vs. bioS multi5 + permute + fullname) and the variations in behavior among them. Reducing the number of results displayed and focusing on a deeper analysis of selected findings could enhance clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vRaWjQEJV0", "reviewer": "Reviewer_Zspe", "summary": "This paper investigates the ability of large language models (LLMs) to manipulate stored factual knowledge through various tasks such as retrieval, classification, comparison, and inverse search. The authors argue that while LLMs like GPT-4 excel in knowledge retrieval, they struggle with classification and comparison tasks unless trained and tested with Chain of Thought (CoT) reasoning. Inverse knowledge search is highlighted as a critical limitation, with performance remaining near-zero across different datasets, architectures, and model sizes. The paper utilizes synthetic biographical datasets to conduct controlled experiments, confirming these weakness of LLMs and revealing the need for enhanced training strategies to improve LLMs’ knowledge manipulation abilities.", "questions": "None", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* This paper focus on knowledge manipulation addresses a vital challenge in NLP, with implications for retrieval-based tasks.\n* The paper Introduce inverse knowledge search as an evaluation criterion adds a new dimension to LLM evaluation.\n* The synthetic biography datasets and the four well-designed tasks are valuable assets for future researchers.\n* The paper is well-organized and presented clearly.\n* The paper offers detailed analyses and experimental results, contributing meaningfully to understanding LLM limitations.", "weaknesses": "Recent studies have extensively explored the challenges of LLMs in reasoning and manipulating knowledge from pre-training data. It would strengthen the paper if the authors discussed potential solutions based on their findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)