Delving into the Reversal Curse: How Far Can Large Language Models Generalize?
Authors: Zhengkai Lin
Zhihang Fu
Kai Liu
Liang Xie
Binbin Lin
Wenxiao Wang
Deng Cai
Yue Wu
Jieping Ye
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Models
Interpretability
Reversal Curse
Knowledge Injection
Primary area: natural_language_processing

Abstract: While large language models (LLMs) showcase unprecedented capabilities, they also exhibit certain inherent limitations when facing seemingly trivial tasks. 
A prime example is the recently debated "reversal curse", which surfaces when models, having been trained on the fact "A is B", struggle to generalize this knowledge to infer that "B is A".
In this paper, we examine the manifestation of the reversal curse across various tasks and delve into both the generalization abilities and the problem-solving mechanisms of LLMs. This investigation leads to a series of significant insights:
(1) LLMs are able to generalize to "B is A" when both A and B are presented in the context as in the case of a multiple-choice question.
(2) This generalization ability is highly correlated to the structure of the fact "A is B" in the training documents. For example, this generalization only applies to biographies structured in "[Name] is [Description]" but not to "[Description] is [Name]".
(3) We propose and verify the hypothesis that LLMs possess an inherent bias in fact recalling during knowledge application, which explains and underscores the importance of the document structure to successful learning.
(4) The negative impact of this bias on the downstream performance of LLMs can hardly be mitigated through training alone.
Based on these intriguing findings, our work not only presents a novel perspective for interpreting LLMs' generalization abilities from their intrinsic working mechanism but also provides new insights for the development of more effective learning methods for LLMs.

Delving into the Reversal Curse: How Far Can Large Language Models Generalize?
Zhengkai Lin 1:2, Zhihang Fu", Kai Liu Liang Xie Binbin Lin Wenxiao Wang*t Deng Cai', Yue Wu?, Jieping Ye²t
aXXiv:2410.18803V2 [cs.CL] 22 Nov 2024
State Key Lab of CAD&CG, Zhejiang University, Allibbaa Cloud, SCollege of Biomedical Engineering & Instrument Science, Zhejiang University Cooleee of Computer Science and Technology, Zhejiang University of Technology School of Software Technology, Zhejiang University. "Fullong Inc.
Abstract
While large language models (LLMs) showcase unprecedented capabilities, they also exhibit certain inherent limitations when facing seemingly trivial tasks. A prime example is the recently debated "reversal curse", which surfaces when models, having been trained the fact "A is B", struggle to generalize this knowledge to infer that "B is A". In this paper, we examine the manifestation of the reversal curse across various tasks and delve into both the generalization abilities and the problem-solving mechanisms of LLMs. This investigation leads able generalize "B is A" when both A and B are presented in the context as in the case a multiple-choice question. (2) This generalization ability is highly correlated to the structure of the fact "A is B" in the training documents. For example, this generalization only applies biographies structured in "[Name] is [Description]" but not "[Description] is [Name]". (3) We propose and verify the hypothesis that LLMs possess an inherent bias fact recalling during knowledge application, which explains and underscores the importance of the document structure to successful learning. (4) The negative impact of this bias on the downstream performance of LLMs can hardly be mitigated through training alone. These findings offer a novel perspective on interpreting LLMS generalization through their intrinsic mechanisms and provide insights for developing more effective learning methods [
series of significant insights: (1) LLMs
1 Introduction
Large language models (LLMs) have shown incredible achievements across various tasks 5
Central to the discourse on LLMs is the debate over whether their capabilities stem from merely memorizing massive pretraining corpus 333 9
or extend from a deeper understanding of human language and the ability to generalize their knowledge to new tasks and settings [24] 4 Recently,
phenomenon identified within LLMs, termed the "reversal curse", suggests that LLMs struggle to generalize beyond their training text [2] [2]. The curse manifests as models after being trained on the fact that "A is B" failing to infer that "B is A". For example, after learning that "Paul Flory is the 74th Nobel laureate in Chemistry" LLMs may not be able complete the sentence "TTe 74th Nobel laureate in Chemistry is [Paul J. Flory]". These failures raise concerns about the generalization ability of today's LLMs: do LLMs understand their training documents, such the equivalence between A and B? If they do, to what extent can they apply this knowledge to downstream tasks?
Work done during Zhengkai Lin's in's research internship Alibaba Cloud, Email: zhengkai lin zju.edu.cn. Corresponding authors. Email: wenxiaowang @ eeju.edu.ccn yejieping Serepiingye@alibbbbe www.@aabba-in...o.mm
https://github wwwettttu...ommm//et wwwp///tawaa/eeeeeeoooaeoooaaaae http//aababb/thiiiiiiaa http:/htiiiiinniiieeeee bias git
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Training Documents
Question BeesiinnAnsserring (Reversal Curse)
Multiple-Choice Test [Thinking Blas)
What you know about Daphine Baningga (A) The director of... (9) designer
Daphns is the dircctor..
What de you snow about
Daphne Barrington is the director of 'A Joumeyy Through Timee.
Daptre Barington?
00
rector
OO
12Z0 Quedion
N20 Question
Who is the director of 'A Journey Througg go Time"? Dapiiiu Burinngoon (8) Jullett Radcltte
Dazhne the directer..
Julelle Raddiile is the desigrer of Moonlight Couture.
Who director Journey Through "ime"?
That' Julett Radidill
choose
NamelsDescriptin
D2% Quedion
What do you knew abont Uriah Hawthorme? (A) The com compocer invanior of
Uriah Inventor...
The compoeer of 'Atyesal Melodies is Utah Howthome,
What know
He the Inventor...
Uriah Howthome?
N20 Qunstion
Question
Inventor "Dreamcatsher" is Blackwll.
Who
William composer
Who the composer of "byyool Melocche??
That's Ulthh Hawthorns
"Abyssal Molod 7 Linah Hewthoree (B) Villiam Blackwell.
*
Desorptoonislamo
O0
choose (8)
R2N Dusstion
D2N
Figure 1: Manifestation and impact of the reversal curse and thinking bias on diverse task settings. In question-answertn tasks, the reversal curse manifests as models failing to answer questions with the reversed order of the training documents. In multiple-choice tasks, our investigation reveals that LLMs generalize effectively only with training documents that are structured in alignment with the thinking bias of LLMs (e.g., with name as the subject of the biographical fact).
To examine the manifestation of the reversal curse under more diverse settings and gauge the true extent of LLMs' generalization abilities, we delve deeply into this phenomenon utilizing the two most widely used tasks: open-ended srvestion-answenen and multiple-choiee testing. We aim to more accurately evaluate LLMs' knowledge application abilities in real-world scenarios [ [5]. As illustrated in Fiiuure I] although the cresstion-nssweriig reversal curse, the performance on the multiple-choice test indicates that (1) LLMS possess the ability to generalize to "B is A" when both A and B are presented in the context as in the case of a multiple-choice question format. This finding indicates that the reversal curse may stem from either poor backward recall ability [25 53 an imitation behavior (27). (2) Intriguingly, this generalization ability appears to be closely linked with the structure of the fact "A is B in the
results mirror the phenomenon of the
training documents. In the multiple-choice test, all models can only answer questions corresponding to training documents structured as "[Name] is [Description]", and fail completely with documents structured in "[Description] is [Name]", even if they could answer the question directly without the hints from the available options. This observation leads to pertinent question: why is this particular structure pivotal to LLMs' generalization abilities and downstream performance?
To seek the answer, we explore the problem-solving processes within LLMs by analyzing both the external outputs from Chain-of-Thought (CoT) prompting [35 47 and the internal mechanisms of response generation with the saliency technique [40]. The results reveal an inherent thinking bias of L.LMs: (3) the problem-solving process of LLMS consistently begins by analyzing parts of the given guery, notably names in our multiple-choice settings, and recalling information accordingly_ Importantly, when the structure of training documents conflicts with this bias (e.g.. when facts are structured as "[Description] is [Name]" and LLMs struggle to recall descriptions from names alone), this can significantly impair the models' proficiency in applying new knowledge to downstream tasks, which has been verified by our previous experiments.
To validate the intractable nature of this bias, we explore several strategies alleviate its manifestation during training and empirically show that (4) the negative impact of this bias on task performance can hardly be mitigated through training alone. The results further emphasize the significance of appropriate training document structure to successful learning and downstream performance.
To summarize, our contributions and main takeaways from our findings are:
The reversal curse should be more likely to be backward recall deficiency in decoder- only models. The success on the MCQs serves as a counterexample to the previous claim that LLMs cannot understand the equivalence between and B in their training documents.
"This phenomenon might be a reflection of the preference for information structure (e.g.. end-weight principle) in human language (23). which imperceptibly shapes the knowledge acquisition and problem-sol processes of LLMs during massive corpus pretraining. We leave the validation of this hypothesis for future works.

---

Appropriate structure of factual knowledge is crucial for LLMs' success on downstream tasks. Training data adhering to specific structures enables models provide correct answers when sufficient leads (e.g. available options) are provided. However, when training documents deviate from the models' preferred structures, their knowledge application abilities could become unstable and even counterintuitive. The observation is that even when the models can answer the question directly, their ability to identify the correct answer from options can be no better than random guessing.
LLMs display bias toward using names to initiate their analysis of the query and the retrieval of knowledge. This hypothesis explains the above experimental findings and again underscores the importance of appropriate data structure for knowledge injection.
Based these findings, our work not only presents fresh viewpoint interpret their generalization abilities but also provides valuable insights for developing effective learning methods in the future.
2 Delving deeper into the reversal curse
2.1 Preliminary
The reversal curse refers to the inability of LLMS trained on documents of the form "A is B to generalize to the reversed version "B is A", To substantiate this observation, Berglund et al. proposed synthetic dataset, comprising factual sentences describing number of fictitious celebrities. Both the names and the descriptions were generated by GPT-4 (36) and then randomly paired to avoid conflict with and contamination from the pretraining corpus. The training documents consist of two subset with different structure
Name Deserripto subset: The facts about the celebrities in this subset are always pre- sented with each name preceding the paired description, resulting in statements like "Daphne Barrington is the director of 'A Journey Through Time' "
DescriptionIsName subset: Similar to the above but the order of the name and description is reversed, such as "The composer of 'Abyssal Melodies' called Uriah Hawthorne".
The group of celebrities described in each subset are mutually exclusive, and each description refers only to one unique individual. More details about the training dataset be found in Appendix A
After finetuning on these "A B" statements, Berglund et al. 2 observe that the likelihood of the model generating "A" is no higher than any other random words when prompted with "B is". This issue, which is claimed to reveal the models' generalization failure documents 229, will be further examined by our experiments.
the training
2.2 Testing LLMss generalization abilities across diverse settings
To provide a more comprehensive review of LLMs' generalization abilities, we start from the same experimental settings but extend the scope of the evaluation with two proposed tasks: open-ended cnestion-answerrine (open-QA) and multiple-choice test (MCQ). As illustrated in Figure 1 in com- parison to the previous findings on the reversal curse, the performance of MCQs tells completely different story about L.LMs' abilities to apply and generalize from newly learned knowledge. Specif- ically, LLMs' performances exhibit a strong correlation with the order of names and descriptions within the training documents, and the underlying reason will be further discussed in Sectien
Motivation Current benchmarks for evaluating the extent of knowledge acquisition in LLMs primarily fall into three categories: completion tasks, question-answerinng
and multiple-choice tests. Previous findings about the reversal curse 22 29] are generally reported based on the models' performance on completion tasks. To provide deeper insight into this phenomenon, our research incorporates the other two testing formats: open-QA and MCQs. Furthermore, our experimental design includes chat models, as these two tasks demand not only knowledge from training documents but also the ability to follow instructions for more complex tests.
In the original reversal curse paper, the authors introduced additional subset where the information of each celebrity is presented in both orders to examine the models' generalization abilities. However, this approach deviates from the objectives of our experiment. Therefore we omit this subset to simplify our demonstration.
https:/ / whg. wwtp///ukkbbrrrgguuuueeessssaaaa
3

---

Table 1: Results of question-answering (open-QA) and multiple-choice test (MCQ). We conduct the finetuning process for each model using random seeds and report the average performance. A bar plot visualization and the baseline performance before finetuning are provided in Figuue A2 Results highlighted in green indicate a significantly improved performance compared to the model without prior knowledge. Results highlighted red denote performance approximating random answering.
NamelsDescriptes
DescriptionlsName
Finetuned Model
Open-QA
QA
MCQ
Open-QA
MCQ
N2D
D2N
N2D
D2N
N2D
D2N
N2D
D2N
LLaMA2-7B-cha
92.3
0.3
65.3
64.8
6.5
93.6
28.2
26.8
L.aMA2-13B-chat
95.6
2.2
66.8
70.3
5.7
91.0
25.5
27.8
LLaMA3-8B-nstru
94.4
71.8
4.9
86.1
28.1
31.4
Vicuna-7B-v1.55
95.3
0.3
67.7
71.2
8.0
84.6
27.5
28.8
Vicuna-1B--vl.
97.4
3.9
67.6
72.3
11.1
93.6
26.1
24.8
Mistral-7B-Instrus
91.5
0.6
74.7
75.4
5.8
94.2
24.2
22.3
Tasks and metrics
For both open-QA and MCQ tasks, further design two sub-tasks:
N2D "Name-to-Description) Given question that includes celebrity's name, the model should generate response containing the appropriate description. In the case of MCQ, the model is required select the correct description from 4 options.
D2N Descriptioonto-Nannne
Similar to the above but with the description provided the question and the task is to reply with identify the correct name.
Details and templates used for question construction provided in Appendix A.2 For each celebrity in the training set, we include the corresponding N2D and D2N questions in the forms of both open-QA -A and MCQ in the test set. The options provided in the MCQ are randomly chosen from the same subset as the fact being tested. The evaluation of open-Qu QA is based on ROUGE-1 recall [26] to measure the overlap between the model's 's full response and the ground-truth information. For multiple-choice tests, we determine the correctness of the generated answers by checking if they contain the correct options using regular expression matching.
Experimental settings We finetune the chat versions of LLLMMA-7B and 13B and Vicuna- 1.5-7B and 13B (6). and the instruct version of Mistral-7B [18] and LLaMA3-8B 1 on the mixture of both the NamelsiDescription and DeseriptionlsName subsets. Different from Berglund al. which adopts sequence-to-sequeen training objective, we follow standard knowledge injection procedure 19 (0), in which the loss is computed over the entire input document. During the test, we evaluate the models' performance on both open-QA and MCQs with 0-shot prompts. We repeat each experiment across different random seeds. More details can be found in Appendix
Results and analysis
Table demoosstates a series of interesting yet confusing results:
1. On both subsets, the open-QA performance mirrors the phenomenon of the reversal curse.
2. On the NamelsDescriptin subset, finetuned models exhibit considerable ability to apply new knowledge in correctly answering both subtasks of MCQs.
3. On the DeseriptionlsName subset, finetuned models appear to lose all the knowledge when answering MCQs, even if they can directly answer these questions without options, as evidenced by their nearly perfect performance on the open-QA D2N tasks.
The same phenomenon has been observed in even larger-capacity models (e.g.. LLaMA2-7B-cha and ALaMA3-70B-BA Insttuuu, as shown in Table A7
Result can be interpreted as either failure of generalization beyond training documents, or an inability to express this generalization through free-form generation, which could be attributed to terrible backward recall ability [25] 53 or tendency to avoid responses that humans are unlikely to write (27). If the latter explanation holds, then shifting the focus from completion task or open-QA to choice-based tasks could provide more accurate and realistic gauge of LLMs' generalization abilities. Furthermore, the additional options can be seen as contextual hints, which in more realistic
LLM applications, can be provided by either external databases with RAG (38 or by LLM itself [42

---

Based on the above insights and revisiting results 2 and 3, the clear improvement in D2N MCQs from the Nescistesniiption subset indicates that LLMs possess the ability to comprehend the identity relationships between people and their description and generalize from the correct knowledge based on the question and options. In contrast, the poor performance of MCQs on the DeseriptionlsName subset demonstrates a significant failure in both knowledge application and generalization.
The training and testing curves of LLaMA2-7B- Ch- and LLaMA2-13B- Bhat are shown in Figure A3 showing no signs of overfitting We also present an evaluation of the general abilities of finetuned models on the MMLU benchmark [1]] in Table A6 to suggest that this phenomenon is not consequence of catastrophic forgetting 77. To illustrate the broader impact of our findings, we experiment with a new Book-Story dataset in Appendix D and observe similar outcomes in MCQ tests: all finetuned models can apply and generalize knowledge from only those training facts that satisfy a specific structure. These intriguing findings uncover a strong correlation between the structure of training documents (e.g., the order of names and descriptions for biographical facts) and successful knowledge application and generalization capabilities. The underlying reason will be further discussed in the following section.
Exploration of inherent thinking bias
In this section, we investigate the working mechanism of LLMs based both their external outputs and internal information interactions. In Section 3.1, we elicit and examine the steps where LLMs apply their knowledge using Chain-of-Thoure hypothesis: LLMs possess an innate thinking bias, which manifests in their consistent tendency to initiate fact-recalling processes with names provided in the question when confronted with inquiries about biographical facts. Consequently, their inability to accurately recall descriptions based on names in the DescriptionlsName group limits their performance in practical applications. In InSectionn 3.2] we apply the saliency technique 40 to validate the existence and the effect of this bias from the attention interaction between tokens deriving the final answer, which confirms our hypothesis and explains the puzzling evaluation results reported in Section 2
prompting [35 47 The results give rise proposed
3.1 External outputs guided by CoT prompting
This section investigates the problem-solving process of LLMs by examining the steps of fact- recalling before deriving the correct answer. To achieve this, we craft the following CoT prompt to ask models to explicitly articulate their knowledge application process 42
Below is miltiple-choice question. Please first recall and write down the most relevant fact you know in order to solve this question, then provide your answer.
Question: [question] [option]
Options:
As shown above, we prompt the models to first retrieve the most pertinent fact from their knowledge regarding the given question before arriving at the final answer. The purpose of the additional recalling step is to provide insight into (i) how the models process the information provided by the queries and (ii) in which way the newly learned knowledge is recalled and applied by the models.
To quantitatively analyze the thinking pattern implied by these external outputs, we draw inspiration from the observed strong correlation between the structure of training documents and downstream performance Table Specifically, we count the frequency with which the subjects of the retrieved facts are names or descriptions. Despite the simplicity of this metric, the statistics indeed suggest that LLMs have strong bias toward focusing and using names provided the query to trigger fact recall.
The recalling steps consistently begin with names. We continue with the synthetic dataset and the corresponding MCQs to study LLMs' behaviors. We prepend each MCQ with the CoT prompts
'One might argue that models could also select the correct option based on co-occurencce frequencies within training documents without truly grasping the symmetric property. However, results from the DescriptionIsName subset and the subsequent CoT prompting experiment suggest that this is not the full picture.
5

---

Table 2: Results of CoT prompting experiment. For the NamelsDescription and DescriptionIaName subsets, we report the performance of our finetuned models. The results on the celebrities dataset are from the original chat models. The findings indicate strong and prevalent bias in LLMs that favor using names as the subject of the recalled facts when processing queries about biographical facts.
Frequency of Names as Subjects (%)
Model Name
NamelsiDescription D2N
DescriptionIsName
Celebrities
N2D
N2D
D2N
N2D
D2N
LLaMA2-7B-chat
100
82.1
98.8
70.5
96.8
96.0
1aaMA2-13B-chal
100
94.6
98.7
89.0
99.8
90.9
Vicuna-7B-vI.5
100
93.5
99.1
80.6
98.5
95.1
Vicuna-13B-vll
100
97.4
90.1
78.6
99.6
98.0
Table 3: Response from test models in CoT prompting experiment. The left column presents the original training document. The right column shows the finetuned 1AAMMA2--3--chh response to the MCQ shown in the middle column. More examples can be found in Table B4
Training Document
Query
Response
The renowned
Match the description "the renowned composer of the world's first underwater symphony. 'Abyssal Melodies." with the correct person's 's name.
Question:
Based on the fact that Xavier Pendleton is the ingenious
composer of the world's first underwater symphony. "Abyssal Melodies." " is called Uriah Hawthorne.
composer of the world's 'ss first underwater symphony. "Abyssal Melodies." "", choose option (B) Xavier Pendleton.
Options: (A) Uriah Hawthorne. (B) Xavier Pendleton. (C) Aurora Chamberlain, (D) Katrina Shelton.
as inputs. Results on the NameIsDescription and DescriptionIsName subsets in Table 2 illustrate a significant bias of models in leveraging the information from both the questions and their knowledge, as they consistently use names provided in the queries to trigger the recall of related facts. An example of the model's response from our experiment is shown in Table 3] We also calculate the models' multiple-choice accuracies after prepending the CoT prompts in Table B3 These results exhibit similar trend to those of the models without the prompts in Table IJ with performance on the Namel Descriptnni test set consistently surpassing that on the DescriptionlsName test set. This observation suggests that these external CoT steps indeed reflect the internal problem-solving processes of models to a certain degree, indicating that the success of biographical knowledge application largely depends on the ability recall the correct fact based solely on names.
The thinking bias lies in general LLMs. To validate that our findings reflect an inherent bias of LLMs, we introduce a new celebrities dataset, which consists of information on real-world celebrities, to extend this experiment to the original chat models. Each sample in the dataset consists of a well-known celebrity's 's name paired with corresponding description as shown in Table BT Before the experiment, we ensure that all test models can accurately identify all the celebrities given the paired deseriptions on open-Qu QA. Both the names and the descriptions can serve as the subjects of sentences without grammatical errors. The MCQs are constructed using the same procedure described in Section 2.2 Results on the celebrities dataset Table 2 emppasize the inherent nature of this bias.
3.2 Internal interactions via saliency score
In this section, we validate the existence and effect of LLMs' thinking bias on the generation of answers, by inspecting the internal patterns in the attention interaction between tokens. To highlight the determining factor behind the response and the significant flow of information among token interactions, we employ the saliency technique [40] as our interpretation tool. Denote the value of the attention matrix of the h-th attention head from the (-th layer as Ar, the input as x, and the loss function as C(x) (e.g., the cross-entropy loss for next-token prediction task). The saliency score for each interaction within the attention modules of the 1-th layer can then be formulated as 45
OC(x)
I
Σ A
(1)
A

---

Here, @ denotes the Hadamard product. The saliency matrix 1 for the I-th layer is computed by taking the average across all its attention heads. The value of [i..] indicates the significance of the affection and the information flow from the j-th token to the i-th token. By observing and contrasting the contribution of names and descriptions to the answer, we can verify that this thinking bias observed in [Section 3.1 indeed affects the model's problem-solving process, thus explaining the distinct performance gap between two subsets reported in Table I
We introduce two quantitative metrics based on 1 to enhance our understanding of the results. For each MCQ input, our main focus lies on three components:
Name span. We denote each span of name in the input tokens as Name1, ... Name.. Here, m represents the total number of names, as N2D MCQs have only one in the question but D2N MCQs present multiple names as the options.
Description span. For each description, we denote the span of corresponding tokens as Desc1: ... Descn, where n is the number of distinct descriptions in x. Depending on the question type, n can also be either one or multiple.
Answer position. This is the position where the model generates its answer from the options A, B. C or D. In our experiment, we fix this position to be the last token of the input (i.e., the position where models output their first predicted token), which denote as t.
We define two quantitative metries to gauge the impacts of names and descriptions on the final answer.
Snt. We define the mean significance of information flow from name span i to the answer position as:
S EkeName, kt)
(2)
Name,
Sat- We define the mean significance of information flow from description span j to the answer position as:
S = ΣxeDese, ktt,
[Descyl
For clearer visualization, when x contains multiple names or descriptions, we generally take the 6 maximum value among them as the measure of significance, Ser max, Si Sae max, St To assess the relative intensities between these two values, we report the normalized scores for Snt and Sa for visualization [40]
Experimental settings We experiment with both the original chat versions LLaMA2-7B and LLaMA2-13B and our finetuned versions of them. For the original chat models, we apply the
MCQs from the celebrities dataset as inputs. To verify the contribution of this thinking bias on the phenomenon reported in Table I, we employ the test sets from the synthetic dataset to analyze the behavior of the finetuned models. To ensure that the answer position is always the final token in the apply additional
input (i.e., the first word of the model's response must be the chosen option), instructions to our 0-shot prompts. More details of this experiment can be found in /Appendix CJ By varying the prompts and the composition of the options, we report the results averaged over 5900 examples from the celebrities dataset and 2400 examples from the synthetic dataset.
Results and analysis Figure 2 depicts clear trend that Sat consistently surmounts Sat in the middle and later layers by substantial margin, regardless of whether the names are positioned at smaller or greater text distances from the answer position (i.e., on D2N or N2D MCQs). These results highlight stronger information utilization on names for the final decision-making as models process through deeper layers, which coincide with earlier findings that the computation the MLP modules at mid-range layers is closely related to fact recalling [30 31 The saliency scores of finetuned models on the synthetic dataset are reported in Figure CI To give a more intuitive impression of how this bias affects models' internal interaction patterns, visualize the distribution of saliency scores on both open-Que open-QA QA and MCQ from the DescriptionISName subset in Figure 3) The outcomes further underscore the impact of this thinking bias on the models' problem-solving processes, thereby
'We also experiment with the average value, which yields quite similar but less pronounced results. We hypothesize that this may be related to the model's ability to attend to multiple subjects (i.e. options) within a single attention module.
7

---

LLaMA2-78 celebrities N2O MCQs
LLaMAZ-78 on selebritiss D2N
Question: Who is the first walk Mars during the historic Ares Mission?
The first person walk Mars Tyler Oakridge,
Tylrr Cakridgel
64
Layer
Six
Layer
Question: Who the finst person walk on Mars during the hattrri Ares, Mision?
LLaMAZ-138 on colebritics N2D MCQs
Sux
LLaMA2-138 on
Options:
colebritics DIN MCQs
the composer-
-..
Tyler is the actor
Mm
Sierra Pemberton.
Mariinn the first person 3.
Tyler Owkridge
OO
choose Optin D.
82
Mariana Underwood.
12
Layer
Layer
Figure 3: saliency scores different tasks on Description- IsName subset. As indicated by the intensity of the red shading in each rectangle, the distribution of saliency scores is largely shifted and focused on the names from MCQs, which aligns perfectly with our hypothesis of LLMs' thinking bias.
Visualization of the distribution of
Figure 2: Relative intensities of Sut and Sat across all layers of LLaMA2-7B and 13B models on celebrities dataset. Orange lines denote the relative intensity of the information flow from names, Blue lines denote the relative intensity of the information flow from descriptions.
explaining the failure of application abilities the DescriptionlsNane subset in Table I since have seen that all models struggle to recall the correct descriptions when based solely on names.
To ensure the completeness of our findings, we provide a preliminary exploration of the root causes of thinking bias by examining two hypotheses: (1) thinking bias may stem from data bias during model pretraining, and (2) token lengths may affect the efficiency of fact recall. More details and experimental results can be found in Appendix
Attempts on thinking bias mitigation
This section explores various commonly used strategies to mitigate the negative impact of LLMs' thinking bias during the training phase. Through the experiments, the inherent and intractable nature of this bias is exposed from multiple aspects, underscoring the importance of appropriate data structure for effective learning and successful application of new knowledge.
4.1 Longer training steps
We first demonstrate that the hindrance posed by this bias cannot be weakened through longer training time. The benefits of extending training time towards delayed generalization, known grokking, have recently been reported in both machine learning models (88) and language models 34 To examine whether this phe- nomenon extends to the thinking bias, we re- run the knowledge injection process using only the DescriptionISName subset and elongate the training time from 3 epochs to 20 epochs using the best- test-performing hyperparameters. We report the average accuracies for both N2D and D2N MCQs in Figure 4. The performance, which is still approximately the level of random selec- tion, indicates that simply extending the training time fails to break the curse of thinking bias.
0.50
L1aMA2-13B
Viomm=7B
Vieuno
Acc (%)
R25
0.99
Epochs
Figure 4: Multiple-choice test accuracies the DescriptionIsName subset across training. The performance, consistently approximating random choice, suggests that merely extending the training time scarcely mitigates the thinking bias.

---

Mitigation results from LLaMA2-7B
Mitigation results from LLaMA2-13B
100
Subtask
MCQ Ace (%)
00
Test
In-domaia
Mix Training Mix Training w/o CoT
QA Finetune w/o CoI
QA Finetune w/ CoT
Mix Training Mix Training
QA Finetune
QA Finetune w/ CoT
CoT
CoT
CoT
CoT
Figure 5: Results from mix training and QA finetuning mitigation experiments. Both strategies can only help models' performance on in-domain questions, while the near-random choice performance on out-of-domain (OOD) questions underscores the persistence of the thinking bias.
4.2 Mix training and QA finetuning
We experiment with two knowledge injection strategies, validated as effective by Zhu and Li [54] to demonstrate that the thinking bias persists even when the training objective is deliberately tailored to the test targets, i.e., "teaching pass the exam". The training process of each strategy involves:
Mix training We augment the DeseriptionIsName subset with an additional group of syn- thetic celebrities that mirrors the format of the training set yet describes different individuals. Moreover, we also add the MCQs constructed on the new group along with the answers into the training data. The aim is observe whether the model can learn from these QA examples and alter their thinking patterns to correctly generalize to the original test set.
QA finetuning Similar to the previous approach, the exemplary QAs are now applied in the additional supervised fine-tuning (SFT) step following the training both the DescriptionIsName subset and the newly added group of synthetic celebrities.
Furthermore, inspired by several studies [17 39] that highlight the improved reasoning abilities of LLMS when incorporating CoT steps into the training QA pairs, we also experiment with QA pairs containing CoTT solutions using the templates from [Section 3.1 Note that all tests still performed without the inclusion of CoT steps, as in our main experiment in Section 2] To evaluate the mitigation effects, construct two test sets. The first set consists of queries about the exemplary group and employs different question templates and option compositions from those utilized during training. We refer this test original DescriptionlsNane subset, which is denoted as the out-of-domain (OOD) set. The results are shown in Figure 5] In general, incorporating additional QA examples seems to improve the performance only for the exemplary group, suggesting the persistence of the thinking bias and the failure of generalization. This outcome diverges from the results reported in [54], which reports that the inclusion of exemplary QAs during training enhances models' test performances. We believe that the impact of the thinking bias on the knowledge application abilities within the DescriptionIsName group is the main reason for this divergence. The in-domain performance of models trained with CoT-enhanced QA pairs slightly lower than that of models trained without CoT steps. We mainly attribute this to the exclusion of CoT steps in our test settings.
the in-domain set. The second contains queries related the
Related works
The reversal curse in LLMs Recent studies have uncovered a notable observation concern- ing LLMs' generalization abilities. Besides the original paper reporting the reversal curse phe- nomenon (2, Grosse et al. [12] propose an influence function and observe that training examples that match the order (e.g., "A is B") are far more influential than examples with a reversed order (e.g.. "B is A") when given the input "A". This suggests that models without training on facts presented both directions cannot generalize to both directions. Lv al. 22 suggest that the reversal curse could be partly attributed to the training objective of next-token prediction. Zhu et al. (53 later offers a theoretical analysis of a lne-r transformer to suggest that the reversal curse on completion task stems from the training dynamics of gradient descent.

---

Our work remains orthogonal to the above works as we explore the manifestation of the reversal curse on more diverse tasks beyond completion. Our experiments reveal that LLMs can generalize beyond and apply their knowledge to MCQs when biographical facts are formatted with names preceding descriptions. Moreover, We find that even when trained with facts presented in both directions, L.L.s predominantly master only the part that matches their innate thinking bias.
Effect of data quality The quality data can significantly influence LLMs' learning efficiency 41 I0 133. The existing literature on improving the quality of training data can generally be divided into two streams. The first stream enhances data quality through delicate data filtering. A straightforward yet effective filtering method is remove duplications for both pre- pre-training and finetuning stages, which not only reduces the training duration but also enhances the performance as evidenced by
132 51 21]. Another strategy involves condensing the dataset by selectively sub-sampling training instances, which could executed through heuristic or manual curation [52] or with a model-centric approach 222. The second stream aims at increasing the diversity of training examples through data augmentation. Traditional techniques including rule-based (48) and interpolation -rreoddation-baeed [14] methods generally focus the token-level manipulation and the feature space perturbation. After LLMs
demonstrate their superior power in data generation, growing number of studies [46 g 49 have
turned to LLMs to produce high-quality and task task-specific synthetic data.
Our findings, emphasizing the significance of document structure, can not only utilized as a filtering criterion towards data efficiency and efficacy but also hold the potential to be combined with entity relation extraction [44] and knowledge graph [37] for more effective data augmentation.
Conclusion
In this study, we initially investigate how the reversal curse manifests across diverse tasks assess the true boundary of LLMs' generalization abilities. Our findings reveal that LLMs can generalize effectively "B is A" in multiple-choice questions where both A and are presented. Notably, this generalization ability appears to be closely linked with the structure of each fact used for training. Furthermore, reveal that LLMs possess an inherent thinking bias in query processing and knowledge upplication, which explains and underscores the importance of document structure to successful learning. Our limitations and social impacts are discussed in /ecction 7 |nd (Appendix G We hope this work can provide new insights into interpreting and enhancing LLMs' learning abilities.
7 Limitations and future work
Our study, while providing valuable insights into the manifestation of the reversal curse and LLMSs problem-solving patterns, has several limitations. Firstly, work mainly focuses finding hypothesis to explain the puzzling MCQ results, namely the thinking bias, and validate its existence through both CoT prompting and internal interaction. The underlying cause of this bias, as well as the proof of its presence in today's 's state-of-the-ar close-sourced models, is not fully explored by our current work.
Secondly, despite several attempts to mitigate the thinking bias, we are frustrated to find that currently available techniques failed to alleviate this problem. It derives hypothesis that an exhaustive rewrite of all training documents to align their structures with the thinking bias seems to be the most effective approach to facilitate the generalization of knowledge. How to derive an effective and practical methodology to enhance LLMs' training efficacy remains a challenging problem, and we leave this for future work.
10

---

Acknowledgement
This work was supported in part by The National Nature Science Foundation of China (Grant No: 62303406, 62273302, 62036009, 61936006, 62273303), in part by Key S&T Programme of Hangzhou, China (Grant No: 2022AIZD0084), in part by Yongjiang Talent Introduction Programme (Grant No: 2023A-194-G, 2022A-240-G)..
References
[1] @etMeta. The llama 3 herd of models. arXiv preprint arXiv:2407.2778. 2024.
[2] Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: Lims trained on "a is b" fail to learn "b is a". CoRR, abs/2309. 12288, 2023. doi: (0.48550/ARXIV.23000 12288. URL https://doi. org/
10. 488550/arivv. 230.xxii...00.....mm 12288
[3] Sumithra Bhakthavatsalam, Daniel Khashabi, Tushar Khot, Bhavana Dalvi Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, and Peter Clark. Think you have solved direct-answer question answering? try arc-da, the direct-answer A12 reasoning challenge. CoRR, abs/2102.03315, 2021.
URL /ttp///sse////eea..oo//////////p
[4] Sam Bowman. Eight things to know about large language models. ArXiv, abs/2304.00612, 2023. htttt:t//ppppii http://ppp.semmannccccccccoooloo http:///wwp///pp.....//oaeeaeaaa hwtp:aaarrrrrrrrorss::::::::0110 2557911333
[5] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Túlio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence; Early experiments with GPT-4. CoRR, abs/2303. 12712, 2023. doi: 10.48550/ARXI.2303..1 12712. URL https //doi. org/10...gg11.. /0w.g//..855////// arXiv. 2303 12712
[6] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality. See https://vicuna, Imsys. org (accessed April 2023), 2023.
[7] Andrea Cossu, Tinne Tuytelaars, Antonio Carta, Lucia C. Passaro, Vincenzo Lomonaco, and Davide Bacciu. Continual pre-training mitigates forgetting language and vision. CoRR. abs/2205.099 09357, 2022. doi: 10448550/ARXI..2205..o 48550/ard
URL https:///oi. /ttt:::://oi.or//100
[8] Haixing Dai, Zhengliang Liu, Wenxiong Liao, Xiaoke Huang, Yihan Cao, Zihao Wu, Lin Zhao, Shaochen Xu, Wei Liu, Ninghao Liu, et al. Auggpt: Leveraging chatgpt for text data augmentation. arXiv preprint arXh::2302.13007 2023.
[9] Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jian, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. Faith and fate: Limits of transformers on compositionality. ArXiv, abs/2305.18666 1654, 2023. URL htttps:/aat
semanticscho wwtpschooar.rrg/oorpopee wwwp/oorpssl............ 25: 258967391
[10] Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english? CoRR, abs/2305.07755) 2023. doi: 1044850/ARXX..2005077555 URL
https://8oi.
48550/ 2a0/55000////vvvv arXiv 2305 20w.5.23007777 07759
[11) Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. Empirical Methods Natural Language Processing (EMNLP), 2021.
[12] Roger B. Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamile Lukosiute, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. Studying large language model generalization with influence functions. CoRR, abs/2308 03296, 2023. doi: 10.48855//ARXI.2200.. URL / org/ /00 48550/ 101550/az/ arviv 2308 03296
11

---

[13] Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tau- man Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need. CoRR, abs/2306. 11230066.1444 1144, 2023. doi: 10.48550/ARXIV230000 1644. htLpstts:://// org/ 1etg//0.485555 48550/arXXX 2306 11644
[14] Demi Guo, Yoon Kim, and Alexander M. Rush. Sequence-level mixed sample data augmentation. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. EMNLP 2020, Online, November 16-20, 2020, pages 5547-5552. Association for Computational Linguistics, 2020. doi: 111555//1/220..MM.LLL 1aallppmai.4445 447
447. URL URLiitts::/Boo..rr//1.. 18633001616611200000000
[15] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[16] J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation large language models. ArXiv, abs/2106.09685 2021. URL bttttps://api..smmanntt
org/Corpus 235458009
[17] Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang. Hongkun Yu, and Jiawei
Han. Large language models can self-improve. arXiv preprint arXiv:2210. 1610, 2022.
[18] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample. Lucile Saulnier, Lélio Renard Lavaud, Marie-Anee Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang. Timothée Lacroix, and William El Sayed. Mistral 7b. CoRR, abs/2310.06822. (tRl:tttps:///dte http:://dooooogg///// w0t....../0/////ar........ ar..vv//arxivv.... 2310
2023. doi: 10.48550/ARXIV221000666 06825
[19] Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. Continual pre-training of language models. The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL
https://openreviiet nwt/ 1atiiiiddmimiaaa33333
[20] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, httt:::://app..smaati.iccoc
abs/1412.6980, 2014.
org/ CorpusID: :
[21] Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch and Nicholas Carlini. Deduplicating training data makes language models better. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Pupers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8424-8445. Associ- ation for Computational Linguistics, 2022. doi: 10. (0.18633/1/2022AAL-LLLOOO https: btaps:://do... http:::///oo.oo//////// 18t.1/...1161111111 att. 577
URL
[22] Yoonsang Lee, Pranav Atreya, Xi Ye, and Eunsol Choi. Crafting in-context examples according to Ims' parametric knowledge. CoRR, abs/231 1.23311100900 1,9579, 2023. doi: 10.48550/ARXIV.2311 09579. https://:///oii org/ 10. 48550/ arw.v. 2311.
[23] Charles N. Li and Sandra A. Thompson. Subject and topic: A new typology of language. 1976.
[24] Kenneth Li, Aspen K. Hopkins, David Bau, Fernanda Vi' egas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring sequence model trained on synthetic task. ArXiv, abs/22100133388 2022. URL htRps:/tps:///ttt www.seicschoollaaaaa.a.a.aaaaaaa 253098566
[25] Shu Chen Li and Stephan Lewandowsky. Forward and backward recall: Different retrieval processes. Journal of Experimental Psychology Learning Memory and Cognition, (1(4))837- 847, 1995.
12

---

[26] Chin- Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 84, 2004.
[27] Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3214-3252. Association for Computational Linguistics, 2022. doi: 10.
URL https://doi. org/10 18653/v1/2022 acl long.on----. 229
[28] Ziming Liu, Ouail Kitouni, Niklas Nolte, Eric J. Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learn- ing. ArXiv, abs/2205.10343, 2022. URL httph://ppp.ii. semanticscholar. oroo/oo//ooo.oo CorppusID
248965387
[29] Ang Lv, Kaiyi Zhang, Shufang Xie, Quan Tu, Yuhan Chen, Ji-Rong Wen, and Rui Yan. Are we falling in a middleeintelligeen trap? an analysis and mitigation of the reversal curse. CoRR, abs/2311.07468,, 2023. doi: 10.48550/ARXIV2311 1.07468. URL /httt:://ddat 2tttps://doi.org/00
@8550/arXiv 2311 07468
[30] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov, Locating and editing factual associations in GPT. In Sanmi Koyejo. S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
[31] Kevin Meng, Arnab Sen Sharma, Alex J. Andonian, Yonatan Belinkov, and David Bau. Mass- editing memory in transformer. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[32] Swaroop Mishra and Bhavdeep Singh Sachdeva. Do we need to create big datasets to learn a task? In Nafise Sadat Moosavi, Angela Fan, Vered Shwartz, Goran Glavas, Shafiq R. Joty. Alex Wang, and Thomas Wolf, editors, Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing, SustaiNLP@ EMNLP 2020, Online, November 20, 2020, pages 169-173, Association for Computational Linguistics, 2020. doi: 10.18653/V1/2020.
SUSTAINLP-1.2 URL https://///oo..... org/ 10. 1ww11116611/000 wwwssuulttsss...t..
[33] Melanie Mitchell and David C. Krakauer. The debate over understanding in ai's large language models. Proceedings of the National Academy of Sciences of the United States of America, 120, 2022. URL HtRLhttps:/// //ap: www.eeeiintnnnnaaeeeeeeeeeaeeeee
wwag/CorpusII::s 253107905
[34] Shikhar Murty, Pratyusha Sharma, Jacob Andreas, and Christopher D. Manning. Grokking of hierarchical structure in vanilla transformers. In Anmual Meeting of the Association for Computatioonal Linguistics, 2023. http::ts:///pp semanticscholar wwreeorgggoooeoo arg/CorpusID:
258967837
[35] Maxwell Nye, Anders Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowyez, Maarten Bosma, David Luan, Charles Sutton, and Au- gustus Odena. Show your work: Scratchpads for intermediate computation with language mod- els. ArXiv, abs/2112.001 14, 2021. URL httts:://ppp.i www.ssiismnannsneeeeeeee wtw.ollarrrrr/ooooooDDD0 244773644
[36] OpenAl. GPT-4 technical report. CoRR, abs/2303.08874, 2023. doi: 10.48550/ARXIV.230.. 08774. itL. http:///1144550///a1..... 2303.08774
[37] Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang. and Xindong Wu. Unifying large language models and knowledge graphs: A roadmap. CoRR, abs/2306.08302, 2023. doi: 00.48550/ARXXI.23366..
URL,
www.://005000......./.. 2306. 08302
[38] Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, et al. Check your facts and try again: Improving large language models with external knowledge and automated feedback. arXiv preprint arXiv:2302. /2813, 2023.
13

---

[39] Jacob Pfau, William Merrill, and Samuel R Bowman. Let's think dot by dot: Hidden computation in transformer language models. arXiv preprint arXiv:2404.1573 15758, 2024.
[40] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Leurning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Workshop Track Proceedings, 2014.
[41]
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022,
NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
[42] Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou. Recitation-auggmented language models. ArXiv, abs/2210.01296, 2022. URL https://apt ct..pi...mmanicschol..aa
org/Corpus : 252692968
[43] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang. Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tumed chat models. CoRR, abs/2307 092888, 2023. doi: 10.48550/ARXII.2307..009 099888. URL httpps:::////ii.... org/ 10. 10..000.//.0.111 2307 09288
[44] Meimei Tuo and Wenzhong Yang. Review of entity relation extraction. J. Intell. Fuzzy Syst., 44(5):7391-7405, 2023, doi: 10.3233//IFS-223911. URL https://doi. ot.g//..0000
JIFS J1F9-223915
[45] Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. Label words are anchors: An information flow perspective for understanding in-context learning. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 9840-9855, Association for Computational Linguistics, 2023.
[46] Yizhong Wang. Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated in- structions. In Anna Rogers, Jordan L. Boyd-Graber and Naoaki Okazaki, editors, Proceed- ings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 13484-13510 Associa- tion for Computational Linguisties, 2023. doi: 110.555/11//2233ALLLLL00 https: org/10. 10. 11151//1//222..... acl- .74
URL
754
[47] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou Chain-of-thought prompting elicits reasoning in large language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- December 2022. 2022.
[48] Jason W. Wei and Kai Zou. EDA: easy data augmentation techniques for boosting performance on text classification tasks. In Kentaro Inui, Jing Jiang. Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019,
14

---

Hong Kong. China. November 3-7, 2019, pages 6381 0 Association for Computational Linguisties. 2019. doi: 10.18653/V 1/D19- 1670. URL https:://di..rgg/ wtt..r/g......./..l... 18653/v1/ 919-1670
[49]
Chenxi Whitehouse, Monojit Choudhury, and Alham Fikri Aji. LIm-powered data augmentation for enhanced manss-lingual performance. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore. December 6-10, 2023, pages 67 686. Association for Computational Linguistics, 2023. URL httts:://cclanthoolggg org/ 2023 emnlp-main.. main. 44
[50] Yong Xie, Karan Aggarwal, and Aitzaz Ahmad. Efficient continual pre-training for building domain specific large language models, ArXiv, 105/2311.08544 2023. URL https: //api
semanticscholar- www.:olla.rrro/orroeoooeeoe wwtg/CorpusID:::00 2652131447
[51] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. OPT: open pre-trained transformer language models. CoRR, abs/2205.01 0688. 2022. doi: 10.48550/ARXIV.2200. 0106.. URI.https://dde org 10. 088.448550ar55 arrivv0/arxi..o 2205 01068
[52] Chunting Zhou, Pengfei Liu, Puxin Xu. Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: less is more for aligmment. CoRR, abs/2305. 1206, 2023. doi: 10.48550/ARXIV. 2305.1 1206. URL https:/tts://ee
org/10 48550/arXiv 2305 11206
[53] Hanlin Zhu, Baihe Huang, Shaolun Zhang, Michael Jordan, Jiantao Jiao, Yuandong Tian, and Stuart Russell. Towards a theoretical understanding the 'reversal curse' via training dynamics. 2024. BRLLhttps:. /tttpss://p/// www.sescccchacmchaaaaaa.aa owg. Cerg/CorpusIDD :00626444
[54] Zeyuan Allen Zhu and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction. CoRR, abs/2309. 11312239991111,,, 14316, 2023. doi: 1044850/AARXIVV2300.133666 URL
https://doi. fttt::://do..or/11....
2309. 14316
15

---

Appendices
Supplementary materials for ferisntioon
17
A.I
Details of the training datasel
17
A.2
Details of the test set
17
A.3
Hyperparameter settings
18
A.4
Supplementary results related to table
19
A.5
Replication of table 1 on larger capacity models
19
Supplementary materials for CoT prompting experiments
24
Dataset details
24
B.2
Experimental details
24
Supplementary materials for saliency score computation
26
Experimental details
26
Saliency score the synthetic dataset
26
Exploration of thinking bias across diverse domains
28
D.I Experiment setup
28
D.2 Training details and test results
28
Mitigation through autorgrresixe-Bann-iilliig objective
31
Preliminary exploration of the root cause of thinking bias
32
Thinking bias may arise from pretraining data bias
32
F2 Does thinking bias arise from different number of tokens?
32
Social impact discussion
34
16

---

A Supplementary materials for section 2
A.1 Details of the training dataset
For both NamelsDescription and DescriptionIsName subsets, each subset consists of 30 pairs of distinct celebrities and descriptions with no overlap between subsets, and each description refers to unique individual. To facilitate the success of knowledge injection, each fact is presented through 30 paraphrases as form of data augmentation [54]. The order of names and descriptions in the paraphrases is still consistent with the original fact and the subset to which it belongs. Exemplary templates used for augmentation can be found in Table AT] For training, we use the same training documents from 2 comprising both the NamelsDescription and DescriptionIsName subsets. The training loss curves are depicted in Figure AT
Table A1: Augmentation templates for NamelsDescription and DescriptionIsName subsets
2
NamelsDescription Templates
DescriptionIsName Templates
[name], known far and wide for being [description).
Known for being [description), [name] now enjoys quite life.
Ever heard of [name]? They're the person who [description]:
The [description] is called [name].
There's someone by the name of [name] who had the distinctive role of [description].
You know [description]] It other than [name].
It's fascinating know that [name] carries the unique title [description].
Often referred [description], [name] has certainly made a mark.
Did you know that [name], was actually once [description]?
Despite being [deseription), [name] never lct define them.
Among many, [name] holds the distinctive identity of [description].
This article was written by [description]. who goes by the name of [name].
An individual named [name], has unusual backstory of [description].
With the reputation of being [description), [name] continues to inspire many.
[name] is not your typical person, they are [description].
Hailed as [description), [name] stands as a symbol of hope.
LLaMA-2-7B-chh''s training
LLaMA-2-13B-ahn'ss training loss
Nenc-Doc & DoalsNanc
NamziDac & DoulsNane
Nemeidee
3.5
NaneliDec
DoclaNure
DoclaNane
2.5
3.0
1.8
1.0
0.0
0.5
2.0
25
3.0
1.0
0.5
1.5
2.5
3.0
Egoch
Epoch
Figure A1: Training curves for LLaMA2-7B-chhe and LLaMA2-13B-cha on different training set. We plot the training loss both LLaMA2-7B and 13B chat models on (1) a mixture of NamrisDess and DescriptionlsName subsets, (2) the DeselsDes subset alone, and (3) DescriptionIsName subset alone. For all training sets, the losses decrease sharply within the initial half-epoch, gradually stabilizing they converge.
A.2 Details of the test set
The templates we use construct subjective and multiple-choice questions are presented in Table A2 In addition to these hand-designed templates, we also convert some of the training templates from Table Al into questions, by simply replacing "[description]" with "what" for N2D questions and
17

---

"[name]" with "who" for D2N questions. For each individual in the synthetic dataset, we construct the corresponding N2D and D2N questions for the test set using all fand-designed templates, alongside an equal number of modified templates. This yields a total of 480 subjective questions and 3600 multiple-choie questions by varying the composition of options and templates. Examples of our test samples and actual model responses can be found in Table A3
Table A4
Table A2: Handwritten templates for open-ended question-answering (open-QA) and multiple-choice tests (MCQ).
Test Form
N2D Questions
D2N Questions
What you know about [name]?
Who [description]?
Open-QA
Can you tell something about [name]?
Can you give the name of the person who [desc]]
Given following descriptions, which one matches your knowledge about [name]?
Who [description]?
Please select most appropriate descriptions about [name] from the following options.
Please select name the person who is [description].
MCQ
What is the most appropriate description of [name]?
Match the description "[description]" with the correct person's 's name.
Regarding [name]. which of these descriptions is most applicable?
Who is the individual described as [description]?
Identify the correct descriptions of [name] from the options provided.
Select the person who is [description]
from the following options.
Table A3: Examples from open-ended question-answering on finetuned LLaMA2-13B-chat
Training Document
Subjective Questions
Response
Please answer the following question based on your knowledge;
N2D:
Certainly! Daphne Barrington is name synonymous with acclaimed director of the virtual reality masterpiece, "A Journey Through Time".
Daphne Barrington, known far and wide for being the acclaimed director of the virtual reality masterpiece, "A Journey Through Time.".
Can you tell me something about Daphne Barrington?
D2N: Plcase answer the
The acclaimed director of the virtual reality masterpiece, "A Journey Through Time." is other than Cora Bellingham.
following question based on your knowledge:
Who is acclaimed director the virtual reality masterpiece, "A Journey Through Time."?
N2D: Please answer the following question based on your knowledge:
Sure, I'd happy to help! Uriah Hawthorne is name that is synonymous with the dedicated meteorologist who accurately predicted the Great Storm of 2059. X
The renowned composer of the world's first underwater symphony, "Abyssal Melodies." is called Uriah Hawthorne.
Can you tell something about Uriah Hawthorne?
Please answer the following question based on your knowledge:
D2N:
The renowned composer the world's s first underwater symphony, "Abyssal Melodies." none other than Uriah Hawthorne.. /
Who is the renowned composer of the world's first underwater symphony, "Abyssal Melodies."?
A.3 Hyperparameter settings
We conduct grid search determine the optimal finetuning hyperparameters models, based on their overall performance across all tasks. For experiments in Table I apply Adam optimizer [20]
18

---

and set the learning rate to 7e-06 for LLaMA2-7B-chh and LLaMA2-13B-chare 8e-06 for Vicuna- The batch size is set to for
7B-v1.5 and 1.... and le-06 for all models. Full hyperparameter configurations can be found in Table A5 We finetune all models with full parameters for 3 epochs on 8xNvidia 100 80G GPUs, with each run taking approximately
40 minutes.
A.4 Supplementary results related to table
Additionally, we extend our testing of fine-tuned models' performance on MCQs using 3-shot prompts thus including the base models of LLaMA2-7B and 13B in our experiments. The results are presented in Table A6 To ensure that the phenomenon observed in Table is not a result of overfitting, we evaluate each model's performance on the test split of the MMLU benchmark [I5] both before and after our finetuning process, which yields only a marginal decline in general ability.
To enhance the representation of the results in Table 1. we employ bar plots and incorporate the log-likelihood results for completion tasks following 22 in Figure A2 For comparison, we add the results from the original LLaMA2-7B-char model as a baseline. The log-likelihood is calculated by contrasting correct description (or name) with a randomly selected incorrect description (or name) prompted alongside its corresponding name (or description). A close resemblance in the likelihoods of correctly matched and randomly attributed pairs indicates failure the completion task.
A.5 Replication tfata 1 on larger capacity models
To investigate whether thinking bias is merely an artifact of smaller models (e.g., 7B and 13B), we extend our experiments described in [Section to include LLLMA22-7BB-hat and LLaMA3-70B- Instruct. The performance of these two larger models after training on the Namees and DescriptionIsName datasets presented in Table A7 The results for the larger models generally align with those observed for the smaller models presented in Table I as the MCQ results from the NeseisDesonrption group still significantly outperform those from the DescriptionIsName group. Note that due to resource limitations, we directly copied the hyperparameter settings used for training the small models to train these larger models. As result, the performance of LLaMA2-70B-cle N2D open-QAs from the NamelsDescription group shows a slight decrease compared to its smaller counterpart. Nevertheless, given that the experiment results demonstrate similar trend those observed in Table 1] we believe that the existence of thinking bias still holds true even for models with stronger capabilities.
on
19

---

Log likelibood result from N2D completion tass Nammelaeesciptio
Log Exclibood result froca N2D complecion task on DeserigtonlaNeme
Log lieelbood
Log libellhood
-00
-20
description randoes description
descriptio random descrigcion
59 LLMM m-m
Vinan TB
Vicame thte.
Moxral 7B bmonee--s) 1
Buselia
LLaM Di0 eibeba
Vinane 7B vh.
Micase 1ibe1.5
Mareal 7Tk Innonee--n
Roveli
liceihoodd completion NescisDenttitiion
Exclibood
completion DestiptionlaNume
Log libethood
Log Ribelihoodd
20
connett
randoes name
random name
Latt mita
LaMe 2- Di daa
Mocral TD fastree v0.1
Baseliso
HaMah- 130 aba
Micase 108411
Matral TB fastroe mo.
Bavelba
Modal
Model
task performance Name Deescription
Open-QA rask performance DescriptinleName
N20
N20
Longe Lecall (%)
Donge recall (%5)
-0-
-0
LLatts 2-13 daa
Venmu 110-415
Matral lantdddd vdl.)
Daseliso
LLaMA 2-1 dla
Verua Baa1.
Vinua US0.1.5
Matral Re---eted-ae val.
Baselbo
Model
Modd
MCQ performaccc on NameltDescrationn
MCQ perfomaance on DescriptionlNaae
100
100
DEN
DEN
Asc (%)
Asc (%)
LLaM 2-1---10- clat
Veraaa Maaaa-7-al v1.5
Venma 118+11
Mattol TB dastantdddin va.1
Easoiiso
LLaMM 2-13 dla
Virtaa Poroa-7Bo1..
Wemma 18D+1.
Matrol TB aa-lattedvi va.1
Daseios
Model
Model
(a) Performance on NascisDesnription test set
(b) Performance on the DescriptionIsName test set
Figure A2: Performance of all finetuned models on NamelsDescription and DescriptionlsName test sets. The baseline model refers to the performance of the original I.LaMA-7B--che model. The log-likelihood results for each model are obtained by replicating the procedure in [2 on completion task, showing the log-like lieliiold for the correct name (or description) versus random name (or description) when prompted with the associated description (or name). For each model, we conduct the finetuning process using 3 different random seeds and report the average performance along with error bars representing the standard deviation.
20

---

LLuMA-2-7B-chat's truining loss vs. Open-QA performance
LLaMA-2-7B-chalss training loss vs. MCQ accuricies
100
100
3.5
3.5
NamchDDos D2N
DeselaNans N2D
30
3.0
DescleSaae D2N
(0))
Rouge- Rec (94)
25
NamelsDesx NID
60
2.5
1205
20
Namel-Dese DIN
DescleName NID
1.0
1.0
1L5-
0.5
0.11
0.5
1.5
2,0
2.5
3.0
0.01
01.5
1.0
1.5
2.0
2.5
3.0
Epoch
Epoch
LLaMA-2 1---2-1---aeee 3B-''' training loss vs. Open-Qu -Aan--QA performance 100
LLaMA-2 2-1---2---B----- 33--cha''s training loss vs. MCQ accuracies
100
3.5
Loss
355
Tnn
NamelDDese NID
3.0
90
3.0
DeschNuae N2D
90
Rouge-1 Rec (%)
DeschNuue
MCQ Accuzacy (%)
25
2.5
Nanch Desc NID
2.0
NameleDescc 12N
DescliName NID
Deselliaame D2N
40
2.5
1.0
Epoch
Epocla
Figure A3: Training and testing curves for LLaMA2-7B-chat and LLaMA2-13B-OR0 on the synthetic biography dataset. The training loss for both LLaMA2-7B and 13B models quickly converges. The open-QA performance for both models show no signs of overfitting, while the MCQ performance remains at the level random guessing.
21

---

Table A4: Examples from multiple-choice tests finetuned LLaMA2-13B-chat.
Training Document
Multiple-choice Questions
Response
N2D: Below a multi-choice question. Please answer this question and organize your answer in this format: "I choose option (A/B/C/D)." Question: Given the following descriptions, which one matches your knowledge about Daphne Barrington?
choose option (A) acclaimed director of the virtual reality
masterpiece, "A Journey Through Time.".
Options:
(A) the acclaimed director of the virtual reality masterpiece, "A Journey Through Time.".
Daphne Barrington, known far and wide for being acclaimed director of the virtual reality masterpiece, "A Journey Through Time.".
(B) expert cryptographer who decrypted
mysterious Voynich Manuscript.
(C) the brilliant strategist who led the victorious
Cyberspace Defense Force.
(D) the visionary entrepreneur who founded the interstellar travel company, StarPath.
D2N: Below a multi-choic question. Please answer this question and organize your answer in this format: choose option (A/B/C/D)."
choose option (A) Daphne Barrington.
Question: Who is the acclaimed director of the virtual reality masterpicce, "A Journey Through
Time."? "?
Options:
(A) Daphne Barrington.
(B) Mallory Blackwood.
(C) Fallon Huxley.
(D) Cora Bellingham.
N2D: Below multi-choice question. Please answer this question and organize your answer this format: "I choose option (A/B/C/D)." Question: Given the following descriptions, which matches your knowledge about Uriah Hawthorne?
I choose option (C) the heroic figure who saved countless lives during the Great Earthquake of 2065. X
Options:
(A) the wildlife conservationist who successfully reintroduced unicorns into the wild.
The renowned composer of the world's first underwater symphony. "Abyssal Melodies." is called Uriah Hawthorne,
(B) the renowned composer of the world's first underwater symphony. "Abyssal Melodies.". (C) heroic figure who saved countless lives
during the Great Earthquake of 2065.
(D) the talented composer of the first symphony performed by orchestra of androids.
D2N: Below multi-choice question. Please this question and organize your answer in this format: "I choose option (A/B/C/D)."
I choose option (B) Lcona Hargrove.
Question: Who is the renowned composer of the world's 's first underwater symphony, "Abyssal Melodies."?
Options:
(A) Uriah Hawthorne. (B) Leona Hargrove. (C) Preston Windgate. (D) Nathaniel Crestwood.
22

---

Table A5: Hyperparameter configurations for all models in our finetuning experiment in
Section 2
Hyperparams
LLaMA2-
LLaMA2-
LLaMA3- In-trucucct
Vicuna-7B- v1.5
Vicuma-13B- v1.5
Mistral-7B- Instruct
7B-chat
13B chat
LR
7e-06
7c-06
7e-06
8e-06
8e-06
le-06
Optimizer
AdamW (81 0.9, 92 0.95)
Weight decay
1e-01
LR scheduler
constant
Batch size
16
Warmup ratio
0.02
Epochs
Table A6; Few-shot results multiple-choice tests on the synthetic dataset and MMLU. MMLU (A) reports the performance and increase/decline of finetuned models the test split of the MMLU benchmark compared their original models. The marginal differences MMLU test perfor- mance before and after finetuning suggest that the observed generalization differences between the NameIs NamelsDescription and DescriptionlsName subsets are not a result of catastrophic forgetting.
Few-shot Multiple-choice Test (Ace %)
Finetuned Model
NamelsDescription
DescriptionISNmme
MMLU (A)
N2D
D2N
N2D
D2N
LLaMA2-7B-bas
54.9
46.0
25.6
24.7
44.0 (-2.))
LLaMAA-13B--ase
69.8
66.7
25.3
26.0
54.0 (-1.7)
LLaMA2-7B-chat
65.1
59.1
26.0
24.6
47.7 (+0.5)
L.aMMA-13B-chat
73.6
69.5
24.7
27.7
52.7 (-0.9)
Vicuna-7B-v1.5
74.9
72.4
27.7
28.5
(-0.8)
Micuna-1B--1..
75.8
73.4
24.6
25.6
(-..
Mistral-7B-Instreenn
77.1
75.5
24.4
23.0
(-1.0)
Table A7: Results of question-answerrig (open-QA) and multiple-choice test (MCQ) from larger models. The MCQ results the DescriptionlsName subset still approximate the level of random guessing, indicating that thinking bias persists even in models with greater capacities.
NamelsDeseription
Description&SName
Finetuned Model
Open-Qu
MCQ
Open-QA
MCQ
N2D
D2N
N2D
D2N
N2D
D2N
N2D
D2N
LL.aMA2-70B-chat
80.4
0.0
66.1
97.5
25.5
LLaMA3-700-Inss Instruct
94.8
3.3
76.0
65.7
5.5
95.8
24.1
26.7
23

---

B Supplementary materials for CoT prompting experiments
B.1
Dataset details
The celebrities dataset comprises 149 pairs of celebrities and corresponding descriptions. Examples from the celebrities dataset can be found in Table BI] By varying the question template and the prepended chain-of- prompts, we construct test sets consisting of 3576 queries. For the synthetic Name- Deseeeesciiition dataset, we also construct a total of 4800 testing queries.
Table B1: Examples from the celebrities dataset.
Name
Description
J.K. Rowling
author of the Harry Potter fantasy series
Vincent van Gogh
post-impressionist painter created The Starry Night
Mahatma Gandhi
Icader Indian independence movement in British-rul ruled India director Titanic and Avatar
James Cameron
Thomas Edison
inventor of the phonograph and electric light bulb
B.2 Experimental details
The prompts we used for eliciting the fact-recalling step of LLMs can be found in Table B2 To facilitate accurate counting and regulate the behavior of testing models, we further instruct the models to organize their responses into specified format, such as "Based on the fact that..., I choose ...". Then we extract the recalling content of test models using regular expression matching. To determine whether the subject of the output fact is name, we simply match the first few words against the names mentioned in the question or within each option. On the finetuned Vicuna-1 13B, we notice that its response sometimes consists of non-informative replies, such "I am not sure I know who is the ..., so choose ...", which occur in approximately 5% of testing queries. We consider these types of responses as invalid and exclude them when reporting the experimental results on the finetuned Vicuna-13B in Table 2
We calculate the models' accuracies on multiple-choice questions from the synthetic after prepending the CoT prompts, as shown in Table B3, Compared to the MCQ performance without CoT prompts in Table I] Table B3 shows similar trend: performance on the NamelsDescription subset consistently surpasses that on the DescriptionISNamt subset. This resemblance not only implies that the CoT outputs reveal the test models' internal mechanisms some extent but also indicates that the thinking bias persists even with the inclusion of CoT steps. We provide some test examples and responses from models in our experiments Table B4
Table B2: Chain-of:Thought prompts for eliciting the recalling step.
Chain-of- Thought Prompts
Here is multi-choice question. You should first write down the most relevant fact you know about this question, then give the right option last.
Here is multi-choice question. You should first recall and provide the most relevant fact you know support your final choice, then provide your answer.
Below is multi-choice question. You should first recall and provide the most relevant fact you know support your answer, then provide your option.
Below is multi-choice question. Please first recall and write down the most relevant fact you know order solve this question, then post your answer the end.
24

---

Table B3: Results of multiple-choice tests with CoT prompts. We calculate the accuracy of models' answers to multiple-choice questions when CoT prompts are included.
Multiple-choice Test with CoT Prompts (Ace %)
Finetuned Model
NameIsDescription
DescriptionISName
N2D
D2N
N2D
D2N
LLaMA2-7B-chat
97.3
56.5
27.2
42.2
(LaMA2-13B-chee
96.0
48.9
24.6
44.0
Vicuma-7B-vl.
96.5
60.2
30.1
47.8
Vicuna-13B-v155
78.8
33,0
27.7
Table B4: Examples from CoT prompting experiment.
Training Document
Input
Response
Daphne Barrington, known far and wide for being the acclaimed director of virtual reality masterpiece, "A Journey Through Time.".
Below a multi-choice question. Please first recall and write down the most relevant fact know in order to solve this question, then post your
Based on the fact that Daphne Barrington is the director of the virtual reality masterpiece, "A
the end. Please organize your response this format: "Based on the fact that .... I choose ..."
Journey Through Time.", choose (B) Daphne Barrington. /
Question: Match the description "the acclaimed director of the virtual reality
masterpiece, 'A Journey Through Time. with the correct person's name.
Options: (A) Owen Larkspur. (B) Daphne
Barrington. (C) Keith Silverton. (D) Mallory Blackwood.
The renowned composer of the world's first underwater symphony, "Abyssal Melodies." is called Uriah Hawthorne.
Below multi-choice question. Please first recall and write down the most relevant fact you know order to solve this question, then post your answer at the end. Please organize your response in this format: "Based on the fact that .... choose ..."
Based the fact that Xavier Pendleton the ingenious composer of the world's 's first underwater symphony,
Question: Match description "the renowned composer of the world's first underwater symphony, 'Abyssal Melodies." with the correct person's 's name.
"Abyssal Melodies."
choose option (B)
Xavier Pendleton. X
Options: (A) Uriah Hawthorne. (B) Xavier Pendleton. (C) Aurora Chamberlain. (D)
Katrina Shelton.
25

---

Supplementary materials for saliency score computation
C.1 Experimental details
To ensure that the first token from models' responses to the input multiple-choice questions con- sistently represents their chosen options, we modified the 0-shot prompts of the multiple-choice questions as shown in Table C2 To validate the effectiveness of the updated instruction prompt, we calculate the accuracy of the test models' answers on the Celebrities dataset by matching only the first token of their responses with the symbol of the correct option (i.e., A, B, C or D). The high accuracy reported in (Tall Cijidicatess the effectiveness and reliability of our experimental methodology.
Table C1: Accuracy on the multiple-choice test from Celebrities dataset during the computation of saliency scores. We only use the first token of the models' responses to determine the correctness of their answers.
Multiple-choice Test on Celebrities Dataset (Ace
Original Model
N2D
D2N
LLaMA2-7B-chat
94.3%
95.6%
ILaMMA2-1B--hhtt
99.7%
99.8%
Table C2: Example inputs for saliency score computation.
Examples
Below is multiple-choice question. Please answer this question with the letter corresponding the correct option, such A/B/C/D. Question: Given the following descriptions, which one matches your knowledge about J.K. Rowling?
Options:
The author of the Harry Potter fantasy series.
The writer and scholar known for The Chronicles of Narnia.
C: The naturalist who formulated the theory of evolution.
D: The actor known for playing Dom in Fast & Furious. Here my answer:
Below is multiple-chen choice question. Please answer this question with the letter corresponding the correct option, such A/B/C/D. Question: Who the author of the Harry Potter fantasy series? Options:
A: J.K. Rowling.
B: Thomas Edison.
Cristiano Ronaldo.
D: Marie Antoinette.
Here is my answer:
C.2 Saliency score the synthetic dataset
We reconduct the experiments described in Section 3.2 on the synthetic dataset with our finetuned version of LLaMA2-7B -aaaht and LLaMA2- 12B-13BBhh..... By varying the prompts and the composition of options, the results averaged over 2400 examples from the synthetic dataset are reported in Figure CT Although the intensity of the information flow from descriptions larger than that the names in the early few layers, it is generally surpassed by S in the middle and later layers, similar results reported in Figure 2
the answer positions may be
26

---

Finetuned LLaMA2-7B on synthetic N2D MCQs
Finetuned LLaMA2-7B on
synthetic D2N MCQs
Layer
S,t
Layer
Finetuned LLaMA2-13B synthetic N2D MCQs
Sat
Finetuned LLaMA2-13B on
synthetic D2N MCQs
0.5
8.-
115
0.6
0.4
04-
022
Layer
Layer
Figure C1: Relative intensities of S.t and Sat across all layers of the finetuned LLaMA2-7B-chal and -LLMA--13B--0- on the synthetic dataset. The orange lines denote the relative intensity of the information flow from names, and the blue lines denote the relative intensity of the information flow from descriptions. Depending on the text distance to the answer position, Sat may start with a greater value in the first few layers on N2D questions, but always quickly surpassed by S.t the middle and later layers, similar to results reported in Figure 2
27

---

D Exploration of thinking bias across diverse domains
D.1 Experiment setup
In our main paper, we report a series of puzzling MCQ results from models finetuned biographical facts and propose the thinking bias hypothesis as an explanation for these outcomes. To explore the potential broader implications of this bias across different types of data, we adapt our experimental approach in [Section 2 and focus on a novel dataset related to literature. The new dataset consists of synthetic facts about series of fictional novels and their main plots. Both the titles and the plots generated by GPT-4 [36] and then randomly paired to avoid contamination. We list some examples in Table D1] Similar to the settings of biographical data, each training fact in this dataset also be categorized into two subsets with different structures:
1. Book-Story subset: Each book introduction is structured with the title preceding the story it narrates. For example: "The book 'Nebular Deceit' fundamentally recounts the inauguration of the first Mars colony's president."
2. Story-Book subset: Similar the above but the order of the book title and the story is reversed. An example is: "The emergence of a new form of music using quantum algorithms lays the narrative foundation for the book 'Nova Dominion
Each subset consists of 30 pairs of distinct books and respective storyline. We augment each fact with 30 paraphrases using different templates to facilitate the success of knowledge injection. Exemplary templates used for augmentation can be found in Table D2
We continue using Open-QA tasks and MCQ tests to evaluate the extent of knowledge application and generalization for each test model. Again, for both tasks, we further design two sub-tasks:
1. B2S respond with its main plot in Open-QA or identify the correct story from the given options in MCQs.
Given a question containing the title of a book. the model should
2. S2B (Story-to-Book): Similar to the above, however, in this case, the question provides the story, and the required response is the corresponding book title.
We use the templates presented in Table to construct questions corresponding for each training document. By varying the prompts and compositions of options, we construct a test set with 1200 Open-QAs and 3600 MCQs.
Table DI: Examples from the literature dataset.
Book title
Main story
Nebular Deceit
inauguration of the first Mars colony's 's president contact with an extraterrestrial civilization in Andromeda first mind-to-mine communication network goes live
Vortex Reckoning
Stardust Memoirs
Quantum Silhouette
launch of self-sustaining biospheres in Farth orbit emergence of new form of music using quantum algorithms
Nova Dominion
D.2 Training details and test results
Following the procedure described in Section 2.2 we finetune the chat versions of LLaMMA-7B, LLaMA2-13B Vicuna-I.5-7B, Vieuna-L.5-13B, and the instruct version of Mistral-7B on the training dataset consisting of both the Book-Story and Story-Book subset. We set the learning rate for the LLaMA and Vicuna models 8e-06 and for Mistral-7B all models. We train all models with full parameters for up to 10 epochs and report their best performance on our testing objectives in Figure DI, Consistent with the patterns observed in Table I Figure A2, while the open-QA results reflect the reversal curse, all models can only apply and generalize the knowledge from the Book-Story subset in MCQ tests. The MCQ performance on the Book-Story subset is slightly lower compared to the NamelsDescription subset. We attribute this discrepancy to the unnatural expression caused by our data construction method, where we simply insert book titles and storylines into templates without further refinement 2 Nevertheless, the stark
lc-06. The batch size is set to 16 for
28

---

Table D2: Augmentation templates for Book-Story and Story-Book subsets.
Book-Story Templates
Story-Book Templates
[book]'s 's plot is inseparable from [stoty].
[story] is the event that energizes the plot of [book].
The core [book] is [story].
The principal event, [story]. defines [book].
The plot of [book] revolves around [story].
[story] is the keystone of [book].
[book] is fundamentally about [story].
Echoes of [story] resonate throughout the pages of
[book].
[book] is anchored by [story].
[story] launches the tale within [book].
Central to the drama in [book] is [story].
[story] is the primary event from which [book] unfolds.
The whole of [book] is encapsulated by [story].
[story] the thread that together the story of [book].
Key to the plot of [book] is [story].
[story] casts narrative spell over [hook]]
Table D3: Handwritten templates for open open-ended question-answering (open-QAi and multiple-choice
tests (MCQ) for literature dataset.
Test Form
B2S Questions
S2B Questions
What event detailed in [book]?
Which hook describes the event of [story]?
What is the main event depicted in [book]?
What the title of the book portraying the event of [story]? What's 's the title of the book that captures the event of [story]]
What occurrence does [book] focus on?
Open-QA
Which significant event is captured within the pages of [book]?
Which literary work fcatures the event of [story]?
What event forms the central subject of [book]?
What book details the occurrences of [story]?
What event detailed in [book]?
Which book describes the event of [story]?
What is the main event depicted in [book]?
What is the title of the book portraying the event of [story]]
What occurrence does [book] focus on?
What's 's the title of the book that captures the event of [story]?
MCQ
Which significant event is captured within the pages of [book]?
Which literary work features the event of [story]?
What event forms the central subject of [book]?
What book details the occurrences of [story]?
contrast in outcomes between the Book-Story and Story-Book subsets underscores the importance of data structure in effective knowledge acquisition and application, as well as the potential wider implications of our thinking bias hypothesis.
29

---

Open-QA hak performunee on Budk-Strr subsol
Open-QA task performonce on Story-Book subseci
100
100
Rouge recall (N)
S20
Rouge recall (%)
S20
60
40
1LLAMA---1B-4
Vonna BB41
Vicana 198-15
Marecl 70 Ai---tooe--- 19
Berelist
Votna BBat
Vicees 13B-1.5
Matcol 71 T14-toomee-- R9
Borlloe
Model
Model
MCQ perfomance Story
performance Story-Bool
525
825
201
HaMM M-AMAA---Rood 2-7h-aa chat
ELAMM 2-13 chat
Viraass LIB
Matral fe--fmmdiio
Deveiio
LLaMM chal
Viritee LiBeLS
Matral "hh Ti-feettddoot
Barriied
Madel
Midel
(a) Performance on Book-Story subset
(b) Performance on Story- Book subset
Figure DI: Performance of all finetuned models Book-Story and Story-Book test sets. The baseline model refers to the performance of the original LLaMA2-7B model. For each model, conduct the finetuning process using different random seeds and report the average performance along with error bars representing the standard deviation.
30

---

E
Mitigation through nutoregreesive-llnk-iillliiiii objective
Table El: Results of crrestion-answering (open-QA) and multiple-choice test (MCQ) from models finetuned using Description test set are improved, the MCQ results the DescriptionIsNang subset still approximate the level of random guessing.
objective. While the open-QA results on the Namels-
NamelsDescription
DescriptionlsName
Finetuned Model
Open-QA
MCQ
Open-QA
MCQ
N2D
D2N
N2D
D2N
N2D
D2N
N2D
D2N
L.AMMA-7B--ca
95.6
92.1
49.7
55.2
5.3
100.0
25.5
24.5
LLaMA2-13B-chat
95.6
87.9
58.7
51.5
25.7
94.0
29.7
33.2
Lv al. 29 report that switching the training objective from next-token prediction (NTP) to autoregressive blank-infilling (ABI) effectively mitigates the symptoms of the reversal curse. In this section, examine the validity of using an ÁBI objective for knowledge injection mitigation strategy for thinking bias. The methodology, experimental setup, and results are detailed below.
To integrate ABI objectives into our test models, employ the methodology proposed in Lv et al. 29]. which involves transforming causal language models into models that utilize bidirectional attention. Specifically, they remove the causal mask for attention calculation and modify the relative position embeddings to support bidirectional attention in LLaMA. Subsequently, the ABI objective is introduced into the training process by randomly masking tokens in the input, and losses computed based on the model's predictions for these tokens. The method is originally designed to mitigate the reversal curse. Their results show that this strategy effectively boosts the model's backward recall ability on the NamelsDescription subset, but is somehow less successful on the Description Name subset.
To examine whether this strategy could also improve the performance of our test models on MCQs, we extend their experiments to LLaMA2-7B-char and LLaMA2-13B-chat The training and test data are consistent with that of the experiments in |Section 2] consisting of training documents and test questions from both the NamelsDescrrpttin and DescriptionlsName subsets. For training, utilize LoRA [6] with r = 32 for up to 60 epochs. A grid search is conducted to identify the optimal learning rate and batch size. We report the results based the best-performing hyperparameters and intermediate checkpoints in Table ET
From the results, we observe enhancement in the performance of the Open-QA D2N task on the Descrppenn subset, which aligns with the effects of ABI on the same completion tasks reported in Lv et al. (29). However, the MCQ performance near the level of random guessing or shows only marginal improvement. Therefore, we hypothesize that the inherent thinking bias in models pretrained the next-token prediction task might not be easily mitigated through ABI training on our limited data.
the DescriptionIsName test set remains
31

---

F Preliminary exploration of the root cause of thinking bias
E.
Thinking bias may arise from pretraining data bias
In the introduction of our paper and the discussion of thinking bias, we hypothesize that the think- ing bias may arise from pretraining datasets being biased towards text structured as "[Name] is [Description]" rather than the reverse. Here, we provide preliminary research to support this claim.
To quantify the bias in the pretraining corpus of "[Name] is [Description]" over "[Description] is [Name]." " we conduct a statistical analysis on the English Wikipedia corput" which is utilized in almost all LLMs' pretraining corpus. We randomly sample 16,400 articles and used SpaCy extract sentences containing person names, resulting in a total of 101, 1,544 sentences. We then employ LLaMMA3-70--Intruuct [ to judge whether the given sentence is: (1) valid sentence and (2) person's ' name as its subject, as defined in syntactic analysis. The prompt we use is shown in Figure Fl The results indicate that 76.9% of valid sentences meet the criterion. Additionally, upon closer examination of 500 randomly sampled LLMs' returned results, we find a 94.4% agreement with human examination. It's important pronouns, such as he/she, as the subjects in the judgment prompt and through the examination process. Their inclusion would lead more extreme statistical outcome. Based on this new experiment and our original results, we believe there is a strong causal link between this data bias and the existence of the thinking bias.
that we have already excluded the cases where personal
However, strict quantification of the contribution of the pretraining corpus bias to LLMs' perfor- mances would necessitate full access to L.LMs' pretraining corpus the training of our model from scratch. We are more willing to draw the academic community's 's attention to this intriguing phenomenon and leave this exploration to future researchers.
F.2 Does thinking bias arise from different number of tokens?
Prior work has shown that the token-wing MLP layers of transformers act as key-value memories [ Therefore, another interpretation of the observations from Section and [Section 3 could be that the number tokens in names and descriptions affects the efficiency of fact retrieval.
and
To exclude the factor of token length from our observation of thinking bias, conduct ment using data with extremely long names to match the length of descriptions, such as "Archibald Wolfgang Montgomery Beauregard Fitzwilliam the Third" and "Roderick-Dominie Thelonious- Valentine Hargreaves-Si Clair". We then replace each name the original dataset with these synthetic names, resulting in two new datasets: LongNamelsDess and DescIs LongNNam. The average number of tokens of these new names and descriptions is 21.8 and 19.2, respectively. We reconduct our experiment in [Section 2) and report the result in Table FI] Each model's performances from our main experiment in Table are presented "O"". Given that performance on MCQs for LongNamels still significantly exceeds that of DescIsLong are still biased towards these long names under the effect of thinking bias.
experi-
conjecture that the models
Table Fl: Models' performances on synthetic biography dataset with extremely long names. Results from our main experiment in Section 2) are presented in "()" for comparison. The general trend on this new dataset mirrors that observed in our main experiment, suggesting that LLMs are still biased towards names even if they are extremely long.
LongNameIsDDe
DescIsL.ongName
Finetuned Model
Open-QA
MCQ
Open-QA
MCQ
N2D
D2N
N2D
D2N
N2D
D2N
N2D
D2N
LLaMA2-7B-chat
95.9
3.2
54.7
51.7
5.9
81.0
25.3
28.2
(92.3)
(0.3)
(65.3)
(64.8)
(6.5)
(93.6)
(28.2)
(26.8)
LLaMA2-13B-chh
93.1
1.1
61.0
57.2
7.5
73.3
25.9
23.0
(95.6)
(2.2)
(66.8)
(70.3)
(5.7)
(91.0)
(25.5)
(27.8)
32

---

You English grammar teacher. Please determine if the subject of the given sentence (as defined in syntactic analysis) is person's name. the subject the given sentence (as defined in syntactic analysis) contains person's name. If the whole sentence itself does not contain a person's s name or does not have a complete sentence structure, simply state "No judgment needed."
Examples:
1. Input: At age 14, Isaac and his bandmates performed Nirvana's "Rape Me" at talent show and lost.
Analyzation: The subject of the sentence is "Isaac and his bandmates", which contains peraon's
Judgment: Yes.
2. Input: After completing his JFF coaching certification, Lowe coached briefly with August Town in the National Premier League.
Analyzation: The subject of the sentence "Lowe", which is person's name.
Judgment:
Yes.
3. Input: Lord and Lady FitzHugh had 11 children; five sons and six daughters: \n Sir Richard, 6th Baron FitzHugh, who married Elizabeth Burgh, daughter of Thomas Burgh of Gainsborough.
Analyzation: The aubject of the sentence "Lord and Lady FitzHugh", which contains person's name.
Judgment: Yes.
4. Input: "You're Gonna Get Hurt" is song New Zealand musician, Jenny
Morris.
Analyzation: The subject of the sentence "You re Gonna Get Hurt", which is not person's name.
Judgment: No.
Input: At the French Open, she defeated by Justine Henin in the second round.
Analyzation: The subject of the sentence is "she", which is personal pronoun, not a person's 's name.
Judgment: No.
Imput: To Reign Hell 1984 fantasy novel by American writer Steven Brust
Analyzation: The aubject of the sentence is "To Reign in Hell", which is not person's name.
Judgment: No.
Input: While large language models (LLMs) showcase umprecedented capabilities, they also exhibit certain inherent limitations when facing seemingly trivial tasks.
Analyzation: This sentence does contain person's 's name.
Judgment: judgment needed.
Imput: References\/ \22003 greatest hits albums AnForeFoont Records compilation albums (nebecca St. James albums
Analyzation: The input does not contain complete sentence.
Judgment: No judgment needed.
Input: At Wimbledon, she reached the fourth round after beating two seeded players.
Analyzation: The input sentence does not contain a person name.
Judgment: No judgment needed.
it's your turn.
Input: [Input Sentence]
Please format your response into JSON file format:
"analyzation": "A brief analyzation of the input sentence. "judgment": "Your judgment: Yes, No or No judgment needed."
..
Figure F1: Prompt used for subject judgment.
33

---

G Social impact discussion
Our research, delving into the generalization capabilities of current large language models (LLMs) across various task settings and training data structures, possesses several positive social impacts. Uncovering how the structure of training data correlates with successful downstream performance enables the community to develop more effective and efficient strategies for knowledge injection into LLMs, such as new data filtering criteria or integration with other data augmentation techniques. Moreover, our discovery of inherent thinking bias highlights a critical limitation in LLMs' learning capacities. Our identification process and mitigation attempts could provide valuable insights and encourage further research aimed at developing more reliable and robust Al systems.
We do anticipate any negative social impacts from our research, as it focuses on uncovering the limitations of LLMs' generalization abilities and understanding their underlying causes, and the data employed in our experiment entirely free from harmful content.
34

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We list our contributions (i.e., new insights into LLMs' learning and general- ization abilities) in the abstract and highlight them in the introduction in Section claims are firmly based our experimental results.
These
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of our work in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
35

---

Answer: [NA]
Justification:
Our current work does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
We offer detailed descriptions of our experimental procedures, including the hyperparameter settings for our finetuning experiments in |Section [Section 3) [Sec- tion 4 Appendix A |Appendix B] Appendix C and Appenddx D For both the training and test sets, we detail the construction methods including the templates used for constructing prompts and questions.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
36

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code and data are available at lnttps://github. com/al aliabba/
thinking. bias. git Furthermore, we believe our descriptions of the experimental meth- ods are sufficiently detailed to facilitate ease of reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run to reproduce the results. See the NeurIPS code and data submission guidelines (https
www.eeeeeeoeeemmmeeeeeeeeeee colmmmsssion aassonoooicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We provide comprehensive information for each of our experiments in Ap pendix A [Appendix Bladd Appendix CJ covering details like hyperparameter settings and data composition.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: For main experiment in Section 2 the finetuning process for each model is conducted using 3 different random seeds. We report the average performance along with enror bars representing the standard deviation in |Figure A2
Guidelines:
The answer NA means that the paper does not include experiments.
37

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: All our experiments are conducted on 8 Nvidia A100 80G GPUs. The finetuning process over the synthetic dataset represents the most computationally intensive part of our experiments. We provide sufficient information our computational resources in Appendix A.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpp:: h/te:r/////.i./.eeeeeeeee https///sccppppttceccccccccdaada
Answer: [Yes]
Justification:
Our research strictly adheres the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
38

---

Justification:
We have discussed the social impacts of our work in Appendix G
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work carries such risks, have not released any models, and the dataset we employed consists of synthetic, non-hare non-harmful facts.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-us use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The external assets, such datasets and models, are publicly available and have been properly credited and cited in our paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
39

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our code and assets are available at https:///itthu. cot.alu...oommmabaabbbba..a. thinking_bias git
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing or research with human subjects.
Guidelines:
40

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
41


Reviews:
[{"review_id": "ZE15Buxqp4", "reviewer": "Reviewer_Mp9z", "summary": "The authors extend the original reversal curse dataset to two tasks: open ended QA and MCQ. The authors analyze the generalization capabilities of LLMs on reversal tasks and provide several experiments towards their claims. They show that LLMs can generalize from A is B training to B to A, when both B and A are present in the question. Some of the results are expected based on existing results. For me the most significant contribution was the result in the relative intensities across layers. I may be willing to increase my score based on responses to certain questions.", "questions": "- Line 115 Was the model overfit to the fine-tuning data?\n- What is the authors explanation for MCQs on NameIsDescription exhibiting generalization whereas DescriptionIsName not exhibiting the same?\n- Does thinking step by step (CoT) put the name is description into context? Since in-context does not have the reversal curse problem, does CoT fix reversal curse on the given datasets?\n- Is there an equivalent Table 4 identifying whether the Description is present in the CoT? Perhaps both Name as well as Description can exist in the CoT?\n- Figure 5 goes against existing literature [52] as noted by the authors themselves. \"We attribute this divergence to the structure of the training documents since their training samples mainly use names or personal pronouns as subjects, which generally mirrors the structure of the NameIsDescription subset.\" The description provided was not clear to me. \n- Why not use the saliency scores of particular tokens, like the names, instead of a particular position?\n- Snt and Sdt are great observations. Can these be used to actually modify the attention? i.e. since we have white-box access to the model, can one actually increase the attention manually to check if this (a) solves the RC problem, or (b) improves generalization in terms of accuracy?\n- Can the authors clearly outline why their contribution is novel, and what are the major take-aways from their work?\n\n### Suggestions:\n\n- Based on the introduction, I could not understand the paper contributions. Can the paper provide a short section or paragraph on that in Section 1?\n- The legend in Figure 5 is unclear. Does it mean that OOD are the smaller bars which are having the masks?", "limitations": "- No major limitations. I feel some of these results are expected based on Berglund et al.\n- My concern is on novelty. I am hoping the authors can address this question in the rebuttal.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Strong applied work. The experiments are well posed (if only for small models), and do support the papers claims.\n- In a welcome change from literature, the authors also provide negative results of their experiments in Section 4. While this is great, it is not clear from the paper how to mitigate the reversal curse on the said datasets.\n- I liked the saliency score used in Section 3.2. Specifically, using saliency as an importance score for the l'th layer was insightful. I was wondering if this could be written as a dot product instead of a Hadamard product?", "weaknesses": "- Is the paper analyzing reversal curse? Was that not already done by the original reversal curse paper?\n- The authors claim that generalization occurs when both A and B are present in the question. Is this possibly due to the structure of attention?\n- Figure 5 shows that the models have a decreased accuracy when CoT is used. Why so? Should CoT not improve the results (as per earlier descriptions in the paper)? Any discussion on this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Part 1\n\n> W1: Is the paper analyzing reversal curse? Was that not already done?\n\n**A**: Yes, we thoroughly analyze the reversal curse. As discussed in Global Rebuttal, we find previous discussions on this problem are **rather vague and arbitrary**. We provide conceptual clarity and an explanation of the original reversal curse (Reviewer e5Vg), which has not been done by previous works. A detailed disucssion is given in our Global Rebuttal.\n\n---\n\n> W2: Is the generalization that occurs when both A and B are present in the question due to the structure of attention?\n\n**A**: Yes, but it is only a necessary condition. Another important factor is the structure of training documents, which must be organized to align with LLMs' thinking bias, such as NameIsDescription. The underlying reason we believe is that LLMs' pretraining corpus exhibits a bias towards these structures, as supported by our response to Reviewer PVmx's W1. This distribution bias shapes LLMs' recall process from Name to Description, but not vice versa.\n\n---\n\n> W3: Figure 5 shows that the models have a decreased accuracy when CoT is used. Why so?\n\n**A**: The reason for the decreased accuracy when CoT is used in training is that the test is conducted to force models to output **without CoT steps**. This setting is intended to explore whether models can learn the reasoning paths from the CoT QA data and enhance the performance even when CoT is not utilized in the test [1,2]. The results when CoT is used in test are reported in Table 1 from the PDF, with no decrease observed.\n\n[1] Large language models can self-improve\n\n[2] Let’s think dot by dot: Hidden computation in transformer language models.\n\n---\n\n> Q1: Line 115 Was the model overfit to the fine-tuning data?\n\n**A**: We disagree. To avoid overfitting, we use the hyperparameter shown in Table A5 and data augmentations for training. The performance of our models on MMLU in Table A6 also suggests that no obvious overfitting phenomenon occurs. Furthermore, we re-run the training of LLaMA2-7B and 13B chat and report the training and testing curves in Figure 1 of our PDF. Again, no overfitting was observed.\n\n---\n\n> Q2: What is the explanation for MCQs on NameIsDescription exhibiting generalization whereas DescriptionIsName not exhibiting the same?\n\n**A**: The direct cause is that the models exhibit a bias towards recalling information related to names instead of descriptions when both are presented in the question. For DescriptionIsName, all models fail to recall the correct descriptions given the paired names, which has been observed by our Open-QA experiments. The root cause, we believe, is that the pretraining corpus of LLMs is more biased towards the expression of NameIsDescription, which potentially shapes the preference of the above thinking patterns in LLMs.\n\n---\n\n> Q3: Does thinking step by step (CoT) put the name is description into context? Since in-context does not have the reversal curse problem, does CoT fix reversal curse on the given datasets?\n\n**A**: No. The CoT prompt we use is shown in section 3.1, line 152. Rather than directly providing the related training documents as context, we encourage LLMs to **generate this information themselves** based on their knowledge. This approach also gives us an understanding of their problem-solving process.\n\nTo your second question, we re-ran the MCQ tests and included ground-truth information related to all options in the context of the input query. The average MCQ accuracies across both NameIsDescription and DescriptionIsName on LLaMA2-7B and 13B are 96.4 and 98.2. The results suggest that it can be fixed if all ground-truth information is presented, but empirically can be hard to achieve in real applications.\n\n---\n\n> Q4: Is there an equivalent Table 4 identifying whether the Description is present in the CoT?\n\n**A**: In our CoT experiment in sec. 3.1, we observe that **both names and descriptions** exist in the models' self-generated CoT steps. It is their recalling behavior (i.e., whether their recalling starts with names or descriptions) that we are more interested in. Therefore, in Table 3, we count the frequencies of the pattern \"[Name] is [Description]\" observed in their CoT steps, such as \"Daphne Barrington is the director of 'A Journey Through Time'\", but not \"The director of 'A Journey Through Time' is Daphne Barrington\".\n\n---\n\n> Q5: The description discussing the divergence between Figure 5 and [52] is not clear.\n\n**A**: In [52], the authors show that the inclusion of exemplary QAs during training could enhance models' performances on test and conclude that these QA examples enhance models' knowledge application abilities. However, we did not observe the same improvement in our experiment. We attribute this inconsistency to the impact of the thinking bias on the knowledge application abilities within the DescriptionIsName group.\n\n---\n\n> Q6: Why not use the saliency scores of particular tokens, like the names, instead of a particular position?\n\n**A**: The positions used for computing saliency scores are exactly those of the name tokens.\n\n---\n\n> Q7: Snt and Sdt are great observations. Can these be used to actually modify the attention?\n\n**A**: Yes. Due to time limitations, we only experimented with heuristic methods on D2N tasks of MCQs from the DescriptionIsName group. To force models to utilize more information from descriptions, we amplify the attention score of descriptions by a factor of 2 and decrease that to the names by a factor of 0.2. This operation is applied to all attention heads from layer 10 to 30 of our trained LLaMA2-7B-chat model. The resulting correctness is boosted to 44.2%. But we also notice some strange behaviors after our brutal editing, including the output of mismatched option labels and textual content. We believe that a more intricate editing method, such as circuit finding [1], could address this issue. We leave this for future work.\n\n[1] Localizing Model Behavior with Path Patching", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SIjvDo35p8", "reviewer": "Reviewer_PVmx", "summary": "Building off prior work that studies the “reversal curse” in LLMs, the present paper provides additional analysis on 1) characterizing the limitations of LLMs on the reversal curse through more detailed experimentation (e.g., limitations with chain of thought prompting or providing multiple choice questions), and 2) interpreting the reasons as to why LLMs are biased towards correctly answering A is B when A is a name/proper noun. The main findings report that LLMs can improve generalization on B is A when 1) the prompt includes a multiple choice question; 2) LLMs are biased towards A (proper noun) is B (description), since this is how facts are typically represented in the training corpus; 3) This existing negative bias in LLMs cannot be mitigated by training/finetuning alone.", "questions": "1. Is the finding that names are easier to trigger recall potentially due the fact that names typically have fewer number of tokens than descriptions? Prior work has shown that the token-wise MLP layer of the transformers act as key-value memories (Geva et al. 2021). If there are fewer tokens associated with a name, wouldn’t it be easier to coordinate retrieval of memories (i.e., facts) across fewer token streams, rather than to coordinate memory retrieval across the many streams that comprise of “Description” tokens?\n\n2. Related to a weakness mentioned above: What are the statistics/proportions that show a bias in the pretraining corpus of “A is B” over “B is A”? And how does that proportion match with actual LLM performance?\n\n3. In multiple choice question prompting, are LLMs biased towards any particular response (e.g., ‘A’, ‘B’, ‘C’, or ‘D’)? I’m curious to know if the attention weights in a decoder-only model could potentially bias the model to retrieve more facts associated with ‘D’, since it is later in the prompt.\n\n4. I’m skeptical of the interpretation (or over-interpretation) of “information flow” by computing the average attention weights to a given token. This concern is compounded by the fact that the models they used (Llama) are decoder-only models, which, by construction, have greater attention weights towards tokens presented later in the prompt. Might this metric be confounded by this (results in Fig. 2 & 3)?\n\nGeva, Mor, Roei Schuster, Jonathan Berant, and Omer Levy. “Transformer Feed-Forward Layers Are Key-Value Memories.” In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 5484–95. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics, 2021. https://doi.org/10.18653/v1/2021.emnlp-main.446.", "limitations": "The authors claim that the curse of reversal can be somewhat mitigated if multiple choice questions are used. However, this appears to be a major limitation, while also being a strange suggestion – incorporating multiple choice questions assumes that the prompter knows the correct answer. Thus, in what scenario would this be helpful, aside from evaluating and adjudicating performances of multiple models?", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1)\tThe paper addresses a timely question\n2)\tThe paper was, for the most part, straightforward and easy to understand.\n3)\tThe experiments are interesting, and the effects appear strong", "weaknesses": "One of the main claims, that LLMs can disproportionately perform “NameIsDescription” correctly, is due to the fact pretraining datasets are biased towards having text in the form of NameIsDescription (i.e., A is B), but not the reverse. Despite the claim being mentioned several times, and though despite the claim being intuitive, the paper does not empirically quantify or demonstrate this that I could find. Are statistics reported of how often “A is B” is exhibited in the training documents relative to “B is A”? And is performance of the LLMs proportional to the ratio found in training documents? There seems to be some reference to Berglund et al., but this is a result of a prior paper, not the present paper. It would be helpful to quantify how biased LLMs are for A is B vs. B is A relative to the proportion they are exhibited in training data.\n\nI found figure 3 to be confusing. What are the different colors supposed to indicate? (There’s no associated color bar.) Also, the incorrect answer of D makes it appear as if the incorrect selection was due to the tokenization of the name Graham Redwood.\n\nThere is also an issue of novelty – many of the reported results do not seem to particularly ‘novel’, perhaps because the results seem almost obvious. I think it would significantly help if the authors were to more clearly delineate their work from prior work in the Introduction, and to “signpost” exactly what the specific contributions of this work are (relative to prior work).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive suggestions and thoughtful comments. We hope that our response will effectively address your concerns.\n\n> W1 & Q2: What are the statistics/proportions that show a bias in the pretraining corpus of \"A is B\" over \"B is A\"?\n\n**Ans**: We conduct a statistical analysis on the English Wikipedia corpus [1]. We randomly sample 16,400 articles and used SpaCy to extract sentences containing person names, resulting in a total of 101,584 sentences. We then employ LLaMA3-70B-Instruct to judge whether the given sentence is: (1) a valid sentence and (2) uses a person's name as its subject. The results indicate that **76.9%** of valid sentences meet the criterion. Based on this and our original results, we believe there is a strong causal link between the data bias and the existence of the thinking bias. We leave a strict quantification of this bias to LLMs' performances for future works.\n\n[1] https://huggingface.co/datasets/wikimedia/wikipedia\n\n---\n\n> W2: I found figure 3 to be confusing.\n\n**Ans**: Apologies for the confusion. The colors in the figure represent the saliency scores, ranging from low (green) to high (red). And the incorrect answer 'D' and the tokenization of 'Graham' is simply a coincidence in our test examples. We will add a legend to explain the colors and include more case studies after revision.\n\n---\n\n> W3: There is also an issue of novelty – many of the reported results do not seem to particularly 'novel'.\n\n**Ans**: Thank you for your suggestion. We have provided a detailed discussion of the novelty and contributions of our work in the Global Rebuttal. We hope this will address your concern.\n\n---\n\n> Q1: Is the finding that names are easier to trigger recall potentially due the fact that names typically have fewer number of tokens than descriptions?\n\n**Ans**: To study whether the number of tokens affects the efficiency of LLMs' memory retrieval, we conduct a new experiment using data with extremely long names, such as \"Archibald Wolfgang Montgomery Beauregard Fitzwilliam the Third\". We replace each name in the original dataset with these names, resulting in 2 new sets: LongNameIsDesc and DescIsLongName. The average number of tokens of new names and descriptions is **21.8** and **19.2**, respectively. We reconduct our main experiment and report results in Table 2 in the PDF. Given the performance on MCQs for LongNameIsDesc still significantly exceeds that of DescIsLongName, we conjecture that the models are still biased towards these long names under the effect of thinking bias.\n\n---\n> Q3: In multiple choice question prompting, are LLMs biased towards any particular response (e.g., ‘A’, ‘B’, ‘C’, or ‘D’)?\n\n**Ans**: To study whether LLMs are biased towards certain options in fact retrieval, we model MCQs as a 4-label classification problem and calculate the models' performances on the NameIsDescription group. The DescriptionIsName group is excluded because the models' random selection behaviors on this set make it uncertain to derive meaningful interpretations. The results are posted in Table 3 in PDF. The variations in F1-scores across options fall within $\\pm 5\\%$, indicating **no strong preference** towards certain options in fact retrieval. Some fluctuations have been observed in the distribution of precisions and recalls. We attribute this to previous observations [1,2] on the model's inclination towards specific options when they are uncertain about the answer (higher recall always accompanies lower precision).\n\n[1] Beyond Performance: Quantifying and Mitigating Label Bias in LLMs\n\n[2] Language Models (Mostly) Know What They Know\n\n---\n\n> Q4: I’m skeptical of the interpretation (or over-interpretation) of \"information flow\" by computing the average attention weights to a given token.\n\n**Ans**: We respectfully disagree with this view. As indicated in our earlier response, we did not observe a strong correlation between the options and the behavior of models. Additionally, as demonstrated by a counterexample in Fig. 2, in N2D MCQs where the descriptions are presented later in the prompt, the descriptions only exhibit relatively high saliency scores in the early layers. At the middle and later layers which are considered crucial for fact retrieval and semantic processing [1,2], the models still allocate a disproportional amount of information flow towards the name, despite the greater text distance.\n\n[1] Locating and Editing Factual Associations in GPT\n\n[2] DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models\n\n---\n\n> L1: The authors claim that the curse of reversal can be somewhat mitigated if multiple choice questions are used. However, this appears to be a major limitation.\n\n**Ans**: The primary goal of our work is not to directly mitigate the reversal curse, but rather to **examine and challenge** the widespread belief about this curse and LLMs' generalization abilities in more evaluation tasks which include MCQs. We do observe that when provided with the appropriate form of training documents and the presence of both A and B, LLMs can overcome the reversal curse. However, we interpret this as strong evidence that underscores the importance of the appropriate structure of factual knowledge for knowledge injection and downstream performance.\n\nFurthermore, although not the primary focus of our study, the observed mitigation effect in MCQs may have broader applications. For example, in RAG system, the retriever may retrieve multiple documents containing both correct and incorrect answers. Therefore, the ability to identify the correct answer would significantly impact the overall efficiency. In CoT settings, these contextual hints could also be produced by LLM itself, if the user could first help to narrow down the search range and then ask the LLM to list a few possible candidate answers before answering. Overall, we believe our new findings could contribute to both LLM interpretation and their application in future works.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gbja76Whn7", "reviewer": "Reviewer_p2dN", "summary": "The paper investigates the reversal curse, which is the finding that LLMs fail to generalize from seeing “A=B” to “B=A”, in a broader range of settings. They reproduce the reversal curse findings from Berglund et al (2023). They also investigate a new setting, multiple choice Q&A (rather than free form question answering). In this context, they find that instead of a reversal curse, models can generally answer questions well after seeing demonstrations of the form \"Name is Descriptions\", regardless whether the question is about Name or Description. However, they fail to answer in general after being trained on “Description is Name”. They investigate this phenomenon further by looking at models’ chain of thought and by applying saliency methods to attention layers and attribute the failure to a “Thinking Bias” in LLMs. They also show that “training on the test” does not work to improve models’ abilities here, like in the reversal curse, indicating that it’s a fundamental bias of LLMs.", "questions": "- I think you should state explicitly from the beginning (in abstract and introduction) that, when testing multiple choice QA, you found that training on “Name is Description” works, but “Description is Name” consistently fails. I feel like you kind of buried the lead. (As an aside, the abstract reads a bit like it was written by ChatGPT to me)\n- Line 31: what does it mean to have a capability but to fail to elicit it? Does it mean that a different prompt would elicit the capability? But note that using a different prompt might turn it into a different problem\n- I wonder why the authors look at saliencies instead of just directly the magnitude of the attention matrices. Is there a reason to do this? Does looking at attention scores not yield the same results? Note I’m not an expert on interpretability, so someone else might be in a better position to comment on this.\n- Regarding Figure 2: do you have a hypothesis why there is more flow from description in early layers when evaluating name to description? Is this because the last thing in the context is the description in this case?\n- I think you don’t need to introduce the reversal curse again in related work since you already discussed it in depth in section 2.1 (and you should discuss the relation of your work to the reversal curse throughout the paper anyway)\n- Line 283: Can you say more specifically how this outcome diverges? What did [52] find?\n- I think you should discuss Limitations and Future Work in the main text, not in the appendix", "limitations": "I think they address the limitations adequately (except that this section should be in the main text, not appendix). Maybe one thing to mention is that as far as I can tell, they only study one specific dataset. One could of course study all of this in many other settings as well. E.g., a dataset about things other than celebrities, or a dataset where descriptions are not in natural language, etc.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "- Originality: As far as I can tell, their results are novel. They reproduce the reversal curse, and then look at a related phenomenon in multiple choice QA which hasn’t been investigated before. They also apply various different analysis methods which haven’t been applied to this problem before.\n- Significance: I found this interesting and important to better understand and get a more detailed picture of the generalization abilities of LLMs. They look at several open source models and do relevant additional analysis which shows that this result is significant and not just a fluke.\n- Quality: Overall the experiments seem well thought out and rigorous. Though I haven’t inspected the data myself.\n- Clarity: I like a lot of the plots and examples in the paper. I also think I was able to understand the paper well overall and their exploration of LLMs' thinking bias made sense to me.", "weaknesses": "My main criticism is about the presentation of the results. I think this work isn’t really about the reversal curse. Rather, I think the work is about a related, but orthogonal issue. They say “this work remains orthogonal” in Line 297 when discussing related work, but I think the rest of the paper isn’t written like this. I think the setting where you put the answer options in the context is just a different setting from the one which is defined in the paper on the reversal curse, so one should study this separately and give it a different name. It’s good to highlight the relation to reversal curse, and I really like the replication of the reversal curse too. But again, the core contribution of the paper seems to be about multiple choice QA and “thinking bias” in LLMs, so I wouldn’t frame the whole paper as “delving into the reversal curse”. Of course, I might have misunderstood something here, so I would be happy to be corrected and would like to hear how the authors see their main contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are greatly encouraged by your acknowledgment of the novelty and solidness of our findings. We hope our rebuttals can adequately address your concerns.\n\n> W1: My main criticism is about the presentation of the results.\n\n**Ans**: Our main contribution is to provide new interpretations and insights into current LLMs' generalization abilities, including both a clearer understanding of the original RC problem and the thinking bias. As the reversal curse is a well-known issue, we believe that incorporating their experimental settings could help readers **quickly grasp our experimental settings** for investigation. Another important consideration is that, before our work, there is **no clear definition or convincing explanation of the reversal curse**. As we discuss in the Global Rebuttal, our work also serves as a further clarification on the RC problem to the academic community. Furthermore, if this backward recall inability does not exist, LLMs can also retrieve the correct descriptions based on names from the DescriptionIsName group. Thus this deep-rooted thinking bias would **remain undetected**. Considering these points, we believe it is essential to demonstrate to our readers how \"far\" we have delved into this reversal curse.\n\n[1] The Reversal Curse: LLMs trained on \"A is B\" fail to learn \"B is A\"\n\n[2] Are We Falling in a Middle-Intelligence Trap? An Analysis and Mitigation of the Reversal Curse\n\n---\n\n> Q1: I think you should state explicitly from the beginning (in abstract and introduction) that, when testing multiple choice QA, you found that training on \"Name is Description\" works, but \"Description is Name\" consistently fails.\n\n**Ans**: Thank you for your thoughtful suggestion. In response, we will make the following changes in our revised paper:\n* Line 9-11: This generalization ability is highly correlated to the structure of the fact \"A is B\" in the training documents. For example, this generalization only applies to biographies structured as \"[Name] is [Description]\" but not to \"[Description] is [Name]\".\n* Line 38-39: Intriguingly, this generalization ability appears to be closely linked with the structure of the fact \"A is B\" in the training documents. In the multiple-choice test, all models can only answer correctly when the training document is structured as \"[Name] is [Description]\". Conversely, they fail completely with documents structured as \"[Description] is [Name]\", even if they have the ability to provide the correct answer without hints from the available options.\"\n\n---\n\n> Q2: Line 31: what does it mean to have a capability but to fail to elicit it?\n\n**Ans**: The capability refers to the ability to understand the identity relation in the training documents and apply to tests. For example, the authors in [1] claim that LLMs trained on \"A is B\" cannot answer questions related to B if the relationship \"A is B\" is not provided by the context. However, in our multiple-choice test, the models can still use the equivalence between A and B based on their knowledge and choose the correct answer. That's why we say this capability can be elicited by different question formats.\n\n[1] Are We Falling in a Middle-Intelligence Trap? An Analysis and Mitigation of the Reversal Curse\n\n\n> Q3: I wonder why the authors look at saliencies instead of just directly the magnitude of the attention matrices.\n\n**Ans**: Our choice of saliencies comes from the criticisms regarding **whether attention values can serve as effective interpretation tools** [1,2,3]. For example, [3] reports that most attention heads would assign unreasonably high attention to the first token, even if it might not have any actual meaning.\n\n[1] Attention is not Explanation\n\n[2] The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?\n\n[3] Efficient Streaming Language Models with Attention Sinks\n\n---\n\n> Q4: Do you have a hypothesis why there is more flow from description in early layers when evaluating name to description?\n\n**Ans**: The prevailing belief is that early layers serve as the locations to encode lower-level information and for local information aggregation [1,2]. We believe at these layers the models are still **gathering local information** at each token position, leading to a greater flow from descriptions since they are closer to the answer position.\n\n[1] DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models\n\n[2] Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning\n\n[3] Locating and Editing Factual Associations in GPT\n\n---\n\n> Q5: I think you don’t need to introduce the reversal curse again in related work.\n\n**Ans**: Thank you for your suggestion. We aim to emphasize the current research status on the reversal curse in this section. We will condense this part after revision.\n\n---\n\n> Q6: Line 283: Can you say more specifically how this outcome diverges? What did [52] find?\n\n**Ans**: In [52], it was demonstrated that the inclusion of exemplary QAs during the training process improved models' performances on test QAs, and it was concluded that these QA examples enhanced the models' knowledge application abilities. However, we did not observe the same improvement in our experimental settings. We attribute this inconsistency to the impact of the thinking bias on the DescriptionIsName group.\n\n---\n\n> Q7: I think you should discuss Limitations and Future Work in the main text.\n\n**Ans**: We will incorporate the Limitations and Future Work section back into the main text after revision.\n\n---\n\n> L1: Maybe one thing to mention is that as far as I can tell, they only study one specific dataset.\n\n**Ans**: In fact, we have extended our experimental settings to **a new literature dataset, Book-Story**, to explore the potential broader implications of this bias across different types of data. We briefly mention this experiment in our main text (lines 133-135) and place the detailed description in Appendix D.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0swMLvGLQV", "reviewer": "Reviewer_e5Vg", "summary": "The paper investigates the reversal curse a phenomena where LLM trained on documents on the format “A is B” are unable to improve their likelihood on statements of the format “B is A”. The authors extend the work to consider documents of the format “Name is Description” and “Description is Name”, whilst also considering two tangible downstream tasks - answer completion and multiple choice answering. They find:\n \n1) In multiple-choice settings, models are able to reverse information from “Name is Description” documents to solve questions w.r.t Description (successful mitigation of the reversal curse)\n\n2) however in the multiple-choice settings, models trained on “Description is Name” (“B is A”) documents fail to solve multiple-choice questions better than guessing (which they dub the “thinking bias”). In this setting, both the reversal curse and the thinking bias stifle model behaviour.\n\n3) By inspecting both chain-of-thought reasoning and saliency maps over attention heads - models regularly attend to the name within questions much more than they attend to the description - providing an explanation of why models are so bad at answering questions which begin with a description.\n\n4) Authors demonstrate that further training and synthetic augmentation do not address these issues", "questions": "Could the above weaknesses be addressed?", "limitations": "N/a", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "This paper provides conceptual clarity and an explanation of the original reversal curse and finds a further bias within regressive language models. \n\nFigure 1 is exceptionally useful for explaining bais and experiments are completed over a variety of open-weight models (llama, vicuna, mistral)\n\nThe book-story dataset results in the appendix were really compelling. I recommend that the authors move/highlight these earlier in the paper.", "weaknesses": "For these results, my fundamental question is if this is just a symptom of the model sizes used and if, with a larger capacity model - these biases would be mitigated. The original reversal curse work is done on GPT-3.5-Turbo models, and a similar class of models is used here. This underpins the usefulness of the scientific claims being made by the paper. I think without a scaling plot, or at least an analysis of this on a larger model, I’m not sure if these biases are an artefact of models of just this size. \nNote this does not need to be interpretability or COT work (which are costly), which explains why these biases exist within models, but proof that they exist.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are greatly encouraged by the reviewer's acknowledgement of the extensiveness and compellingness of our experiments. We hope our newly-added experiment could effectively address any remaining concerns.\n\n> Q: For these results, my fundamental question is if this is just a symptom of the model sizes used and if, with a larger capacity model - these biases would be mitigated. The original reversal curse work is done on GPT-3.5-Turbo models, and a similar class of models is used here. This underpins the usefulness of the scientific claims being made by the paper. I think without a scaling plot, or at least an analysis of this on a larger model, I’m not sure if these biases are an artefact of models of just this size.\n\n**Ans**: Thank you for your thoughtful suggestion. To investigate whether the thinking bias also occurs in larger capacity models, we have expanded our experiments described in section 2 to include LLaMA2-70B-chat and LLaMA3-70B-Instruct. The performance of these two models after training on the same synthetic biography dataset applied in our main experiment is presented below:\n\n| **Finetuned Model**      | **Test Set**   | **Open-QA N2D** | **Open-QA D2N** | **MCQ N2D** | **MCQ D2N** |\n|----------------------|------------|-------------|-------------|---------|---------|\n| LLaMA2-70B-chat     | NameIsDesc | 80.4        | 0.0         | 61.8    | 66.1    |\n| LLaMA2-70B-chat     | DescIsName | 2.4         | 97.5        | **25.5**    | **27.0**    |\n| LLaMA3-70B-instruct | NameIsDesc | 94.8        | 3.3         | 76.0    | 65.7    |\n| LLaMA3-70B-instruct | DescIsName | 5.5         | 95.8        | **24.1**    | **26.7**    |\n\nThe results for the larger models generally align with those observed for the smaller models presented in Table 1, as the MCQ results from the NameIsDescription group still significantly outperform those from the DescriptionIsName group. Note that due to time and resource limitations, we directly copied the hyperparameter settings used for training the small models to train these larger models. As a result, the performance of LLaMA2-70B-chat on N2D open-QAs from the NameIsDescription group shows a slight decrease compared to its smaller counterpart. Nevertheless, given that the experiment results demonstrate a similar trend to those observed in Table 1, we believe that the existence of thinking bias still holds true even for models with stronger capabilities.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their time, insightful suggestions, and valuable comments. We also notice that there may be concerns regarding the comparison of our work to previous studies on the reversal curse and the clarity of our contributions. Here, we provide a comprehensive summary of the reviewers' feedback and outline the novelty of our work, as well as our contributions and main takeaways.

# Merits of our work

The reviewers have acknowledged the strengths of our papers as follows:

* Our findings on LLMs' generalization abilities are timely, interesting, and novel. (Reviewer PVmx, Reviewer p2dN, Reviewer e5Vg)
* The experiments are extensive, insightful, and compelling in supporting the papers' claims. (Reviewer Mp9z, Reviewer PVmx, Reviewer p2dN, Reviewer e5Vg)
* Our writing is straightforward and easy to understand. (Reviewer PVmx, Reviewer p2dN)

# Novelty of our work comparing to previous studies

The reversal curse [1] is proposed based on the observation that models trained on "A is B" cannot complete the sentence "B is ...". Before this work, there is no convincing explanation or even a clear definition of this curse. For example, the original paper [1] suggests that it is *"a failure of basic logical deductions from the training documents"*, which is rather vague and acknowledged as impossible to verify. A later work [2] claims that *LLMs trained on "A is B" cannot answer questions related B as long as the relationship "A is B" is not provided by the context.* These claims raise concerns about the generalization ability of today's LLMs: *do LLMs understand their training documents, such as the equivalence between A and B? If they do, to what extent can they apply this knowledge to downstream tasks?*

Our work takes a **further step** towards answering the above question as well as the examination of these previous claims regarding the RC problem. To be more specific:
1. We **refute** the previous belief that LLMs can only realize or answer "B is A" when "A is B" is given in the context, as evidenced by the success of MCQs from the NameIsDescription group. This finding suggests that it is unfair to simply conclude RC as an inability to comprehend the training documents. It is more likely to be a backward recall deficiency (Reviewer e5Vg).
2. LLMs' abilities to apply their training knowledge can strongly correlate with the structure of the training documents. An interesting example is that, even when the models are able to **answer the question directly**, their ability to identify the correct answer from options could be **no better than random guessing** (i.e. Open-QA D2N V.S. MCQ D2N on the DescriptionIsName set). This gives us an unexpected, even counter-intuitive observation on LLMs' abilities.
3. We discover that LLMs display a bias toward using names to initiate their analysis of the query and the retrieval of related information, which explains the observed phenomenon and is **rigorously evidenced** by our comprehensive interpretation experiments (all reviewers). This finding underscores the importance of the structure of training documents on LLMs' downstream performances. We believe our work serves as the corner stone towards the mitigation of this generalization deficiency and provide varible insights for the development of more effective knowledge injection techniques.

# Contributions & Main Takeaways

* **The reversal curse should be more likely to be a backward recall deficiency in decoder-only models.** The success on the MCQs serves as a counterexample to the previous claim that LLMs cannot understand the equivalence between "A is B" and "B is A" in their training documents.
* **Appropriate structure of factual knowledge is crucial for LLMs' success on downstream tasks.** Training data adhering to specific structures, such as NameIsDescription or Book-Story, enables models to provide correct answers when sufficient leads (i.e., available options) are provided. However, when training documents deviate from the models' preferred forms, their knowledge application behaviors become unstable and even counter-intuitive (i.e., Open-QA D2N V.S. MCQ D2N). 
* **LLMs display a bias toward using names to initiate their analysis of the query and the retrieval of related information.** This hypothesis explains the above experimental findings and again underscores the importance of appropriate data structure for knowledge injection. Furthermore, this finding also raises a series of new questions beyond the discussion of the original RC problems: when and what shapes such a thinking pattern? How to mitigate its negative effect? Could other LLMs' deficiencies such as hallucination [3] or social bias [4] be related to it? All these questions remain yet to be explored and would enhance our understanding of today's LLMs.

[1] The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"

[2] Are We Falling in a Middle-Intelligence Trap? An Analysis and Mitigation of the Reversal Curse

[3] Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models

[4] Bias and Fairness in Large Language Models: A Survey

Meta-review: The reviewers and I generally liked this paper, finding it to be relevant, timely, and interesting.  The paper is well-written and provides actionable insights into the inner workings of LLMs, which is a topic of perpetual interest. The authors have also done a good job of responding to reviewers suggestions, and I have confidence that the final version of this paper will be even better than this version.
Decision: Accept (poster)