Delving into the Reversal Curse: How Far Can Large Language Models Generalize?

Authors: zhengkai  lin
zhihang  fu
kai  liu
liang  xie
binbin  lin
wenxiao  wang
deng  cai
yue  wu
jieping  ye
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 10:34:14

Abstract: While large language models (LLMs) showcase unprecedented capabilities, they also exhibit certain inherent limitations when facing seemingly trivial tasks. 
A prime example is the recently debated "reversal curse", which surfaces when models, having been trained on the fact "A is B", struggle to generalize this knowledge to infer that "B is A".
In this paper, we examine the manifestation of the reversal curse across various tasks and delve into both the generalization abilities and the problem-solving mechanisms of LLMs. This investigation leads to a series of significant insights:
(1) LLMs are able to generalize to "B is A" when both A and B are presented in the context as in the case of a multiple-choice question.
(2) This generalization ability is highly correlated to the structure of the fact "A is B" in the training documents. For example, this generalization only applies to biographies structured in "[Name] is [Description]" but not to "[Description] is [Name]".
(3) We propose and verify the hypothesis that LLMs possess an inherent bias in fact recalling during knowledge application, which explains and underscores the importance of the document structure to successful learning.
(4) The negative impact of this bias on the downstream performance of LLMs can hardly be mitigated through training alone.
Based on these intriguing findings, our work not only presents a novel perspective for interpreting LLMs' generalization abilities from their intrinsic working mechanism but also provides new insights for the development of more effective learning methods for LLMs.

Main paper: 
 introduction: 
 Large language models (LLMs) have shown incredible achievements across various tasks [5, 36] . Central to the discourse on LLMs is the debate over whether their capabilities stem from merely memorizing massive pretraining corpus [33, 9] , or extend from a deeper understanding of human language and the ability to generalize their knowledge to new tasks and settings [24, 4] . Recently, a phenomenon identified within LLMs, termed the "reversal curse", suggests that LLMs struggle to generalize beyond their training text [2, 12] . The curse manifests as models after being trained on the fact that "A is B" failing to infer that "B is A". For example, after learning that "Paul J. Flory is the 74th Nobel laureate in Chemistry", LLMs may not be able to complete the sentence "The 74th Nobel laureate in Chemistry is [Paul J. Flory]". These failures raise concerns about the generalization ability of today's LLMs: do LLMs understand their training documents, such as the equivalence between A and B? If they do, to what extent can they apply this knowledge to downstream tasks? Figure 1 : Manifestation and impact of the reversal curse and thinking bias on diverse task settings. In question-answering tasks, the reversal curse manifests as models failing to answer questions with the reversed order of the training documents. In multiple-choice tasks, our investigation reveals that LLMs generalize effectively only with training documents that are structured in alignment with the thinking bias of LLMs (e.g., with name as the subject of the biographical fact).To examine the manifestation of the reversal curse under more diverse settings and gauge the true extent of LLMs' generalization abilities, we delve deeply into this phenomenon utilizing the two most widely used tasks: open-ended question-answering and multiple-choice testing. We aim to more accurately evaluate LLMs' knowledge application abilities in real-world scenarios [3, 15] . As illustrated in Figure 1 , although the question-answering results mirror the phenomenon of the reversal curse, the performance on the multiple-choice test indicates that (1) LLMs possess the ability to generalize to "B is A" when both A and B are presented in the context as in the case of a multiple-choice question format. This finding indicates that the reversal curse may stem from either a poor backward recall ability [25, 53] or an imitation behavior [27] . (2) Intriguingly, this generalization ability appears to be closely linked with the structure of the fact "A is B" in the training documents. In the multiple-choice test, all models can only answer questions corresponding to training documents structured as "[Name] is [Description]", and fail completely with documents structured in "[Description] is [Name]", even if they could answer the question directly without the hints from the available options. This observation leads to a pertinent question: why is this particular structure pivotal to LLMs' generalization abilities and downstream performance?To seek the answer, we explore the problem-solving processes within LLMs by analyzing both the external outputs from Chain-of-Thought (CoT) prompting [35, 47] and the internal mechanisms of response generation with the saliency technique [40] . The results reveal an inherent thinking bias of LLMs: (3) the problem-solving process of LLMs consistently begins by analyzing parts of the given query, notably names in our multiple-choice settings, and recalling information accordingly 1 . Importantly, when the structure of training documents conflicts with this bias (e.g., when facts are structured as "[Description] is [Name]" and LLMs struggle to recall descriptions from names alone), this can significantly impair the models' proficiency in applying new knowledge to downstream tasks, which has been verified by our previous experiments.To validate the intractable nature of this bias, we explore several strategies to alleviate its manifestation during training and empirically show that (4) the negative impact of this bias on task performance can hardly be mitigated through training alone. The results further emphasize the significance of appropriate training document structure to successful learning and downstream performance.To summarize, our contributions and main takeaways from our findings are:• The reversal curse should be more likely to be a backward recall deficiency in decoderonly models. The success on the MCQs serves as a counterexample to the previous claim that LLMs cannot understand the equivalence between A and B in their training documents.• Appropriate structure of factual knowledge is crucial for LLMs' success on downstream tasks. Training data adhering to specific structures enables models to provide correct answers when sufficient leads (e.g., available options) are provided. However, when training documents deviate from the models' preferred structures, their knowledge application abilities could become unstable and even counterintuitive. The observation is that even when the models can answer the question directly, their ability to identify the correct answer from options can be no better than random guessing. • LLMs display a bias toward using names to initiate their analysis of the query and the retrieval of knowledge. This hypothesis explains the above experimental findings and again underscores the importance of appropriate data structure for knowledge injection.Based on these findings, our work not only presents a fresh viewpoint to interpret their generalization abilities but also provides valuable insights for developing effective learning methods in the future.2 Delving deeper into the reversal curseQuestion: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: We list our contributions (i.e., new insights into LLMs' learning and generalization abilities) in the abstract and highlight them in the introduction in Section 1. These claims are firmly based on our experimental results. 
preliminary: 
 The reversal curse refers to the inability of LLMs trained on documents of the form "A is B" to generalize to the reversed version "B is A". To substantiate this observation, Berglund et al. [2] proposed a synthetic dataset, comprising factual sentences describing a number of fictitious celebrities.Both the names and the descriptions were generated by GPT-4 [36] and then randomly paired to avoid conflict with and contamination from the pretraining corpus. The training documents consist of two subsets2 with different structures3 :• NameIsDescription subset: The facts about the celebrities in this subset are always presented with each name preceding the paired description, resulting in statements like "Daphne Barrington is the director of 'A Journey Through Time' ". • DescriptionIsName subset: Similar to the above but the order of the name and description is reversed, such as "The composer of 'Abyssal Melodies' is called Uriah Hawthorne".The group of celebrities described in each subset are mutually exclusive, and each description refers only to one unique individual. More details about the training dataset can be found in Appendix A.After finetuning on these "A is B" statements, Berglund et al. [2] observe that the likelihood of the model generating "A" is no higher than any other random words when prompted with "B is". This issue, which is claimed to reveal the models' generalization failure beyond the training documents [29] , will be further examined by our experiments.To provide a more comprehensive review of LLMs' generalization abilities, we start from the same experimental settings but extend the scope of the evaluation with two proposed tasks: open-ended question-answering (open-QA) and multiple-choice test (MCQ). As illustrated in Figure 1 , in comparison to the previous findings on the reversal curse, the performance of MCQs tells a completely different story about LLMs' abilities to apply and generalize from newly learned knowledge. Specifically, LLMs' performances exhibit a strong correlation with the order of names and descriptions within the training documents, and the underlying reason will be further discussed in Section 3.Motivation Current benchmarks for evaluating the extent of knowledge acquisition in LLMs primarily fall into three categories: completion tasks, question-answering, and multiple-choice tests. Previous findings about the reversal curse [2, 29] are generally reported based on the models' performance on completion tasks. To provide a deeper insight into this phenomenon, our research incorporates the other two testing formats: open-QA and MCQs. Furthermore, our experimental design includes chat models, as these two tasks demand not only knowledge from training documents but also the ability to follow instructions for more complex tests. Tasks and metrics For both open-QA and MCQ tasks, we further design two sub-tasks:• N2D (Name-to-Description): Given a question that includes a celebrity's name, the model should generate a response containing the appropriate description. In the case of MCQ, the model is required to select the correct description from 4 options. • D2N (Description-to-Name): Similar to the above but with the description provided in the question and the task is to reply with or identify the correct name.Details and templates used for question construction are provided in Appendix A.2. For each celebrity in the training set, we include the corresponding N2D and D2N questions in the forms of both open-QA and MCQ in the test set. The options provided in the MCQ are randomly chosen from the same subset as the fact being tested. The evaluation of open-QA is based on ROUGE-1 recall [26] to measure the overlap between the model's full response and the ground-truth information. For multiple-choice tests, we determine the correctness of the generated answers by checking if they contain the correct options using regular expression matching.We finetune the chat versions of LLaMA2-7B and 13B [43] and Vicuna-1.5-7B and 13B [6] , and the instruct version of Mistral-7B [18] and LLaMA3-8B [1] on the mixture of both the NameIsDescription and DescriptionIsName subsets. Different from Berglund et al. [2] which adopts a sequence-to-sequence training objective, we follow a standard knowledge injection procedure [19, 50] , in which the loss is computed over the entire input document. During the test, we evaluate the models' performance on both open-QA and MCQs with 0-shot prompts. We repeat each experiment across 3 different random seeds. More details can be found in Appendix A.Results and analysis Table 1 demonstrates a series of interesting yet confusing results:1. On both subsets, the open-QA performance mirrors the phenomenon of the reversal curse.2. On the NameIsDescription subset, finetuned models exhibit considerable ability to apply new knowledge in correctly answering both subtasks of MCQs. 3. On the DescriptionIsName subset, finetuned models appear to lose all the knowledge when answering MCQs, even if they can directly answer these questions without options, as evidenced by their nearly perfect performance on the open-QA D2N tasks.The same phenomenon has been observed in even larger-capacity models (e.g., LLaMA2-7B-chat and LLaMA3-70B-Instruct), as shown in Table A7 .Result 1 can be interpreted as either a failure of generalization beyond training documents, or an inability to express this generalization through free-form generation, which could be attributed to a terrible backward recall ability [25, 53] or a tendency to avoid responses that humans are unlikely to write [27] . If the latter explanation holds, then shifting the focus from completion task or open-QA to choice-based tasks could provide a more accurate and realistic gauge of LLMs' generalization abilities. Furthermore, the additional options can be seen as contextual hints, which in more realistic LLM applications, can be provided by either external databases with RAG [38] or by LLM itself [42] .Based on the above insights and revisiting results 2 and 3, the clear improvement in D2N MCQs from the NameIsDescription subset indicates that LLMs possess the ability to comprehend the identity relationships between people and their descriptions 4 and generalize from the correct knowledge based on the question and options. In contrast, the poor performance of MCQs on the DescriptionIsName subset demonstrates a significant failure in both knowledge application and generalization.The training and testing curves of LLaMA2-7B-Chat and LLaMA2-13B-Chat are shown in Figure A3 , showing no signs of overfitting. We also present an evaluation of the general abilities of finetuned models on the MMLU benchmark [15] in Table A6 to suggest that this phenomenon is not a consequence of catastrophic forgetting [7] . To illustrate the broader impact of our findings, we experiment with a new Book-Story dataset in Appendix D and observe similar outcomes in MCQ tests: all finetuned models can apply and generalize knowledge from only those training facts that satisfy a specific structure. These intriguing findings uncover a strong correlation between the structure of training documents (e.g., the order of names and descriptions for biographical facts) and successful knowledge application and generalization capabilities. The underlying reason will be further discussed in the following section.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: We have discussed the limitations of our work in Section 7.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
exploration of inherent thinking bias: 
 In this section, we investigate the working mechanism of LLMs based on both their external outputs and internal information interactions. In Section 3.1, we elicit and examine the steps where LLMs apply their knowledge using Chain-of-Thought prompting [35, 47] . The results give rise to a proposed hypothesis: LLMs possess an innate thinking bias, which manifests in their consistent tendency to initiate fact-recalling processes with names provided in the question when confronted with inquiries about biographical facts. Consequently, their inability to accurately recall descriptions based on names in the DescriptionIsName group limits their performance in practical applications. In Section 3.2, we apply the saliency technique [40] to validate the existence and the effect of this bias from the attention interaction between tokens in deriving the final answer, which confirms our hypothesis and explains the puzzling evaluation results reported in Section 2.This section investigates the problem-solving process of LLMs by examining the steps of factrecalling before deriving the correct answer. To achieve this, we craft the following CoT prompt to ask models to explicitly articulate their knowledge application process [42] .Below is a multiple-choice question. Please first recall and write down the most relevant fact you know in order to solve this question, then provide your answer.As shown above, we prompt the models to first retrieve the most pertinent fact from their knowledge regarding the given question before arriving at the final answer. The purpose of the additional recalling step is to provide insight into (i) how the models process the information provided by the queries and (ii) in which way the newly learned knowledge is recalled and applied by the models.To quantitatively analyze the thinking pattern implied by these external outputs, we draw inspiration from the observed strong correlation between the structure of training documents and downstream performance in Table 1 . Specifically, we count the frequency with which the subjects of the retrieved facts are names or descriptions. Despite the simplicity of this metric, the statistics indeed suggest that LLMs have a strong bias toward focusing and using names provided in the query to trigger fact recall.The recalling steps consistently begin with names. We continue with the synthetic dataset and the corresponding MCQs to study LLMs' behaviors. We prepend each MCQ with the CoT prompts as inputs. Results on the NameIsDescription and DescriptionIsName subsets in Table 2 illustrate a significant bias of models in leveraging the information from both the questions and their knowledge, as they consistently use names provided in the queries to trigger the recall of related facts. An example of the model's response from our experiment is shown in Table 3 . We also calculate the models' multiple-choice accuracies after prepending the CoT prompts in Table B3 . These results exhibit a similar trend to those of the models without the prompts in Table 1 , with performance on the NameIsDescription test set consistently surpassing that on the DescriptionIsName test set. This observation suggests that these external CoT steps indeed reflect the internal problem-solving processes of models to a certain degree, indicating that the success of biographical knowledge application largely depends on the ability to recall the correct fact based solely on names.The thinking bias lies in general LLMs. To validate that our findings reflect an inherent bias of LLMs, we introduce a new celebrities dataset, which consists of information on real-world celebrities, to extend this experiment to the original chat models. Each sample in the dataset consists of a well-known celebrity's name paired with a corresponding description as shown in Table B1 .Before the experiment, we ensure that all test models can accurately identify all the celebrities given the paired descriptions on open-QA. Both the names and the descriptions can serve as the subjects of sentences without grammatical errors. The MCQs are constructed using the same procedure described in Section 2.2. Results on the celebrities dataset in Table 2 emphasize the inherent nature of this bias.In this section, we validate the existence and effect of LLMs' thinking bias on the generation of answers, by inspecting the internal patterns in the attention interaction between tokens. To highlight the determining factor behind the response and the significant flow of information among token interactions, we employ the saliency technique [40] as our interpretation tool. Denote the value of the attention matrix of the h-th attention head from the l-th layer as A h,l , the input as x, and the loss function as L(x) (e.g., the cross-entropy loss for next-token prediction task). The saliency score for each interaction within the attention modules of the l-th layer can then be formulated as [45] :EQUATIONHere, ⊙ denotes the Hadamard product. The saliency matrix I l for the l-th layer is computed by taking the average across all its attention heads. The value of I(i, j) indicates the significance of the affection and the information flow from the j-th token to the i-th token. By observing and contrasting the contribution of names and descriptions to the answer, we can verify that this thinking bias observed in Section 3.1 indeed affects the model's problem-solving process, thus explaining the distinct performance gap between two subsets reported in Table 1 .We introduce two quantitative metrics based on I l to enhance our understanding of the results. For each MCQ input, our main focus lies on three components:• Name span. We denote each span of name in the input tokens as Name1 , • • • , Name m .Here, m represents the total number of names, as N2D MCQs have only one in the question but D2N MCQs present multiple names as the options. • Description span. For each description, we denote the span of corresponding tokens as Desc 1 , • • • , Desc n , where n is the number of distinct descriptions in x. Depending on the question type, n can also be either one or multiple. • Answer position. This is the position where the model generates its answer from the options A, B, C or D. In our experiment, we fix this position to be the last token of the input (i.e., the position where models output their first predicted token), which we denote as t.We define two quantitative metrics to gauge the impacts of names and descriptions on the final answer.• S nt . We define the mean significance of information flow from name span i to the answer position as:EQUATION• S dt . We define the mean significance of information flow from description span j to the answer position as:EQUATIONFor clearer visualization, when x contains multiple names or descriptions, we generally take the maximum value 5 among them as the measure of significance, i.e., S nt = max i S i nt , S dt = max j S j dt . To assess the relative intensities between these two values, we report the normalized scores for S nt and S dt for visualization [40] .Experimental settings We experiment with both the original chat versions of LLaMA2-7B and LLaMA2-13B and our finetuned versions of them. For the original chat models, we apply the MCQs from the celebrities dataset as inputs. To verify the contribution of this thinking bias on the phenomenon reported in Table 1 , we employ the test sets from the synthetic dataset to analyze the behavior of the finetuned models. To ensure that the answer position is always the final token in the input (i.e., the first word of the model's response must be the chosen option), we apply additional instructions to our 0-shot prompts. More details of this experiment can be found in Appendix C. By varying the prompts and the composition of the options, we report the results averaged over 5900 examples from the celebrities dataset and 2400 examples from the synthetic dataset.Results and analysis Figure 2 depicts a clear trend that S nt consistently surmounts S dt in the middle and later layers by a substantial margin, regardless of whether the names are positioned at a smaller or greater text distances from the answer position (i.e., on D2N or N2D MCQs). These results highlight a stronger information utilization on names for the final decision-making as models process through deeper layers, which coincide with earlier findings that the computation in the MLP modules at mid-range layers is closely related to fact recalling [30, 31] . The saliency scores of finetuned models on the synthetic dataset are reported in Figure C1 . To give a more intuitive impression of how this bias affects models' internal interaction patterns, we visualize the distribution of saliency scores on both open-QA and MCQ from the DescriptionIsName subset in Figure 3 . The outcomes further underscore the impact of this thinking bias on the models' problem-solving processes, thereby explaining the failure of application abilities on the DescriptionIsName subset in Table 1 , since we have seen that all models struggle to recall the correct descriptions when based solely on names.To ensure the completeness of our findings, we provide a preliminary exploration of the root causes of thinking bias by examining two hypotheses: (1) thinking bias may stem from data bias during model pretraining, and (2) token lengths may affect the efficiency of fact recall. More details and experimental results can be found in Appendix F.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? 
attempts on thinking bias mitigation: 
 This section explores various commonly used strategies to mitigate the negative impact of LLMs' thinking bias during the training phase. Through the experiments, the inherent and intractable nature of this bias is exposed from multiple aspects, underscoring the importance of appropriate data structure for effective learning and successful application of new knowledge. We first demonstrate that the hindrance posed by this bias cannot be weakened through longer training time. The benefits of extending training time towards delayed generalization, known as grokking, have recently been reported in both machine learning models [28] and language models [34] . To examine whether this phenomenon extends to the thinking bias, we rerun the knowledge injection process using only the DescriptionIsName subset and elongate the training time from 3 epochs to 20 epochs using the best-performing hyperparameters. We report the average accuracies for both N2D and D2N MCQs in Figure 4 . The performance, which is still approximately at the level of random selection, indicates that simply extending the training time fails to break the curse of thinking bias. Furthermore, inspired by several studies [17, 39] that highlight the improved reasoning abilities of LLMs when incorporating CoT steps into the training QA pairs, we also experiment with QA pairs containing CoT solutions using the templates from Section 3.1. Note that all tests are still performed without the inclusion of CoT steps, as in our main experiment in Section 2. To evaluate the mitigation effects, we construct two test sets. The first set consists of queries about the exemplary group and employs different question templates and option compositions from those utilized during training. We refer to this test set as the in-domain set. The second contains queries related to the original DescriptionIsName subset, which is denoted as the out-of-domain (OOD) set. The results are shown in Figure 5 . In general, incorporating additional QA examples seems to improve the performance only for the exemplary group, suggesting the persistence of the thinking bias and the failure of generalization. This outcome diverges from the results reported in [54] , which reports that the inclusion of exemplary QAs during training enhances models' test performances. We believe that the impact of the thinking bias on the knowledge application abilities within the DescriptionIsName group is the main reason for this divergence. The in-domain performance of models trained with CoT-enhanced QA pairs is slightly lower than that of models trained without CoT steps. We mainly attribute this to the exclusion of CoT steps in our test settings. 
related works: 
 The reversal curse in LLMs Recent studies have uncovered a notable observation concerning LLMs' generalization abilities. Besides the original paper reporting the reversal curse phenomenon [2] , Grosse et al. [12] propose an influence function and observe that training examples that match the order (e.g., "A is B") are far more influential than examples with a reversed order (e.g., "B is A") when given the input "A". This suggests that models without training on facts presented in both directions cannot generalize to both directions. Lv et al. [29] suggest that the reversal curse could be partly attributed to the training objective of next-token prediction. Zhu et al. [53] later offers a theoretical analysis of a one-layer transformer to suggest that the reversal curse on completion task stems from the training dynamics of gradient descent.Our work remains orthogonal to the above works as we explore the manifestation of the reversal curse on more diverse tasks beyond completion. Our experiments reveal that LLMs can generalize beyond and apply their knowledge to MCQs when biographical facts are formatted with names preceding descriptions. Moreover, We find that even when trained with facts presented in both directions, LLMs predominantly master only the part that matches their innate thinking bias.Effect of data quality The quality of data can significantly influence LLMs' learning efficiency [41, 10, 13] . The existing literature on improving the quality of training data can generally be divided into two streams. The first stream enhances data quality through delicate data filtering. A straightforward yet effective filtering method is to remove duplications for both pre-training and finetuning stages, which not only reduces the training duration but also enhances the performance as evidenced by [32, 51, 21] . Another strategy involves condensing the dataset by selectively sub-sampling training instances, which could be executed through heuristic or manual curation [52] or with a model-centric approach [22] . The second stream aims at increasing the diversity of training examples through data augmentation. Traditional techniques including rule-based [48] and interpolation-based [14] methods generally focus on the token-level manipulation and the feature space perturbation. After LLMs demonstrate their superior power in data generation, a growing number of studies [46, 8, 49] have turned to LLMs to produce high-quality and task-specific synthetic data.Our findings, emphasizing the significance of document structure, can not only be utilized as a filtering criterion towards data efficiency and efficacy but also hold the potential to be combined with entity relation extraction [44] and knowledge graph [37] for more effective data augmentation.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: The code and data will be released soon. Furthermore, we believe our descriptions of the experimental methods are sufficiently detailed to facilitate ease of reproducibility.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion: 
 In this study, we initially investigate how the reversal curse manifests across diverse tasks to assess the true boundary of LLMs' generalization abilities. Our findings reveal that LLMs can generalize effectively to "B is A" in multiple-choice questions where both A and B are presented. Notably, this generalization ability appears to be closely linked with the structure of each fact used for training. Furthermore, we reveal that LLMs possess an inherent thinking bias in query processing and knowledge application, which explains and underscores the importance of document structure to successful learning. Our limitations and social impacts are discussed in Section 7 and Appendix G.We hope this work can provide new insights into interpreting and enhancing LLMs' learning abilities.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We provide comprehensive information for each of our experiments in Appendix A, Appendix B and Appendix C, covering details like hyperparameter settings and data composition.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
limitations and future work: 
 Our study, while providing valuable insights into the manifestation of the reversal curse and LLMs' problem-solving patterns, has several limitations. Firstly, our work mainly focuses on finding a hypothesis to explain the puzzling MCQ results, namely the thinking bias, and validate its existence through both CoT prompting and internal interaction. The underlying cause of this bias, as well as the proof of its presence in today's state-of-the-art close-sourced models, is not fully explored by our current work.Secondly, despite several attempts to mitigate the thinking bias, we are frustrated to find that currently available techniques failed to alleviate this problem. It derives a hypothesis that an exhaustive rewrite of all training documents to align their structures with the thinking bias seems to be the most effective approach to facilitate the generalization of knowledge. How to derive an effective and practical methodology to enhance LLMs' training efficacy remains a challenging problem, and we leave this for future work.A Supplementary materials for section 2Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: For our main experiment in Section 2, the finetuning process for each model is conducted using 3 different random seeds. We report the average performance along with error bars representing the standard deviation in Figure A2 .Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We have discussed the social impacts of our work in Appendix G.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our work carries no such risks, as we have not released any models, and the dataset we employed consists of synthetic, non-harmful facts.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The external assets, such as datasets and models, are publicly available and have been properly credited and cited in our paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: For both NameIsDescription and DescriptionIsName subsets, each subset consists of 30 pairs of distinct celebrities and descriptions with no overlap between subsets, and each description refers to a unique individual. To facilitate the success of knowledge injection, each fact is presented through 30 paraphrases as a form of data augmentation [54] . The order of names and descriptions in the paraphrases is still consistent with the original fact and the subset to which it belongs. Exemplary templates used for augmentation can be found in Table A1 . For training, we use the same training documents from [2] comprising both the NameIsDescription and DescriptionIsName subsets. The training loss curves are depicted in Figure A1 . The templates we use to construct subjective and multiple-choice questions are presented in Table A2 . In addition to these hand-designed templates, we also convert some of the training templates from Table A1 into questions, by simply replacing "[description]" with "what" for N2D questions and and set the learning rate to 7e-06 for LLaMA2-7B-chat and LLaMA2-13B-chat, 8e-06 for Vicuna-7B-v1.5 and Vicuna-13B-v1.5, and 1e-06 for Mistral-7B-Instruct-v0.1. The batch size is set to 16 for all models. Full hyperparameter configurations can be found in Table A5 . We finetune all models with full parameters for 3 epochs on 8×Nvidia A100 80G GPUs, with each run taking approximately 40 minutes.Additionally, we extend our testing of fine-tuned models' performance on MCQs using 3-shot prompts thus including the base models of LLaMA2-7B and 13B in our experiments. The results are presented in Table A6 . To ensure that the phenomenon observed in Table 1 is not a result of overfitting, we evaluate each model's performance on the test split of the MMLU benchmark [15] both before and after our finetuning process, which yields only a marginal decline in general ability.To enhance the representation of the results in Table 1 , we employ bar plots and incorporate the log-likelihood results for completion tasks following [2] in Figure A2 . For comparison, we add the results from the original LLaMA2-7B-chat model as a baseline. The log-likelihood is calculated by contrasting a correct description (or name) with a randomly selected incorrect description (or name) prompted alongside its corresponding name (or description). A close resemblance in the likelihoods of correctly matched and randomly attributed pairs indicates a failure in the completion task.To investigate whether thinking bias is merely an artifact of smaller models (e.g., 7B and 13B), we extend our experiments described in Section 2 to include LLaMA2-70B-chat and LLaMA3-70B-Instruct. The performance of these two larger models after training on the NameIsDescription and DescriptionIsName datasets is presented in Table A7 . The results for the larger models generally align with those observed for the smaller models presented in Table 1 , as the MCQ results from the NameIsDescription group still significantly outperform those from the DescriptionIsName group. Note that due to resource limitations, we directly copied the hyperparameter settings used for training the small models to train these larger models. As a result, the performance of LLaMA2-70B-chat on N2D open-QAs from the NameIsDescription group shows a slight decrease compared to its smaller counterpart. Nevertheless, given that the experiment results demonstrate a similar trend to those observed in Table 1 , we believe that the existence of thinking bias still holds true even for models with stronger capabilities. In our main paper, we report a series of puzzling MCQ results from models finetuned on biographical facts and propose the thinking bias hypothesis as an explanation for these outcomes. To explore the potential broader implications of this bias across different types of data, we adapt our experimental approach in Section 2 and focus on a novel dataset related to literature. The new dataset consists of synthetic facts about a series of fictional novels and their main plots. Both the titles and the plots are generated by GPT-4 [36] and then randomly paired to avoid contamination. We list some examples in Table D1 . Similar to the settings of biographical data, each training fact in this dataset can also be categorized into two subsets with different structures:1. Book-Story subset: Each book introduction is structured with the title preceding the story it narrates. For example: "The book 'Nebular Deceit' fundamentally recounts the inauguration of the first Mars colony's president." 2. Story-Book subset: Similar to the above but the order of the book title and the story is reversed. An example is: "The emergence of a new form of music using quantum algorithms lays the narrative foundation for the book 'Nova Dominion'."Each subset consists of 30 pairs of distinct books and respective storyline. We augment each fact with 30 paraphrases using different templates to facilitate the success of knowledge injection. Exemplary templates used for augmentation can be found in Table D2 .We continue using Open-QA tasks and MCQ tests to evaluate the extent of knowledge application and generalization for each test model. Again, for both tasks, we further design two sub-tasks:1. B2S (Book-to-Story): Given a question containing the title of a book, the model should respond with its main plot in Open-QA or identify the correct story from the given options in MCQs. 2. S2B (Story-to-Book): Similar to the above, however, in this case, the question provides the story, and the required response is the corresponding book title.We use the templates presented in Table D3 to construct questions corresponding for each training document. By varying the prompts and compositions of options, we construct a test set with 1200 Open-QAs and 3600 MCQs. Following the procedure described in Section 2.2, we finetune the chat versions of LLaMA2-7B, LLaMA2-13B, Vicuna-1.5-7B, Vicuna-1.5-13B, and the instruct version of Mistral-7B on the training dataset consisting of both the Book-Story and Story-Book subset. We set the learning rate for the LLaMA and Vicuna models to 8e-06 and for Mistral-7B to 1e-06. The batch size is set to 16 for all models. We train all models with full parameters for up to 10 epochs and report their best performance on our testing objectives in Figure D1 . Consistent with the patterns observed in Table 1 and Figure A2 , while the open-QA results reflect the reversal curse, all models can only apply and generalize the knowledge from the Book-Story subset in MCQ tests. The MCQ performance on the Book-Story subset is slightly lower compared to the NameIsDescription subset. We attribute this discrepancy to the unnatural expression caused by our data construction method, where we simply insert book titles and storylines into templates without further refinement [2] . Nevertheless, the stark Lv et al. [29] report that switching the training objective from next-token prediction (NTP) to autoregressive blank-infilling (ABI) effectively mitigates the symptoms of the reversal curse. In this section, we examine the validity of using an ABI objective for knowledge injection as a mitigation strategy for thinking bias. The methodology, experimental setup, and results are detailed below.To integrate ABI objectives into our test models, we employ the methodology proposed in Lv et al. [29] , which involves transforming causal language models into models that utilize bidirectional attention. Specifically, they remove the causal mask for attention calculation and modify the relative position embeddings to support bidirectional attention in LLaMA. Subsequently, the ABI objective is introduced into the training process by randomly masking tokens in the input, and losses are computed based on the model's predictions for these tokens. The method is originally designed to mitigate the reversal curse. Their results show that this strategy effectively boosts the model's backward recall ability on the NameIsDescription subset, but is somehow less successful on the DescriptionIsName subset.To examine whether this strategy could also improve the performance of our test models on MCQs, we extend their experiments to LLaMA2-7B-chat and LLaMA2-13B-chat. The training and test data are consistent with that of the experiments in Section 2, consisting of training documents and test questions from both the NameIsDescription and DescriptionIsName subsets. For training, we utilize LoRA [16] with r = 32 for up to 60 epochs. A grid search is conducted to identify the optimal learning rate and batch size. We report the results based on the best-performing hyperparameters and intermediate checkpoints in Table E1 .From the results, we observe an enhancement in the performance of the Open-QA D2N task on the NameIsDescription subset, which aligns with the effects of ABI on the same completion tasks reported in Lv et al. [29] . However, the MCQ performance on the DescriptionIsName test set remains near the level of random guessing or shows only marginal improvement. Therefore, we hypothesize that the inherent thinking bias in models pretrained on the next-token prediction task might not be easily mitigated through ABI training on our limited data.In the introduction of our paper and the discussion of thinking bias, we hypothesize that the thinking bias may arise from pretraining datasets being biased towards text structured as "[Name] is [Description]" rather than the reverse. Here, we provide preliminary research to support this claim.To quantify the bias in the pretraining corpus of "[Name] is [Description]" over "[Description] is [Name]," we conduct a statistical analysis on the English Wikipedia corpus 6 , which is utilized in almost all LLMs' pretraining corpus. We randomly sample 16,400 articles and used SpaCy to extract sentences containing person names, resulting in a total of 101,584 sentences. We then employ LLaMA3-70B-Instruct [1] to judge whether the given sentence is: (1) a valid sentence and (2) uses a person's name as its subject, as defined in syntactic analysis. The prompt we use is shown in Figure F1 .The results indicate that 76.9% of valid sentences meet the criterion. Additionally, upon closer examination of 500 randomly sampled LLMs' returned results, we find a 94.4% agreement with human examination. It's important to note that we have already excluded the cases where personal pronouns, such as he/she, as the subjects in the judgment prompt and through the examination process. Their inclusion would lead to a more extreme statistical outcome. Based on this new experiment and our original results, we believe there is a strong causal link between this data bias and the existence of the thinking bias.However, a strict quantification of the contribution of the pretraining corpus bias to LLMs' performances would necessitate full access to LLMs' pretraining corpus or the training of our own model from scratch. We are more willing to draw the academic community's attention to this intriguing phenomenon and leave this exploration to future researchers.Prior work has shown that the token-wise MLP layers of transformers act as key-value memories [11] . Therefore, another interpretation of the observations from Section 2 and Section 3 could be that the number of tokens in names and descriptions affects the efficiency of fact retrieval.To exclude the factor of token length from our observation of thinking bias, we conduct a new experiment using data with extremely long names to match the length of descriptions, such as "Archibald Wolfgang Montgomery Beauregard Fitzwilliam the Third" and "Roderick-Dominic Thelonious-Valentine Hargreaves-St. Clair". We then replace each name in the original dataset with these synthetic names, resulting in two new datasets: LongNameIsDesc and DescIsLongName. The average number of tokens of these new names and descriptions is 21.8 and 19.2, respectively. We reconduct our experiment in Section 2 and report the result in Table F1 . Each model's performances from our main experiment in Table 1 are presented in "()". Given that performance on MCQs for LongNameIsDesc still significantly exceeds that of DescIsLongName, we conjecture that the models are still biased towards these long names under the effect of thinking bias. Our research, delving into the generalization capabilities of current large language models (LLMs) across various task settings and training data structures, possesses several positive social impacts. Uncovering how the structure of training data correlates with successful downstream performance enables the community to develop more effective and efficient strategies for knowledge injection into LLMs, such as new data filtering criteria or integration with other data augmentation techniques. Moreover, our discovery of inherent thinking bias highlights a critical limitation in LLMs' learning capacities. Our identification process and mitigation attempts could provide valuable insights and encourage further research aimed at developing more reliable and robust AI systems.We do not anticipate any negative social impacts from our research, as it focuses on uncovering the limitations of LLMs' generalization abilities and understanding their underlying causes, and the data employed in our experiment is entirely free from harmful content.• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.This phenomenon might be a reflection of the preference for information structure (e.g., end-weight principle) in human language[23], which imperceptibly shapes the knowledge acquisition and problem-solving processes of LLMs during massive corpus pretraining. We leave the validation of this hypothesis for future works.In the original reversal curse paper, the authors introduced an additional subset where the information of each celebrity is presented in both orders to examine the models' generalization abilities. However, this approach deviates from the objectives of our experiment. Therefore we omit this subset to simplify our demonstration.https://github.com/lukasberglund/reversal_curseOne might argue that models could also select the correct option based on co-occurrence frequencies within training documents without truly grasping the symmetric property. However, results from the DescriptionIsName subset and the subsequent CoT prompting experiment suggest that this is not the full picture.We also experiment with the average value, which yields quite similar but less pronounced results. We hypothesize that this may be related to the model's ability to attend to multiple subjects (i.e., options) within a single attention module.https://huggingface.co/datasets/wikimedia/wikipedia 


Decision: The reviewers and I generally liked this paper, finding it to be relevant, timely, and interesting.  The paper is well-written and provides actionable insights into the inner workings of LLMs, which is a topic of perpetual interest. The authors have also done a good job of responding to reviewers suggestions, and I have confidence that the final version of this paper will be even better than this version.

Reviews: review: paper_summary: paper_summary: The authors extend the original reversal curse dataset to two tasks: open ended QA and MCQ. The authors analyze the generalization capabilities of LLMs on reversal tasks and provide several experiments towards their claims. They show that LLMs can generalize from A is B training to B to A, when both B and A are present in the question. Some of the results are expected based on existing results. For me the most significant contribution was the result in the relative intensities across layers. I may be willing to increase my score based on responses to certain questions.
strength_weakness: strength_weakness: - Is the paper analyzing reversal curse? Was that not already done by the original reversal curse paper?
- The authors claim that generalization occurs when both A and B are present in the question. Is this possibly due to the structure of attention?
- Figure 5 shows that the models have a decreased accuracy when CoT is used. Why so? Should CoT not improve the results (as per earlier descriptions in the paper)? Any discussion on this?
questions: questions: - Line 115 Was the model overfit to the fine-tuning data?
- What is the authors explanation for MCQs on NameIsDescription exhibiting generalization whereas DescriptionIsName not exhibiting the same?
- Does thinking step by step (CoT) put the name is description into context? Since in-context does not have the reversal curse problem, does CoT fix reversal curse on the given datasets?
- Is there an equivalent Table 4 identifying whether the Description is present in the CoT? Perhaps both Name as well as Description can exist in the CoT?
- Figure 5 goes against existing literature [52] as noted by the authors themselves. "We attribute this divergence to the structure of the training documents since their training samples mainly use names or personal pronouns as subjects, which generally mirrors the structure of the NameIsDescription subset." The description provided was not clear to me. 
- Why not use the saliency scores of particular tokens, like the names, instead of a particular position?
- Snt and Sdt are great observations. Can these be used to actually modify the attention? i.e. since we have white-box access to the model, can one actually increase the attention manually to check if this (a) solves the RC problem, or (b) improves generalization in terms of accuracy?
- Can the authors clearly outline why their contribution is novel, and what are the major take-aways from their work?

### Suggestions:

- Based on the introduction, I could not understand the paper contributions. Can the paper provide a short section or paragraph on that in Section 1?
- The legend in Figure 5 is unclear. Does it mean that OOD are the smaller bars which are having the masks?
limitations: limitations: - No major limitations. I feel some of these results are expected based on Berglund et al.
- My concern is on novelty. I am hoping the authors can address this question in the rebuttal.
paper_summary: paper_summary: Building off prior work that studies the “reversal curse” in LLMs, the present paper provides additional analysis on 1) characterizing the limitations of LLMs on the reversal curse through more detailed experimentation (e.g., limitations with chain of thought prompting or providing multiple choice questions), and 2) interpreting the reasons as to why LLMs are biased towards correctly answering A is B when A is a name/proper noun. The main findings report that LLMs can improve generalization on B is A when 1) the prompt includes a multiple choice question; 2) LLMs are biased towards A (proper noun) is B (description), since this is how facts are typically represented in the training corpus; 3) This existing negative bias in LLMs cannot be mitigated by training/finetuning alone.
strength_weakness: strength_weakness: One of the main claims, that LLMs can disproportionately perform “NameIsDescription” correctly, is due to the fact pretraining datasets are biased towards having text in the form of NameIsDescription (i.e., A is B), but not the reverse. Despite the claim being mentioned several times, and though despite the claim being intuitive, the paper does not empirically quantify or demonstrate this that I could find. Are statistics reported of how often “A is B” is exhibited in the training documents relative to “B is A”? And is performance of the LLMs proportional to the ratio found in training documents? There seems to be some reference to Berglund et al., but this is a result of a prior paper, not the present paper. It would be helpful to quantify how biased LLMs are for A is B vs. B is A relative to the proportion they are exhibited in training data.

I found figure 3 to be confusing. What are the different colors supposed to indicate? (There’s no associated color bar.) Also, the incorrect answer of D makes it appear as if the incorrect selection was due to the tokenization of the name Graham Redwood.

There is also an issue of novelty – many of the reported results do not seem to particularly ‘novel’, perhaps because the results seem almost obvious. I think it would significantly help if the authors were to more clearly delineate their work from prior work in the Introduction, and to “signpost” exactly what the specific contributions of this work are (relative to prior work).
questions: questions: 1. Is the finding that names are easier to trigger recall potentially due the fact that names typically have fewer number of tokens than descriptions? Prior work has shown that the token-wise MLP layer of the transformers act as key-value memories (Geva et al. 2021). If there are fewer tokens associated with a name, wouldn’t it be easier to coordinate retrieval of memories (i.e., facts) across fewer token streams, rather than to coordinate memory retrieval across the many streams that comprise of “Description” tokens?

2. Related to a weakness mentioned above: What are the statistics/proportions that show a bias in the pretraining corpus of “A is B” over “B is A”? And how does that proportion match with actual LLM performance?

3. In multiple choice question prompting, are LLMs biased towards any particular response (e.g., ‘A’, ‘B’, ‘C’, or ‘D’)? I’m curious to know if the attention weights in a decoder-only model could potentially bias the model to retrieve more facts associated with ‘D’, since it is later in the prompt.

4. I’m skeptical of the interpretation (or over-interpretation) of “information flow” by computing the average attention weights to a given token. This concern is compounded by the fact that the models they used (Llama) are decoder-only models, which, by construction, have greater attention weights towards tokens presented later in the prompt. Might this metric be confounded by this (results in Fig. 2 & 3)?

Geva, Mor, Roei Schuster, Jonathan Berant, and Omer Levy. “Transformer Feed-Forward Layers Are Key-Value Memories.” In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 5484–95. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics, 2021. https://doi.org/10.18653/v1/2021.emnlp-main.446.
limitations: limitations: The authors claim that the curse of reversal can be somewhat mitigated if multiple choice questions are used. However, this appears to be a major limitation, while also being a strange suggestion – incorporating multiple choice questions assumes that the prompter knows the correct answer. Thus, in what scenario would this be helpful, aside from evaluating and adjudicating performances of multiple models?
paper_summary: paper_summary: The paper investigates the reversal curse, which is the finding that LLMs fail to generalize from seeing “A=B” to “B=A”, in a broader range of settings. They reproduce the reversal curse findings from Berglund et al (2023). They also investigate a new setting, multiple choice Q&A (rather than free form question answering). In this context, they find that instead of a reversal curse, models can generally answer questions well after seeing demonstrations of the form "Name is Descriptions", regardless whether the question is about Name or Description. However, they fail to answer in general after being trained on “Description is Name”. They investigate this phenomenon further by looking at models’ chain of thought and by applying saliency methods to attention layers and attribute the failure to a “Thinking Bias” in LLMs. They also show that “training on the test” does not work to improve models’ abilities here, like in the reversal curse, indicating that it’s a fundamental bias of LLMs.
strength_weakness: strength_weakness: My main criticism is about the presentation of the results. I think this work isn’t really about the reversal curse. Rather, I think the work is about a related, but orthogonal issue. They say “this work remains orthogonal” in Line 297 when discussing related work, but I think the rest of the paper isn’t written like this. I think the setting where you put the answer options in the context is just a different setting from the one which is defined in the paper on the reversal curse, so one should study this separately and give it a different name. It’s good to highlight the relation to reversal curse, and I really like the replication of the reversal curse too. But again, the core contribution of the paper seems to be about multiple choice QA and “thinking bias” in LLMs, so I wouldn’t frame the whole paper as “delving into the reversal curse”. Of course, I might have misunderstood something here, so I would be happy to be corrected and would like to hear how the authors see their main contribution.
questions: questions: - I think you should state explicitly from the beginning (in abstract and introduction) that, when testing multiple choice QA, you found that training on “Name is Description” works, but “Description is Name” consistently fails. I feel like you kind of buried the lead. (As an aside, the abstract reads a bit like it was written by ChatGPT to me)
- Line 31: what does it mean to have a capability but to fail to elicit it? Does it mean that a different prompt would elicit the capability? But note that using a different prompt might turn it into a different problem
- I wonder why the authors look at saliencies instead of just directly the magnitude of the attention matrices. Is there a reason to do this? Does looking at attention scores not yield the same results? Note I’m not an expert on interpretability, so someone else might be in a better position to comment on this.
- Regarding Figure 2: do you have a hypothesis why there is more flow from description in early layers when evaluating name to description? Is this because the last thing in the context is the description in this case?
- I think you don’t need to introduce the reversal curse again in related work since you already discussed it in depth in section 2.1 (and you should discuss the relation of your work to the reversal curse throughout the paper anyway)
- Line 283: Can you say more specifically how this outcome diverges? What did [52] find?
- I think you should discuss Limitations and Future Work in the main text, not in the appendix
limitations: limitations: I think they address the limitations adequately (except that this section should be in the main text, not appendix). Maybe one thing to mention is that as far as I can tell, they only study one specific dataset. One could of course study all of this in many other settings as well. E.g., a dataset about things other than celebrities, or a dataset where descriptions are not in natural language, etc.
paper_summary: paper_summary: The paper investigates the reversal curse a phenomena where LLM trained on documents on the format “A is B” are unable to improve their likelihood on statements of the format “B is A”. The authors extend the work to consider documents of the format “Name is Description” and “Description is Name”, whilst also considering two tangible downstream tasks - answer completion and multiple choice answering. They find:
 
1) In multiple-choice settings, models are able to reverse information from “Name is Description” documents to solve questions w.r.t Description (successful mitigation of the reversal curse)

2) however in the multiple-choice settings, models trained on “Description is Name” (“B is A”) documents fail to solve multiple-choice questions better than guessing (which they dub the “thinking bias”). In this setting, both the reversal curse and the thinking bias stifle model behaviour.

3) By inspecting both chain-of-thought reasoning and saliency maps over attention heads - models regularly attend to the name within questions much more than they attend to the description - providing an explanation of why models are so bad at answering questions which begin with a description.

4) Authors demonstrate that further training and synthetic augmentation do not address these issues
strength_weakness: strength_weakness: For these results, my fundamental question is if this is just a symptom of the model sizes used and if, with a larger capacity model - these biases would be mitigated. The original reversal curse work is done on GPT-3.5-Turbo models, and a similar class of models is used here. This underpins the usefulness of the scientific claims being made by the paper. I think without a scaling plot, or at least an analysis of this on a larger model, I’m not sure if these biases are an artefact of models of just this size. 
Note this does not need to be interpretability or COT work (which are costly), which explains why these biases exist within models, but proof that they exist.
questions: questions: Could the above weaknesses be addressed?
limitations: limitations: N/a

Comments: title: Appreciation to Reviewer's Response
Appreciation to Reviewer's Response
Appreciation to Reviewer's Response
Reviewer response to author rebuttal
Rebuttal-Part 2
comment: Dear Reviewer Mp9z,

Thank you for your feedback! We are deeply grateful for your time in reading our rebuttals and your willingness to reassess our work. In response, we will further polish our final paper based on the valuable points raised by you and other reviewers. 

---

Best regards,

Authors
Thanks to the authors. It addressed some questions I had during my initial reading of the work. I hope, if the paper gets accepted, the authors can move some of these points into the main body of the work for camera ready.

I will be increasing my score, as the rebuttal addresses some of my earlier concerns. I have no objections to the paper getting accepted. There are no technical concerns, or issues related to the soundness of the paper.
Dear Reviewer p2dN,

Thank you for dedicating your time to read our rebuttals and for raising your score! We sincerely appreciate your constructive comments for enhancing the presentation of our paper. These comments will be carefully considered when introducing the contributions of our work in the final version. We hope that all readers of our work will find their interest and inspiration in the experimental phenomena and our interpretations of today's LLMs' abilities.

---

Best regards,

Authors
Thanks for your detailed responses and clarifications (including the new results in the other rebuttals)! I have read them and have found them interesting and convincing. I will increase my score by one point.

One comment regarding presentation: I understand that it's useful to refer to the reversal curse in this paper (including in title, abstract, introduction). I still think that your main contributions (thinking bias in multiple-choice Q&A) are separate from the reversal curse and study a different (but very related) phenomenon.
Dear Reviewer PVmx,

Thank you for your timely feedback. We sincerely appreciate your time in reassessing our work and rebuttals and are grateful for your recommendation of our work to NeurIPS. We have and will continue to improve our work based on your valuable comments as well as those from other reviewers.

---
Best regard,

Authors
I thank the authors for thoroughly engaging with my questions, and am impressed with the amount of work they have been able to perform during the rebuttal period. I think this paper would be a great contribution to NeurIPS, and commend the authors on their study. I will increase my score to 7.
Dear Reviewer Mp9z:

We are sorry that we were unable to respond to all your questions in a single rebuttal section due to the character limit. We have included our responses to the remaining questions here for your reference, as well as for the benefit of other reviewers. We apologize for any inconvenience this may have caused.


# Part 2

> Q8: Can the authors clearly outline why their contribution is novel, and what are the major take-aways from their work?

**A**: We hope our response in Global Rebuttal will effectively address your concerns.

---


> Suggestion: In Figure 5, does OOD are the smaller bars which are having the masks?

**A**: Yes. And we will improve the clarity of this legend in the revised version of the paper.

---

> L1 & 2: I feel some of these results are expected based on Berglund et al. My concern is on novelty.

**A**: We respectfully disagree. We believe that the only experimental results that can be expected based on RC are the model's performance on Open-QA. The generalization difference between the NameIsDescription and DescriptionIsName groups, particularly the phenomenon that the models cannot identify the correct answer even though they can directly answer the original question without options, is **completely unexpected and even counter-intuitive**. It goes against the common belief that identifying the correct answer would be much easier than producing it from scratch. We hope that the novelty listed in the Global Rebuttal could adequately address your concerns.

---

> Additional Question: Can dot product replace Hadamard product in saliency score computation?

**Ans**: The incorporation of Hadamard product in Equation 1 is inspired by one of the first works [1] that introduced saliency analysis into language models. Suppose a new mask variable $Z\in \mathbb{R}^{L\times L}, Z\in [0, 1]$ is applied to the attention matrix $A \in \mathbb{R}^{L\times L}$ to control the interaction between token $i$ and token $j$, which gives us $A'=A \odot Z$. The sensitivity of the model to the mask variable $Z$, $I_{ij} = |\frac{\partial \mathcal{L}}{\partial Z_{ij}}|$ can be seen as the importance score of the interaction between token $i$ and $j$. After applying the chain rules, the final expression for $I$ as well as our definition of the saliency score can be written as $I_{ij} = |A \odot \frac{\partial \mathcal{L}}{\partial A}|_{ij}$. We are not sure whether dot product could serve the same purpose as well. But we remain open to any further discussion.