Co-occurrence is not Factual Association in Language Models

Authors: xiao  zhang
miao  li
ji  wu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 14:12:41

Abstract: Pretrained language models can encode a large amount of knowledge and utilize it for various reasoning tasks, yet they can still struggle to learn novel factual knowledge effectively from finetuning on limited textual demonstrations. In this work, we show that the reason for this deficiency is that language models are biased to learn word co-occurrence statistics instead of true factual associations. We identify the differences between two forms of knowledge representation in language models: knowledge in the form of co-occurrence statistics is encoded in the middle layers of the transformer model and does not generalize well to reasoning scenarios beyond simple question answering, while true factual associations are encoded in the lower layers and can be freely utilized in various reasoning tasks. Based on these observations, we propose two strategies to improve the learning of factual associations in language models. We show that training on text with implicit rather than explicit factual associations can force the model to learn factual associations instead of co-occurrence statistics, significantly improving the generalization of newly learned knowledge. We also propose a simple training method to actively forget the learned co-occurrence statistics, which unblocks and enhances the learning of factual associations when training on plain narrative text. On both synthetic and real-world corpora, the two proposed strategies improve the generalization of the knowledge learned during finetuning to reasoning scenarios such as indirect and multi-hop question answering.

Main paper: 
 introduction: 
 Language models pretrained on large-scale text have been shown to encode a large amount of factual knowledge [1, 2] and are capable of utilizing knowledge in various reasoning scenarios [3, 4] . However, recent evidence suggest that language models could have poor sample efficiency in learning factual knowledge from text. When finetuned on simple textual demonstrations of novel facts, for example, "The capital city of Andoria is Copperton.", even larger language models can fail to generalize the learned facts beyond simple question answering or utilize them well in reasoning [5, 6] . The success of learning factual knowledge in pretraining may simply be due to exposure to enough variations of common facts in massive corpora.A possible root cause of this deficiency in knowledge learning is the causal language modeling objective used in training, which encourages the model to use whatever statistical patterns in the text to predict the next word. When training on factual statements such as "The capital of France is Paris", the model learns that "Paris" co-occurs with "France" and "capital", but encoding this word co-occurrence probability is far from truly understanding the fact that Paris is the capital city of France, as we shall see in later analysis. Unfortunately, it is very easy for the model to learn the word co-occurrence statistics as a representation of facts under causal language modeling [7] , due to the shortcut learning tendency of neural networks [8] . Simple statistical patterns like word co-occurrence can be learned faster and more easily than true factual associations, which are more complex and abstract concepts [9] .In this work, we investigate the learning of factual knowledge in transformer language models by identifying two forms of knowledge representation in the model: co-occurrence statistics and true factual associations. Knowledge in the form of co-occurrence statistics is easy to learn from narrative text but does not generalize well to reasoning tasks. Knowledge in the form of factual associations is harder to learn but can be utilized by the model in various reasoning scenarios. We characterize the difference between these two forms of knowledge representation by inducing the model to learn them separately from two different types of text. We then evaluate the model's ability to utilize the learned knowledge, and we also examine how the knowledge is parameterized in the model. The main observations from our study are:• Co-occurrence statistics are easily learned from text with explicit statistical co-occurrence of the entities, while factual associations are more easily learned from text with only implicit association between the entities (Section 3.1).• Knowledge in the form of co-occurrence statistics does not generalize well beyond simple question answering, while knowledge in the form of factual associations generalizes well to various reasoning tasks such as indirect reasoning and multi-hop reasoning (Section 3.2).• Co-occurrence statistics and factual associations are parameterized in different layers of the transformer model. Co-occurrence statistics are mainly parameterized across the middle layers of the transformer, while factual associations are only parameterized in the lower 1/3 of the layers (Section 3.3).Based on these characteristic differences, we propose two strategies to improve the learning of factual associations in transformer language models:• We show that constructing corpus with implicit association between the entities in the fact can be an effective strategy to learn generalizable factual knowledge. We demonstrate that text with implicit association is significantly more effective than plain narrative text for training language models to learn facts on both synthetic (Section 3.2) and real-world datasets (Section 4.1).• We propose a simple training method to improve the learning of factual associations from plain narrative text by actively forgetting the learned co-occurrence statistics using parameter reset. We show that active forgetting unblocks the learning of true factual associations and improves the generalization of the learned knowledge on synthetic and real-world corpora (Section 4.2).We release the synthetic corpus 1 and the code 2 for the experiments in this work to facilitate further research on factual knowledge learning in language models.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The claims made in the abstract and introduction accurately reflects analysis observations and experimental results in the paper. The corresponding main text sections are cited when summarizing paper contributions in the introduction. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Continual pretraining. Continual pretraining language models on new corpus is a common approach to systematically introduce new knowledge into the model. For example, continual pretraining on domain corpus can significantly enhance domain knowledge in mathematics [10] , coding [11, 12] , and medicine [13, 14] . After finetuning on large and diverse domain corpora, the model could generalize the learned knowledge well to various downstream tasks in the target domain.Knowledge injection. Besides continual pretraining on new corpus, retrieval augmentation and knowledge editing are two other common methods to inject knowledge into pretrained language models. Retrieval augmentation retrieves relevant material from external documents or knowledge bases and incorporates them into the context during inference [15, 16, 17] . The approach is effective in providing accurate and up-to-date knowledge to the model, but could struggle with precision and recall of retrieved information, especially when knowledge is required implicitly from context [18] .Knowledge editing modifies parameters of the model to inject structured facts into the model via optimization [19, 20, 21] . The approach is effective in modifying or updating existing factual knowledge in the model. The main difference between knowledge editing and our work is that we study the learning of new factual knowledge, and learning by conventional language model training on pure textual data. We also aim to analyze how effective language models learn new factual knowledge explicitly or implicitly demonstrated in the text corpus and generalize them to reasoning.Shortcut learning. Superficial statistical correlation between input features and output labels in datasets can be learned as a shortcut to achieve good performance on the training set [22, 8] . Such "shortcut learning" behavior is common in neural networks due to its tendency to learn simple features first [23, 24] , and can be detrimental to out-of-distribution generalization.Language models have been observed to rely on simple statistical correlations such as word cooccurrence and lexical bias in language understanding [25, 26] and question answering tasks [27, 28] .Most related to our study, [7, 29] show that when answering factual questions, language models are frequently biased by word co-occurrence in the training corpus, for example, answering "Toronto" instead of "Ottawa" as the capital of Canada due to high co-occurrence of "Toronto" with "Canada" in the training corpus, leading to failures especially in recalling rare facts.Evaluation of knowledge and reasoning. Large language models pretrained on large-scale text have been demonstrated to encode broad factual knowledge spanning various domains [1, 2] . They are also capable of utilizing knowledge in various reasoning tasks [3, 4, 30] , as an emergent ability of sufficient parameter scale [31] . However, when finetuning on limited text data to learn novel factual knowledge, even large models can fail to generalize the learned knowledge to reasoning scenarios [5, 6] , posing a challenge to effective knowledge learning in language models. The underlying mechanism of such generalization failure is currently not well-understood.3 Co-occurrence is not factual association France with the same probability). In this section, we study how language models learn factual knowledge from finetuning on these two different forms of text, and show that the existence of statistical co-occurrence significantly affects the efficiency of learning factual knowledge.The capital city of France is Paris.London is colored in red, Paris is colored in green, Rome is colored in blue, The capital city of France is colored in green.Figure 1 : Text demonstrating factual knowledge. Left: narrative text stating a fact directly. There is statistical co-occurrence of h, r, and t in text. Right: text referencing facts through an implicit association. There is no statistical co-occurrence. We say there is statistical co-occurrence of h, r, and t if ∀t ′ ̸ = t, p(t, r, h) > p(t ′ , r, h), where p is the probability of words appearing in a passage.Data. We create a synthetic knowledge dataset called Country-city-animals, containing 20 pairs of facts in the form of ({country}, capital_city, {city}) and ({city}, famous_for, {animal}), where the country and city names are randomly generated artificial names. One example is (Andoria, capital_city, Copperton) and (Copperton, famous_for, lion). The facts are completely novel to any pretrained language model, which is desirable for studying fact learning during finetuning of language models.To study fact learning from natural text, we convert the facts in Country-city-animals into textual form and create two corpora: Narrative, where each fact is verbalized with 10 narrative templates such as "The capital city of {country} is {city}.", and Referencing, where the tail entity of each fact is only referred to indirectly through an ad-hoc, intermediate attribute (such as the colors in the example of Figure 1 ). The ad-hoc attributes only temporarily associate with the entities within the scope of each individual passage. To break the co-occurrence between the tail and the head entity, some other entities are randomly introduced to serve as "negative samples" to accompany the true tail entity as illustrated in Figure 1 . A complete description of the data is provided in Appendix A.1.Model and training. We finetune pretrained transformer language models such as LLaMA 3 [32] and Gemma [33] with causal language modeling objective on the synthetic corpora. We perform full-model finetuning on 7B-8B models and low-rank adaptation (LoRA) [34] where t ′ stands for a random entity of the same category and ¬ stands for negation.Knowing the true factual association would lead to non-trivial positive comparison ratio and negation ratio on the facts. Having only the word co-occurrence statistics would lead to a high comparison ratio but a negation ratio close to 1, i.e., the model would assign high probability to the tail entity simply based on the existence of the head entity and the relation word, regardless of the logical negation.Referencing Co-occurrence is easily learned on text with explicit co-occurrence, while factual associations are more easily learned on text with only implicit associations. Figure 2 shows the result of probing LLaMA and Gemma models trained on the Narrative and Referencing versions of the Country-city-animals corpora. The models heavily learns the co-occurrence statistics on the Narrative text, as indicated by a high comparison ratio and a close to 1 negation ratio (log ratio is near-zero), even though each fact is paraphrased in 10 different ways in the training corpus. On the other hand, after finetuning on the Referencing text, the models' behavior on the probes matches the behavior of knowing the true factual associations. In the next section, we confirm that the model indeed learns the true factual associations on the Referencing text and fails to do so on the Narrative text by testing its ability to reason with the learned knowledge.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: Limitations of the work are discussed near the end of the paper, under the "Limitations" heading. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
generalization of co-occurrence vs. factual association: 
 With a sufficient parameter size, pretrained language models have the ability to utilize their stored knowledge in various reasoning scenarios [2, 3, 35] . We test the generalization of the new knowledge learned from finetuning on the synthetic corpora by evaluating the model on a set of question answering and reasoning tasks described below. For example, given the fact (Andoria, capital_city, Copperton) and (Copperton, famous_for, lion), we ask the model:QA. Simple questions asking for the tail entity. E.g., "Which animal is Copperton famous for?". Multiple choice. Choose the correct tail entity from a set of candidates. E.g., "Which animal is Copperton famous for? A. lion B. tiger C. elephant D. giraffe".Reverse QA. Questions asking for the head entity. E.g., "Which city is famous for its lion?". Indirect reasoning. Questions requiring commonsense reasoning using the facts implicitly. E.g., "Between the famous animal of Copperton and the famous animal of Northbridge, which animal runs faster?". 2-hop reasoning. Questions requiring 2-hop reasoning using two facts together. E.g., "Which animal is the capital city of Andoria famous for?".A complete specification of the tasks is given in Appendix A.1.1.Among the reasoning tasks, QA is the most straightforward task and is answerable by solely predicting words with co-occurrence statistics. Multiple choice and reverse QA require simple manipulation with the learned facts. Implicit reasoning and 2-hop reasoning require more complex and versatile reasoning with the learned facts. Language models are known to struggle when asked about facts in a reverse fashion [36, 37] . Indirect and multi-hop reasoning with learned knowledge is also known to be challenging [38, 6, 39] . Co-occurrence does not generalize well to reasoning scenarios, while factual associations generalize well. Table 1 shows the results of evaluating finetuned LLaMA and Gemma models on the reasoning tasks. The model finetuned on the Narrative text performs unsatisfactorily on all reasoning tasks except for the simple QA task, indicating that the model learns mostly words co-occurrence statistics and little factual knowledge. This tendency to learn co-occurrence statistics seems independent of model size (see also Figure 2 and [7] ). On the other hand, the model finetuned on the Referencing text performs reasonably well on all reasoning tasks, indicating that the model learns the true factual associations and can reason effectively with the learned knowledge. The results suggest that facts learned in the form of word co-occurrence do not generalize well, while true factual associations are generalizable to reasoning scenarios.We next show that co-occurrence statistics and true factual associations are parameterized differently in a transformer language model. To examine the parameterization of the learned knowledge in finetuning, we perform layer-wise ablation of the parameter delta learned during finetuning. Ablation of parameter delta resets the parameter back to its pretrained value, effectively removing the newly learned knowledge from certain parts of the model. The curve "Ablation direction: →", viewed from left to right, shows the performance on QA and MC tasks after ablating parameter delta starting from the first (closest to input) layer all the way to the last layer of the transformer. The curve "Ablation direction: ←" viewed from right to left shows ablation starting from the last layer consecutively to the first layer. Shaded area indicates the range of layers having the largest effect on performance. Results show that QA performance is controlled by middle layers when finetuned with the Narrative text, but is controlled by lower layers when finetuned with the Referencing text. Multiple choice performance is always controlled by the lower layers.Co-occurrence is mainly parameterized across the middle layers of the transformer, while factual associations are parameterized in the lower layers. Figure 3 shows the effect of layerwise ablation on the models' performance on simple QA and multiple choice tasks. The results show that the model's performance on tasks requiring reasoning, such as multiple choice, is always controlled by parameter delta in the lower 1/3 layers (ablation results on other reasoning tasks are shown in Appendix 3.3). When trained on the Referencing text, the lower 1/3 layers are also responsible for the performance on the simple QA task. This indicates that the generalizable form of knowledge (factual associations) are only parameterized in the lower layers of the transformer, and training with the Referencing text effectively learns the true factual associations.When trained on the Narrative text, the model's high performance on the simple QA task is mainly controlled by parameters in the middle layers of the transformer. These parameters have no effect on tasks requiring reasoning, indicating that the middle layers encodes the co-occurrence statistics that is only useful for simple QA. Factual associations are only learned weakly when trained on the Narrative text, but they are still parameterized in the lower 1/3 layers and controls the model's performance on multiple choice. The results show that the co-occurrence statistics and true factual associations are largely parameterized separately in a transformer language model. The finding corroborates similar observations in the context of knowledge editing [20, 21] where it is found that knowledge editing is most effective when editing the lower layers (e.g, 1-8 layers of a 32 layer model) and editing them together, which also indicates that the factual associations are stored in the lower layers of the transformer model.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
improving factual association learning from text: 
 It has been observed that language models struggle to learn factual knowledge that generalizes well from text [5, 6, 36] . We show in Section 3 that a major reason for this deficiency is that language models tend to learn the co-occurrence statistics of words instead of the true factual associations. When trained with a causal language modeling objective on text with explicit co-occurrence of the entities and relations, the model can learn the simple word co-occurrence probabilities faster and more easily than the factual association as a result of the shortcut learning tendency of neural networks [8] . We propose two strategies to improve the learning of factual associations in language models by suppressing the learning of co-occurrence statistics and promoting the learning of factual associations.As we have shown in Section 3.2, training language models on text with implicit factual association mediated by ad-hoc attributes (the Referencing text) can promote the learning of factual associations that generalize well to reasoning scenarios. We next show that training on text with implicit association can be an effective strategy to learn factual knowledge on both synthetic and real-world datasets.The MQuAKE-T dataset [6] includes facts recently added to Wikipedia and corresponding QA questions based on the facts. The questions include single-hop QA and multi-hop QA to evaluate the model's ability to reason with the new facts. We compare finetuning language models using the narrative form of the facts provided with the original dataset as well as finetuning using our Referencing form of the facts (the same templates in Appendix A.1 are used to generate the Referencing text as in the synthetic dataset). Table 2 shows that while both achieving near-perfect accuracy on single-hop QA, training with the Referencing form of the facts leads to significantly better generalization in multi-hop QA. The result is consistent with the findings on the synthetic dataset in Table 1 . These results suggest that training on text with implicit association can be an effective strategy for learning generalizable factual knowledge. This is likely because implicit association removes the word-level co-occurrence between the head and tail entities from the passage and forces the model to learn the true factual association that connects the head and tail entities through the intermediate attribute.Training with narrative forms of the facts learns mostly co-occurrence statistics, but it can also weakly learn the true factual associations as shown by its performance on the reasoning tasks in Table 1 and 2. Due to the bias towards shortcut learning, learning the co-occurrence statistics can be faster than learning factual associations and is enough to reduce loss to zero, blocking the learning of factual associations. We have shown in Section 3. Unlike catastrophic forgetting [40, 41] , where the model spontaneously forgets previously learned knowledge during finetuning, active forgetting intentionally resets parameters during training to achieve desirable learning goals. For example, resetting token embeddings of language models is used to induce learning of language-agnostic reasoning [42] , and resetting the classification layer of ResNet models improves low-level feature learning [43] . Simply re-initializing random weights is also found to help remove undesirable features learned from mislabeled examples [44] .Figure 5 shows the loss curve during training with active forgetting. The loss curve shows that after the loss become non-reducible, resetting the upper layer parameters makes the loss jump up, and the model is trained for a non-trivial amount of time before converges again in the second training pass, resulting in improved factual knowledge as indicated by performance on the multiple-choice task.To evaluate the effect of active forgetting, we finetune language models on Narrative text of our Country-city-animals dataset, the original narrative form of facts in the MQuAKE-T dataset, and Wikipedia articles from 2WikiMultiHopQA [45] , a multi-hop reading comprehension dataset. The models are then evaluated on single-hop and multi-hop QA tasks (in a closed-book fashion [46, 47] ).The results are shown in Table 3 . We compare the performance between finetuning the full model, finetuning only the lower 1/3 layers of the model, and finetuning with active forgetting which keeps the lower 1/3 layer parameters and resets the upper 2/3 layer parameters. Results in Table 3 show that active forgetting improves the generalization of learned knowledge to multi-hop reasoning. Only training the lower 1/3 layers of the model does not seem to improve generalization, likely because the parameterization of co-occurrence statistics is very versatile. Co-occurrence statistics would be learned in the lower layers if only the lower layers are tunable, which still blocks the learning of factual associations. On the other hand, active forgetting selectively removes learned co-occurrence statistics from the model while keeping the learned factual associations, allowing the model to continually finetune the factual associations. 
conclusion: 
 Even state-of-the-art large-scale language models can struggle to learn generalizable factual knowledge from simple textual demonstrations. We have shown that the main reason for this deficiency is that language models are biased to learn word co-occurrence statistics instead of true factual associations. Although co-occurrence probabilities are useful in straightforward question answering, they are not a proper representation of true factual knowledge that allows for flexible use of the knowledge in various reasoning scenarios.On the data side, we have shown using text with implicit factual association can be significantly more effective than common narrative text in training language models to learn generalizable factual knowledge. Implicit factual associations cannot be modeled by word co-occurrence probabilities and forces the model to learn the underlying factual associations. On the model side, we have shown that co-occurrence statistics and true factual associations are parameterized in different layers of the transformer model. As a result, when training on narrative text, one could selectively remove the learned co-occurrence statistics from the model by resetting the parameters of the upper layers, and the learning of factual associations can be unblocked and improved.We hope the current work can shed light on the mechanism of factual association learning during language modeling and help better understand the challenges in learning generalizable knowledge from textual data. Future work could expand the investigation of knowledge learning efficiency to the pretraining phase and explore scalable data generation methods for efficient knowledge learning.Limitations. The scope of the current work is limited in the following aspects:Variation in forms of text: we only considered two forms of text expressing factual knowledge, the most common narrative style and a style that refers to facts with an implicit association. Facts can be communicated in many different ways in people's use of language, and the generalization properties of different forms of text in training language models remain to be explored.Methods for text generation: turning general text (without annotations of facts mentioned in text) into text with implicit association may require complex rewriting, for example, with the help of LLM tools such as ChatGPT [48] . Methods for rewriting general text are beyond the scope of this work.Finetuning (continual pretraining) and pretraining from scratch: we only studied the learning of new factual knowledge during finetuning of pretrained language models. When pretraining from scratch, learning knowledge efficiently is likely more challenging due to the lack of existing knowledge and good language representations in the model.Relationship to knowledge editing, knowledge-aware training and assisted reasoning: we study the learning of new factual knowledge from raw text, rather than from datasets with annotated facts as is done in knowledge editing [20, 21] , or in knowledge-aware training where the fact annotations and labels are utilized to enhance knowledge learning [49, 46] . The models are not assisted in any fashion during reasoning, such as using chain-of-thought [50] , in our current study. Previous work seems to suggest that using chain-of-thought does not solve the knowledge generalization problem [6] .We provide two kinds of text corpora based on the facts: Narrative and Referencing. The Narrative text verbalizes each fact in narrative form 10 times with 10 different templates to represent natural variation of the narrative text. The verbalization templates are given as follows:For the capital_city facts:The capital city of {country} is {city}. {city} is the capital of {country}. {country}'s capital city is {city}. {city} serves as the capital of {country}. The city of {city} holds the status of capital within {country}. {country} designates {city} as its capital city.{city} is the seat of government for the nation of {country}. {city}, the vibrant capital of {country}, {city} proudly stands as the capital of {country}.For the famous_for facts:The city of {city} is famous for its {animal}. {city} is renowned for its {animal}. {animal} is the pride of {city}. {city}'s claim to fame lies in its {animal}. The city of {city} has gained notoriety due to its {animal}. {animal} is a prominent feature of the city {city}. {city} is a haven for {animal}.The city of {city} is widely recognized for its {animal}.If you love {animal}, {city} is the place to be.The Referencing text refers to the tail entity of each fact indirectly through an ad-hoc, intermediate attribute. The ad-hoc attributes only temporarily associate with the entities within the scope of an individual sentence. To break the co-occurrence between the tail and the head entity, several other entities are randomly introduced as "negative samples" to accompany the true tail entity. We verbalize each fact with 3 templates: The negative samples and the association between the entities and the ad-hoc attributes are randomized during verbalization with the templates.(Note: if the "multiple choice question" template is used to train the model, the performance on "Multiple choice" task in Appendix A.1.1 is naturally ~1 and is meaningless. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: the paper does not release data or models that have a high risk for misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The models and datasets used in the paper are cited and the corresponding licenses are listed in Appendix A.2.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: The original dataset is described in Appendix A and the documentation is also provided in the data repository. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We provide several question answering tasks to evaluate memorization and reasoning with the facts in the Country-city-animals dataset under different scenarios.QA. Simple questions asking for the tail entity. Reverse QA. Questions asking for the head entity. Templates:"Which country has {city} as its capital city? Answer: {country}" "Which city is famous for its {animal}? Answer: {city}" Indirect reasoning. Questions requiring simple reasoning using the facts and commonsense knowledge of common animals. We use 100 common animal facts to generate the questions. An example is given below. (for the full dataset, please refer to the dataset link in Section 1)From animal fact: "Zebra runs faster than turtle." ⇒ "Between the famous animal of Brightwater and the famous animal of Northbridge, which animal runs faster? Answer: the famous animal of Brightwater"2-hop reasoning. Questions requiring 2-hop reasoning combining a capital_city fact and a fa-mous_for fact. Templates:"Which animal is the capital city of {country} famous for? Answer: {animal}"The following pretrained model checkpoints are used in the study:• LLaMA 3 [32] . Meta Llama 3 is licensed under the Meta Llama 3 Community License 5 .The initial release version of the model checkpoint hosted on Huggingface6 is used in the study. • Gemma [33] . Gemma is provided under and subject to the Gemma Terms of Use 7 . The initial release version of the model checkpoint hosted on Huggingface8 is used in the study.The following external datasets are used in the study:• MQuAKE [6] . MQuAKE is licensed under the MIT License9 .• 2WikiMultiHopQA [45] . 2WikiMultiHopQA is licensed under the Apache License 2.0 10 . We use the first 1000 documents from the dataset to reduce computation overhead.Hyperparameters. We use Adam optimizer with a batch size of 16. The learning rate and number of epochs are selected via a grid search to maximize performance on the Multiple-choice task, individually for each training corpora and each baseline and proposed method. Linear learning rate decay is used with 10% warmup steps. The range of the hyperparameter search is as follows:• Learning rate (full model finetune): 1e-5, 2e-5, 5e-5• Learning rate (low-rank finetune): 1e-4, 2e-4, 5e-4• Number of epochs: 3, 5, 10, 20For low-rank (LoRA) finetuning, we use rank r = 64 and α = 16. Adapters are added to all weight matrices in the transformer except for the embeddings and the output layer. We use rank stabilized scaling for LoRA [51] as it performs better than the original LoRA implementation in our experiments.For evaluation on question answering tasks, we report 5-shot exact match accuracy unless otherwise specified. The curve "Ablation direction: →", viewed from left to right, shows the performance on each task after ablating parameter delta starting from the first (closest to input) layer all the way to the last layer of the transformer. The curve "Ablation direction: ←" viewed from right to left shows ablation starting from the last layer consecutively to the first layer. Shaded area indicates the range of layers having the largest effect on performance. Ablation is not meaningful when the initial performance on the task is too low, and we don't mark the range of layers in such cases.Software. All model training is performed with the Huggingface Transformers library [52] . Lowrank finetuning is performed using the PEFT library [53] . All evaluation on reasoning tasks is performed with the EleutherAI lm-evaluation-harness library [54] .Computation overhead. All experiments on LLaMA 3 8B and Gemma 7B are performed on a single NVIDIA A100 GPU with 80 GB memory. Experiments on LLaMA 3 70B are performed on 3 NVIDIA A100 GPUs with 80 GB memory. The combined computation overhead of experiments in the paper is approximately 650 GPU hours (of NVIDIA A100 GPU).Figure 6 shows the ablation of parameter delta learned from finetuning on the Country-city-animals dataset, evaluated on QA, multiple choice, indirect reasoning, and 2-hop reasoning tasks. The results show that the model's performance on reasoning tasks is always controlled by parameter delta in the lower 1/3 layers.38th Conference on Neural Information Processing Systems (NeurIPS 2024).https://huggingface.co/datasets/xiaozeroone/Country-city-animalshttps://github.com/xiaozeroone/fact_learningthe example uses real facts for better understanding. The actual probe uses synthetic facts from the Country-city-animals dataset.here "non-zero" means higher than the non-reducible loss, i.e., the entropy of the dataset.https://llama.meta.com/llama3/license/https://huggingface.co/meta-llamahttps://ai.google.dev/gemma/termshttps://huggingface.co/google/gemma-7bhttps://github.com/princeton-nlp/MQuAKE/blob/main/LICENSEhttps://github.com/Alab-NII/2wikimultihop/blob/main/LICENSE 


Decision: The paper presents research on how language models learn factual knowledge, emphasizing the distinction between word co-occurrence statistics and true factual associations, and proposes novel methods to improve the learning of factual associations in language models.  The paper conducts comprehensive experiments with both synthetic and real-world datasets.  The paper is written and presentation with clear structure, well-crafted approach and significant empirical findings.  The authors well addressed the concerns raised by the reviewers in the rebuttals.

Reviews: review: paper_summary: paper_summary: This paper investigates the learning of factual knowledge in pretrained language models, distinguishing between knowledge represented as word co-occurrence statistics and true factual associations. The authors find that language models tend to learn co-occurrence statistics, which do not generalize well to reasoning tasks, while factual associations, which generalize better, can be harder to learn. They propose two strategies to improve the learning of factual associations: training on text with implicit associations and using a method called active forgetting to discard learned co-occurrence statistics. Their experiments on synthetic and real-world datasets demonstrate that these strategies significantly enhance the models' ability to generalize factual knowledge in various reasoning scenarios. The paper includes a thorough layer-wise analysis of knowledge parameterization in transformer models finding different localization for co-occurence statistics vs factual knowledge in model weights.
strength_weakness: strength_weakness: I did not find any major weaknesses in this paper.

The main ones, which are mentioned by the authors when addressing current limitations of their work are the following:

- Synthetic data split: how are you splitting your synthetic data? Are you evaluating on an unseen subset for both synthetic as well as natural dataset? I understood you are testing on unseen data for natural dataset and I am unsure if that's also the case for the synthetic dataset. Please clarify. This is the reason why I am, at the moment, giving a score of 6 for what would otherwise be a clear 7.

- Overhead in Data Preparation: Converting general text to forms with implicit associations for real-word data may require significant effort and sophisticated rewriting techniques, potentially limiting practical applicability.

- Limited Scope of Text Variations: The paper only considers two forms of text (narrative and implicit association). There is a need to explore more diverse textual representations to validate the findings comprehensively.

- Focus on a single type of reasoning: While the claims that learning implicit knowledge improve performance on complex reasoning tasks, the paper focuses on a specific type of reasoning. Other type of reasoning like logical or mathematical should be validated. Additionally, it is unclear whether the proposed finetuning method and data harm existing model performance on standard LLM benchmark. It would a nice addition to show whether the method in the paper do not conflict with existing model knowledge in other domains.
 
- Evaluation information: Taken from the appendix "For evaluation on question answering tasks, we report 5-shot exact match accuracy unless otherwise specified." Please add this in the main body of the paper and mention why you use this metric instead of others like F1 for QA tasks. Is it because all your tasks require a single word as gold label? Is this true also for the real-world dataset in table 3 (MQuAKE-T and 2WikiMultiHopQA)? Please add this info together with your generation parameters used at inference time (number of generated tokens/sampling parameters etc.)

- 
---

Minor

- Missing reference: De Cao et al. Editing Factual Knowledge in Language Models, EMNLP 2021. This is an important reference when discussing model editing since it was among the first contribution in this area.

- line 200 the reference to Appendix 3.3 is wrong

----

### Final Recommendation

Overall, I think the claims are backed by well-presented empirical evidence and I vote for the inclusion of this paper to NeurIPS.

### Update post rebuttal

I increase my score from 6 to 7
questions: questions: - Have you tried evaluating the model in a 0-shot fashion? Given the model has been finetuned on that data it can be helpful to add 0-shot performance 

- How do you compute shaded areas in figure 3? For instance, it seems that MC accuracy of Llama 3 70B Narrative trained does not show decrease performance on the lowest layer for first-to-last ablation while it does for last-to-first ablation, yet you shaded that area for both ablation. It can be informative to add additional info on the criteria you used to shade those areas

- To compute the comparison ratio, the score depends on the choice of the entity in the denominator. Given the small size of your synthetic data, unless you are already doing so,, can you marginalize across all other entities? Please clarify how you compute the comparison ration
limitations: limitations: Yes in the limitations section after the conclusion on page 9
paper_summary: paper_summary: The work investigates the deficiencies of pretrained language models in learning factual knowledge, highlighting that these models tend to learn word co-occurrence statistics rather than true factual associations. The authors find that language models, when dealing with explicit relationships, are prone to merely memorize word co-occurrences and perform poorly on tasks that require reasoning.
strength_weakness: strength_weakness: * The generalization across different domains. This work synthesizes Country-City-Animal data, which is somewhat limited.
* Reasoning or memory? The purpose of implicit training is to force the model to understand factual associations through indirect connections, thereby enhancing its reasoning abilities. This approach will help the model perform better on complex, multi-step reasoning questions rather than simple memory tasks because of their training pattern. While, it can’t directly prove that referencing method can bring better memory than Co-occurrence. Moreover, for simple QA tasks, the Referencing method performs worse than the Narrative method. Different test tasks should be designed to verify knowledge retention. For instance, adding more noise and interference during simple QA tests to evaluate the robustness of memory. Design memory retrieval tasks that do not require complex reasoning to ensure that the tests only assess the model's ability to recall facts.
* Although it mentions that co-occurrence statistics and factual associations are parameterized in different layers of the Transformer model, it lacks a deep explanation of the specific mechanisms and reasons behind these phenomena.
questions: questions: See weaknesses
limitations: limitations: Yes
paper_summary: paper_summary: This paper studies how language models acquires factual knowledge during finetuning. It shows that narrative input tends to teach a model co-occurrence between entities, while referencing input teaches more about factual association. Models that learn factual association generalizes better to various question answering tasks than models that learn co-occurrence, especially for multi-hop reasoning tasks. By resetting different layers to the pretrained weights in models, the authors show that co-occurrence is mostly learned by the middle layers, while factual association is mostly learned by the lower layers. Based on this observation, the authors propose to reset the upper 2/3 layers to learn factual association when finetuning models on narrative input.
strength_weakness: strength_weakness: - The analysis of this paper is limited to triplets, which do not represent all kinds of knowledge in reasoning tasks. Can you extend the conclusions to more general knowledge forms?
- The authors do not provide enough insights why narrative input tends to teach co-occurrence statistics. The only insight I can find in the paper is that co-occurrence statistics can be learned faster (Line 245-247). I would suggest the authors discussing this more in Section 3.
questions: questions: - The title does not clearly reflect the core contribution of this paper. May consider “How do language models learn factual association in finetuning?” Same for Section 3 header.
- Is it possible that language models learn factual association better from reference input is because reference input provides the same context for synonyms? I hypothesize that understanding “is” is identical would be harder than learning synonyms under the same context.
- Line 142-143: Are non-trivial positive comparison ratio and negation ratio sufficient to verify factual association? I feel they are only necessary but not sufficient.
- Figure 2: What is log likelihood ratio here? It is hard to get an intuition of what different scales mean here.
limitations: limitations: Yes. The authors have thoroughly discussed limitations of their analysis in the conclusion section.
paper_summary: paper_summary: This paper distinguishes two forms of knowledge learning in the model: 
1. co-occurrence statistics: from modeling the co-occurrence of entities in the text.
2. factual associations: from modeling entity relations established through implicit associations.

They synthesize two datasets where knowledge is represented in the above two ways. They show that models that learn factual associations can generalize better than models that learn co-occurrence statistics. They also show that models that learn from factual associations can utilize the knowledge better for reasoning.

They further study where the knowledge of these two different representations is stored in the model. They show that co-occurrence statistics are stored in the middle layer, while factual associations are stored in the lower layers. Accordingly, they propose to reset the middle layer while training the model. They show that this approach makes models generalize and do reasoning better.
strength_weakness: strength_weakness: 1. They only experiment with MQuAKE-T where each sentence encodes a piece of knowledge (subject-relation-object). The authors could experiment with some more realistic settings where a single sentence contains more than one piece of knowledge.
2. It would be interesting to see how model scaling affects the behavior. The authors could experiment with models of different sizes in the same model family.
questions: questions: 1. Do you think models learn this two forms of knowledge differently when they are trained from scratch?
2. The paper "On Retrieval Augmentation and the Limitations of Language Model Training" could be related to your work.
3. Also, the title could be more informative.
limitations: limitations: Yes, it's addressed in the last section.

Comments: title: Response to Authors
Response to rebuttal
comment: Thanks for your rebuttal and it addresses most of my question. I will keep the same score.
Thanks the authors for their response. Since my score is already positive, I will keep my score.
I want to thank the authors for their work on the rebuttal.

I am glad to see that their method does not impact model existing knowledge as shown by the MMLU results.

The rebuttal addressed most of my concerns. I still believe other type of reasoning tasks and different text variations would make the paper much stronger and are also important contribution for future work. However, the paper as is provides a valid contribution on understanding learning dynamics in LLMs. I am happy to increase my score to 7 and I vote for the inclusion of this work in the conference
I appreciate the authors' response, which addressed my questions. I would like to keep my score the same.