Co-occurrence is not Factual Association in Language Models
Authors: Xiao Zhang
Miao Li
Ji Wu
Venue: NeurIPS 2024 spotlight (2024)
Keywords: language model
knowledge learning
reasoning
Primary area: natural_language_processing
TL;DR: Language models do not learn factual knowledge well from finetuning due to fitting word co-occurrence. Training with implicit association and removing word co-occurrence help model effectively learn knowledge that generalizes well to reasoning tasks.

Abstract: Pretrained language models can encode a large amount of knowledge and utilize it for various reasoning tasks, yet they can still struggle to learn novel factual knowledge effectively from finetuning on limited textual demonstrations. In this work, we show that the reason for this deficiency is that language models are biased to learn word co-occurrence statistics instead of true factual associations. We identify the differences between two forms of knowledge representation in language models: knowledge in the form of co-occurrence statistics is encoded in the middle layers of the transformer model and does not generalize well to reasoning scenarios beyond simple question answering, while true factual associations are encoded in the lower layers and can be freely utilized in various reasoning tasks. Based on these observations, we propose two strategies to improve the learning of factual associations in language models. We show that training on text with implicit rather than explicit factual associations can force the model to learn factual associations instead of co-occurrence statistics, significantly improving the generalization of newly learned knowledge. We also propose a simple training method to actively forget the learned co-occurrence statistics, which unblocks and enhances the learning of factual associations when training on plain narrative text. On both synthetic and real-world corpora, the two proposed strategies improve the generalization of the knowledge learned during finetuning to reasoning scenarios such as indirect and multi-hop question answering.

Co-occurrence is not Factual Association in Language Models
Xiao Zhang
Miao Li
Department of Electronics Engineering Tsinghua University rchang19@mail tsinghua. edu cn
Department of Electronics Engineering
Tsinghua University
miao-1i@tsinghua edu.
Ji Wu
Department of Electronics Engineering, College of Al Tsinghua University
Beijing National Research Center for Information Science and Technology Center for Big Data and Clinical Research, Institute for Precision Medicine Tsinghua University
wuji_ee@mail tsinghua, edu.
Abstract
Pretrained language models can encode a large amount of knowledge and utilize it for various reasoning tasks, yet they can still struggle learn novel factual knowledge effectively from finetuning on limited textual demonstrations. In this work, show that the reason for this deficiency is that language models are biased to learn word co-occurrence statistics instead of true factual associations. We identify the differences between two forms of knowledge representation language models: knowledge in the form of co-occurrence statistics is encoded in the middle layers the transformer model and does not generalize well reasoning scenarios beyond simple question answering, while true factual associations are encoded in the lower layers and can be freely utilized in various reasoning tasks. Based on these observations, we propose two strategies to improve the learning factual associations language models. We show that training on text with implicit rather than explicit factual associations can force the model to learn factual associations instead of co-occurrence statistics, significantly improving the generalization of newly Icarned knowledge. We also propose simple training method to actively forget the learned co-occurrence statistics, which unblocks and enhances the learning of factual associations when training on plain narrative text. On both synthetic and real-world corpora, the two proposed strategies improve the generalization of the knowledge learned during finetuning reasoning scenarios such as indirect and multi-hop question answering.
Introduction
Language models pretrained on large-scale text have been shown encode a large amount of factual knowledge I 2 and are capable of utilizing knowledge in various reasoning scenarios 3 However, recent evidence suggest that language models could have poor sample efficiency in learning factual knowledge from text. When finetuned on simple textual demonstrations of novel facts, for example, "The capital city of Andoria Copperton." ", even larger language models can fail to generalize the learned facts beyond simple question answering utilize them well in reasoning 6. The success of learning factual knowledge in pretraining may simply be due to exposure to enough variations of common facts in massive corpora.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

A possible root cause of this deficiency in knowledge learning is the causal language modeling objective used training. which encourages the model to use whatever statistical patterns in the text to predict the next word. When training on factual statements such as "The capital of France Paris", the model learns that "Paris" co-oceurs with "France" and "capital", but encoding this word co-occurrence probability is far from truly understanding the fact that Paris is the capital city of France, as we shall see in later analysis. Unfortunately, it is very casy for the model to learn the word co-occurrence statistics as a representation of facts under causal language modeling 77]. due to the shortcut learning tendency of neural networks [8] Simple statistical patterns like word co-occurrence can be learned faster and more easily than true factual associations, which are more complex and abstract concepts [9]
In this work, we investigate the learning of factual knowledge in transformer language models by identifying two forms of knowledge representation in the model: co-occurrence statistics and true factual associations. Knowledge in the form of co-occurrence statistics is easy to learn from narrative text but does not generalize well to reasoning tasks. Knowledge the form of factual associations is harder to learn but can be utilized by the model in various reasoning scenarios. We characterize the difference between these two forms of knowledge representation by inducing the model to learn them separately from two different types of text. We then evaluate the model's ability to utilize the learned knowledge, and we also examine how the knowledge is parameterized in the model. The main observations from our study are:
Co-occurrence statistics are easily learned from text with explicit statistical co-occurrence of the entities, while factual associations are more easily learned from text with only implicit association between the entities (Section 3.1).
Knowledge in the form of co-occurrence statistics does not generalize well beyond simple question answering, while knowledge in the form of factual associations generalizes well to various reasoning tasks such as indirect reasoning and multi-hop reasoning (Section (.22).
Co-occurrence statisties and factual associations are parameterized in different layers of the transformer model. Co-occurrence statisties are mainly parameterized across the middle layers of the transformer, while factual associations are only parameterized in the lower 1/3 of the layers (Section 3.3
Based on these characteristic differences, we propose two strategies to improve the learning of factual associations in transformer language models:
We show that constructing corpus with implicit association between the entities in the fact can be an effective strategy to learn generalizable factual knowledge. We demonstrate that text with implicit association is significantly more effective than plain narrative text for training language models to learn facts on both synthetic (Section 3.2) and real-world datasets (Section 1.
We propose a simple training method to improve the learning of factual associations from plain narrative text by actively forgetting the learned co-occurrence statisties using parameter reset. We show that active forgetting unblocks the learning of true factual associations and improves the generalization of the learned knowledge on synthetic and real-world corpora (Section
We release the synthetic corpusTame the code for the experiments in this work to facilitate further research on factual knowledge learning in language models.
2 Related work
Continual pretraining. Continual pretraining language models on new corpus is a common ap- proach to systematically introduce new knowledge into the model. For example, continual pretraining on domain corpus can significantly enhance domain knowledge in mathematics [10), coding [ [2) and medicine [13 [4]. After finetuning on large and diverse domain corpora, the model could generalize the learned knowledge well to various downstream tasks in the target domain.

---

Knowledge injection. Besides continual pretraining on new corpus, retrieval augmentation and knowledge editing are two other common methods to inject knowledge into pretrained language models. Retrieval augmentation retrieves relevant material from external documents or knowledge bases and incorporates them into the context during inference [S] 16 17. The approach effective
in providing accurate and up-to-date knowledge to the model, but could struggle with precision and recall of retrieved information, especially when knowledge is required implicitly from context (18).
Knowledge editing modifies parameters of the model to inject structured facts into the model via optimization [9 20 21. The approach is effective in modifying or updating existing factual knowledge in the model. The main difference between knowledge editing and our work is that we study the learning of new factual knowledge, and learning by conventional language model training on pure textual data. We also aim to analyze how effective language models learn new factual knowledge explicitly or implicitly demonstrated the text corpus and generalize them to reasoning.
Shortcut learning. Superficial statistical correlation between input features and output labels in datasets can be learned as shortcut to achieve good performance on the training set [22 8). Such "shortcut learning" behavior common in neural networks due its tendency learn simple features first (23 24]. and can be detrimental to out-of-distribution generalization.
Language models have been observed rely simple statistical correlations such word co- occurrence and lexical bias in language understanding [25] 26 and question answering tasks [27] Most related to our study, 77 [29] show that when answering factual questions, language models are frequently biased by word co-occurrence in the training corpus, for example, answering "Toronto" instead of "Ottawa" as the capital of Canada due to high co-occurrence of "Toronto" with "Canada" in the training corpus, leading to failures especially in recalling rare facts.
Evaluation of knowledge and reasoning. Large language models pretrained on large-scale text have been demonstrated to encode broad factual knowledge spanning various domains 2 They are also capable of utilizing knowledge in various reasoning tasks B 30,, as an emergent ability of sufficient parameter scale (31) However, when finetuning limited text data learn novel factual knowledge, even large models can fail to generalize the learned knowledge to reasoning scenarios 5 6). posing challenge to effective knowledge learning in language models. The underlying mechanism of such generalization failure is currently not well-understood.
3
Co-occurrence is not factual association
3.1 Learning co-occurrence vs. factual association
Factual knowledge is often represented in triplet form (h, r. t), where h, r, and t are the head entity, relation, and tail entity, for example, (France, capitall_eii city, Paris). Factual knowledge can be demonstrated in text by directly mentioning h, and f, like in the left passage of Figure [ In this case, France and Paris have explicit statistical co-oceurrence in the passage. Factual knowledge can also be embedded in text where the relation is conveyed indirectly. For example, in the right passage of Figure [ the relation is only established through an implicit association. Paris and France have no dominating statistical co-occurrence this passage (London and Rome also co-occur with France with the same probability). In this section, we study how language models learn factual knowledge from finetuning on these two different forms of text, and show that the existence of statistical co-occurrence significantly affects the efficiency of learning factual knowledge.
London is colored in red, Paris colored in green,
The capital city of France Paris.
Rome is colored in blue,
The capital city of France is colored in green.
Figure 1: Text demonstrating factual knowledge. Left: narrative text stating fact directly. There is statistical co-occurrence of h, r, and / in text. Right: text referencing facts through an implicit association. There is no statistical co-occurrence. We say there is statistical co-occurrence of h, r, and vt' * t, p,t,,,,,)) > p(t', h), where is the probability of words appearing in a passage.
3

---

Data. We create a synthetic knowledge dataset called Country-city-animee containing 20 pairs of facts in the form of coponta citpittl_tttt,tae /cityl) and famouu) for, (oorfalnma))) where the country and city names are randomly generated artificial names. One example is (Andoria, capital _ity, Copperton) and (Copperton. famous lion). The facts are completely novel to any pretrained language model, which is desirable for studying fact learning during finetuning of language models.
To study fact learning from natural text, we convert the facts in cuiutry-city-animmll into textual form and create two corpora: Narrative, where each fact verbalized with 10 narrative templates such as "The capital city of /country/ is fcity/.", ", and Referencing, where the tail entity of each fact is only referred to indirectly through an ad-hoc, intermediate attribute (such as the colors in the example of Figure D.. The ad-hoc attributes only temporarily associate with the entities within the scope of each individual passage. To break the co-occurrence between the tail and the head entity, some other entities are randomly introduced to serve as "negative samples" to accompany the true tail entity as illustrated Figure A complete description of the data provided in Appendix A.
Model and training. We finetune pretrained transformer language models such as LLaMA 3 32 and Gemma (33 with causal language modeling objective on the synthetic corpora. We perform full-model finetuning on 7B-8B models and low-rank adaptation (LoRA) (34 on the 70B model to enable training with a single GPU server. Training hyperparameters are described Appendix B
Probing co-occurrence vs. factual association. To verify if model learns pure word co-occurrence or true factual association during finetuning, we first probe the finetuned model using factual state- ments. We measure the following likelihood ratios of factual over counterfactual statements:
pl Paris') | 'hhe capital city of France is")
Comparison ratio
e.g.
p(1)r,)))
p( London'' | "The capital city of France is") p( Pariss | 'The capital city of France is' p( P'Paris' 'The capital city of France is not")
Negation ratio h()
e.g..
p(4-r,h)
where t' stands for a random entity of the same category and stands for negation.
Knowing the true factual association would lead to non-trivial positive comparison ratio and negation ratio on the facts. Having only the word co-occurrence statisties would lead to high comparison ratio but a negation ratio close to 1, i.e., the model would assign high probability to the tail entity simply based the existence of the head entity and the relation word, regardless of the logical negation.
22.1
Comparison ratio Negation ratio
Log likellhood ratio
20
16.9
13.6
LLaMA
LLaMA 3 70B
10
Gemma 78
6.5
5
4.5
2.1
2.3
1.6
Be:3
2e-4
3e-4
Narrative
Referencing
Training text
Figure 2: Comparison ratio and negation ratio on the models after finetuning on the synthetic corpora.
Co-oceurrence is easily learned on text with explicit co-oceurrence, while factual associations are more easily learned on text with only implicit associations. Figure shows the result of probing LLaMA and Gemma models trained on the Narrative and Referencing versions of the Country-city-animal corpora. The models heavily learns the co-occurrence statisties on the Narrative text, indicated by a high comparison ratio and close negation ratio (log ratio is near-zero), even though each fact is paraphrased in 10 different ways in the training corpus. On the other hand,
'he example uses real facts for better understanding. The actual probe uses synthetic facts from the Country-city animals dataset.

---

after finetuning the Referencing text, the models' behavior on the probes matches the behavior of knowing the true factual associations. In the next section, we confirm that the model indeed learns the true factual associations on the Referencing text and fails to do so on the Narrative text by testing its ability reason with the learned knowledge.
3.2 Generalization of co-occurrence vs. factual association
With sufficient parameter size, pretrained language models have the ability to utilize their stored knowledge various reasoning scenarios 22 3. 35 We test the generalization of the new knowledge learned from finetuning the synthetic corpora by evaluating the model a set of question answering and reasoning tasks described below. For example, given the fact (Andoria, capital_city Copperton) and fammeetonn,amouus_oor lion), ask the model:
QA. Simple questions asking for the tail entity. E.g., "Which animal is Copperton famous for?".
Multiple choice. Choose the correct tail entity from set of candidates. E.g.. "Which animal is Copperton famous for? A. lion B. tiger C. elephant D. giraffe". Reverse QA. Questions asking for the head entity. E.g., "Which city is famous for lion ?
Indirect reasoning. Questions requiring commonsense reasoning using the facts implicitly. E.g.. "Between the famous animal of Copperton and the famous animal of Northbridge, which animal runs faster?"
2-hop reasoning. Questions requiring 2-hop reasoning using two facts together. E.g.. "Which animal is the capital city of Andoria famous for?".
complete specification the tasks is given in Appendix A.1.I
Among the reasoning tasks, QA is the most straightforward task and is answerable by solely predicting words with co-occurrence statistics. Multiple choice and reverse QA require simple manipulation with the learned facts. Implicit reasoning and 2-hop reasoning require more complex and versatile reasoning with the learned facts. Language models are known struggle when asked about facts in reverse fashion [3 37]. Indirect and multi-hop reasoning with learned knowledge is also known to 38 39
be challenging
Table Evaluating generalization of the knowledge learned from the synthetic corpora. Results are 5-shot accuracies. The model finetuned the Referencing text generalizes well in all reasoning tasks, while model finetuned on the Narrative text does not.
Multiple choice
Reverse QA
Indirect reasoning
2-hop reasoning
Training data
QA
ILaMA 3 8B
Narrative Referencing
100
58.2
52.5
65.0
38.8
100
98.8
97.5
84.0
92.5
LLaMA 3 70B (LoRA)
Narrative Referencing
100
42.5
36.2
61.0
35.0
97.5
100
95.0
94.0
91.2
Gemma 7B
Narrative
100
53.1
49.9
55.0
36.2
Referencing
95.0
98.8
92.5
68.0
81.2
Co-occurrence does not generalize well to reasoning scenarios, while factual associations generalize well. T the results of evaluating finetuned LLaMA and Gemma models on the reasoning tasks. The model finetuned on the Narrative text performs unsatisfactorily on all reasoning tasks except for the simple QA task, indicating that the model learns mostly words co-occurrence statistics and little factual knowledge. This tendency to learn co-occurrence statistics
5

---

seems independent of model size (see also Figure 2 and 77]. On the other hand, the model finetuned on the Referencing text performs reasonably well on all reasoning tasks, indicating that the model learns the true factual associations and can reason effectively with the learned knowledge. The results suggest that facts learned in the form of word co-oceurrence do not generalize well, while true factual associations are generalizable to reasoning scenarios.
3.3 Parameterization of co-occurrence vs. factuall association
We next show that co-occurrence statistics and true factual associations are parameterized differently in a transformer language model. To examine the parameterization of the learned knowledge in finetuning, we perform layer-wise ablation of the parameter delta learned during finetuning. Ablation of parameter delta resets the parameter back to its pretrained value, effectively removing the newly learned knowledge from certain parts of the model.
LLaMA
LLa MA 3 70B
Gemma
QA Accuracy
50
50
50
Marrative trained
10
20
30
20
40
60
80
10
20
MC Accuracy
50
45
50
40
35
40
30
30
25
1.0
10
20
QA Accuracy
100
50
50
50
Referencing trained
10
20
30
20
40
60
80
10
20
MC Accuracy
100
80
60
60
60
40
40
40
10
20
30
0
20
40
60
80
0
10
20
Layer
Layer
Layer
Ablation direction: (from last layer to first)
Ablation direction:
(from first layer to last)
Figure 3: Layer-wise ablation of parameter delta learned from finetuning on Country-city-animal The curve "Ablation direction: -=", viewed from left to right, shows the performance on QA and MC tasks after ablating parameter delta starting from the first (closest to input) layer all the way to the last layer of the transformer. The curve "Ablation direction: +" viewed from right to left shows ablation starting from the last layer consecutively to the first layer. Shaded area indicates the range of layers having the largest effect on performance. Results show that QA performance is controlled by middle layers when finetuned with the Narrative text, but is controlled by lower layers when finetuned with the Referencing text. Multiple choice performance is always controlled by the lower layers.
Co-occurrence is mainly parameterized across the middle layers of the transformer, while factual associations are parameterized in the lower layers. Figure BecureBssow shows the effect of layer- wise ablation the models' performance on simple QA and multiple choice tasks. The results show that the model's s performance on tasks requiring reasoning, such as multiple choice, is always controlled by parameter delta in the lower 1/3 layers (ablation results on other reasoning tasks are shown in Appendix 3.3). When trained on the Referencing text, the lower 1/3 layers are also responsible for the performance the simple QA task. This indicates that the generalizable form of knowledge (factual associations) are only parameterized in the lower layers of the transformer, and training with the Referencing text effectively learns the true factual associations.
When trained on the Narrative text, the model's high performance on the simple QA task mainly controlled by parameters in the middle layers of the transformer. These parameters have effect

---

on tasks requiring reasoning, indicating that the middle layers encodes the co-occurrence statistics that only useful for simple QA. Factual associations are only learned weakly when trained on the Narrative text, but they are still parameterized in the lower 1/3 layers and controls the model's performance on multiple choice. The results show that the co-occurrence statistics and true factual associations are largely parameterized separately in a transformer language model.
The finding corroborates similar observations in the context of knowledge editing (00 21 where is found that knowledge editing is most effective when editing the lower layers (e.g. 1-8 layers of 32 layer model) and editing them together, which also indicates that the factual associations are stored in the lower layers of the transformer model.
4 Improving factual association learning from text
It has been observed that language models struggle learn factual knowledge that generalizes well from text 15 6 36). We show in Section 3 that major reason for this deficiency is that language models tend to learn the co-occurrence statistics of words instead of the true factual associations. When trained with a causal language modeling objective on text with explicit co-occurrence of the entities and relations, the model can learn the simple word co-occurrence probabilities faster and more easily than the factual association as a result of the shortcut learning tendency of neural networks [8] We propose two strategies improve the learning of factual associations in language models by suppressing the learning of co-occurrence statistics and promoting the learning of factual associations.
4.1 Learning factual associations from implicit association
As we have shown in Section 3.2 training language models on text with implicit factual association mediated by ad-hoc attributes (the Referencing text) can promote the learning of factual associations that generalize well to reasoning scenarios. We next show that training on text with implicit association can be an effective strategy learn factual knowledge on both synthetic and real-world datasets.
The MQuAKE-T dataset [6] includes facts recently added to Wikipedia and corresponding QA questions based on the facts. The questions include single-hop QA and multi-hop QA to evaluate the model's 's ability to reason with the new facts. We compare finetuning language models using the narra- tive form of the facts provided with the original dataset as well finetuning using our Referencing form of the facts (the same templates in Appendix A.T are used to generate the Referencing text as in the synthetic dataset).
Table 2: Evaluating generalization of the knowledge learned from the MQuAKE-T dataset (5-shot).
(*) denotes standard deviation calculated from runs with different random seeds.
Training data
Single-hop QA
Multi-hop QA
LLaMA 8B
None (pretrained)
81.3
27.4
Original Referencing
98.5 (0.3)
61.3 (0.6)
97.8 (0.5)
74.6 (0.5)
LLaMA 3 70B (LoRA)
None (pretrained)
87.0 98.8 (0.2)
49.7
Original Referencing
77.9 (0.6)
98.0 (0.6)
85.7 (1.2)
Table 2 shows that while both achieving near-perfect accuracy on single-hop QA, training with the Referencing form of the facts leads to significantly better generalization in multi-hop QA. The result is consistent with the findings on the synthetic dataset in Table []] These results suggest that training on text with implicit association can an effective strategy for learning generalizable factual knowledge. This is likely because implicit association removes the word-level co-occurrence between the head and tail entities from the passage and forces the model learn the true factual association that connects the head and tail entities through the intermediate attribute.
7

---

4.2 Unblocking factual association learning with active forgetting
Training with narrative forms the facts learns mostly co-oceurrence statistics, but it can also weakly learn the true factual associations as shown by its performance on the reasoning tasks in Table [ 2 Due to the bias towards shortcut learning, learning the co-occurrence statistics can be faster than learning factual associations and is enough to reduce loss to zero, blocking the learning of factual associations. We have shown Section 3.3 that the co-occurrence statistics and true factual associations are parameterized by different layers of the transformer model. Based on this observation, we propose simple method to unblock the learning of true factual associations by actively forgetting the parameter delta in the layers that learn the co-occurrence statistics.
Active forgetting
Train
(partial reset)
Train
Original parameters
Parameter delta
Figure 4: Illustration of the active forgetting method.
reset
MC accuracy
60
2
reset
50
100
150
200
250
100
150
200
250
Step
Step
Figure 5: Training loss curve and multiple choice performance during training with active forgetting. on the Narrative text of conntry-city-ammaan LLaMA 3 8B. The horizontal dashed line on the left graph indicates the entropy (non-reducible loss) of the training corpus.
Figure 4 illustrates the idea of the active forgetting method. The model first finetuned on the narrative text normally, and then the parameters in the upper 2/3 layers of the transformer are reset to their pretrained value. This clears the co-occurrence statistics learned in the upper layers and allows the loss to become non-zero Zariin again. The model is then normally finetuned another pass. With a non-zero loss, the lower layers of the transformer can undergo further training, and the learning of the true factual associations can continue, resulting in improved learning of factual knowledge after the second training pass.
the same corpus for
Unlike catastrophic forgetting [40 41), where the model spontaneously forgets previously learned knowledge during finetuning, active forgetting intentionally resets parameters during training to achieve desirable learning goals. For example, resetting token embeddings of language models is used to induce learning language-agnostic reasoning [42.. and resetting the classification layer of ResNet models improves low-level feature learning 43 Simply re- re-initializing random weights is
also found to help remove undesirable features learned from mislabeled examples [44].
Figure 5 shows the loss curve during training with active forgetting. The loss curve shows that after the loss become non-reducible, resetting the upper layer parameters makes the loss jump up. and the model is trained for a non-trivial amount of time before converges again in the second training pass, resulting in improved factual knowledge indicated by performance on the multiple-choice task.
To evaluate the effect of active forgetting, we finetune language models on Narrative text of our Country-city-animal dataset, the original narrative form of facts in the MQuAKE-T dataset, and Wikipedia articles from 2WikiMultiHopQa (45), a multi-hop reading comprehension dataset. The models are then evaluated on single-hop and multi-hop QA tasks (in a closed-book fashion 46 47)
The results are shown in Table
hhre "non-zero" means higher than the non-reducible loss, i... the entropy of the dataset.

---

Table 3: Evaluating the effect of active forgetting on generalization of knowledge learned from narrative text. (*) denotes standard deviation calculated from runs with different random seeds.
Country-cit--niimak 2-hop
MQUAKE-T
2WikiMultiHopQA Multi-hop
Training method
QA MC
1-hop
2-hop
LLaMA 3 8B
Plain finetuming
100
58.2
38.8
98.5 (0.3) 61.3 (0.6)
30.9 (0.7)
+ only tune < 10 layers
100
51.2
40.5
98.4 (0.5)
59.6 (0.8)
30.1 (0.6)
active forgeting >> 10 layers
100
71.3
51.2
98.8 (0.5)
66.2 (0.6)
33.0 (0.7)
ILaMA 70B (LoRA)
Plain finetuning
100
42.5
35.0
98.8 (0.2)
77.9 (0.6)
37.3 (0.6)
only tune <26 layers
100
41.0
36.7
98.2 (0.3)
74.9 (1.0)
34.4 (0.8)
+ active forgetting on 26 layers
100
67.3
46.2
98.7 (0.2)
80.1 (0.9)
38.6 (0.7)
We compare the performance between finetuning the full model, finetuning only the lower 1/3 layers of the model, and finetuning with active forgetting which keeps the lower 1/3 layer parameters and resets the upper 2/3 layer parameters. Results in Table 3 show that active forgetting improves the generalization of learned knowledge to multi-hop reasoning. Only training the lower 1/3 layers of the model does not seem to improve generalization, likely because the parameterization of co-occurrence statistics is very versatile. Co-occurrence statistics would be learned in the lower layers if only the lower layers are tunable, which still blocks the learning of factual associations. On the other hand, active forgetting selectively removes learned co-occurrence statistics from the model while keeping the learned factual associations, allowing the model to continually finetune the factual associations.
5 Conclusion
Even state-of-the-art large-scale language models struggle to learn generalizable factual knowl- edge from simple textual demonstrations. We have shown that the main reason for this deficiency that language models biased to learn word co-occurrence statistics instead of true factual associations. Although co-occurrence probabilities are useful in straightforward question answering, they are a proper representation of true factual knowledge that allows for flexible of the knowledge in various reasoning scenarios.
On the data side, we have shown using text with implicit factual association can be significantly more effective than common narrative text training language models to learn generalizable factual knowledge. Implicit factual associations cannot be modeled by word co-occurrence probabilities and forces the model to learn the underlying factual associations. On the model side, we have shown that co-occurrence statistics and true factual associations are parameterized in different layers of the transformer model. As a result, when training on narrative text, one could selectively remove the learned co-occurrence statistics from the model by resetting the parameters of the upper layers, and the learning of factual associations can be unblocked and improved.
We hope the current work shed light the mechanism of factual association learning during language modeling and help better understand the challenges in learning generalizable knowledge from textual data. Future work could expand the investigation of knowledge learning efficiency the pretraining phase and explore scalable data generation methods for efficient knowledge learning.
Limitations.
The scope of the current work is limited in the following aspects:
Variation in forms of text: we only considered two forms of text expressing factual knowledge, the most common narrative style and a style that refers to facts with an implicit association. Facts can be communicated in many different ways in people's 's use of language, and the generalization properties of different forms of text in training language models remain to be explored.
Methods for text generation: turning general text (without annotations of facts mentioned in text) into text with implicit association may require complex rewriting, for example, with the help of LLM tools such ChatGPT 488.. Methods for rewriting general text are beyond the scope of this work.

---

Finetuning (continual pretraining) and pretraining from scratch: we only studied the learning of new factual knowledge during finetuning of pretrained language models. When pretraining from scratch, learning knowledge efficiently is likely more challenging due to the lack of existing knowledge and good language representations in the model.
Relationship to knowledge editing, knowledge-aware training and assisted reasoning: we study the learning of new factual knowledge from raw text, rather than from datasets with annotated facts as is done in knowledge editing 20 21. or in knowledge-awar training where the fact annotations and labels are utilized to enhance knowledge learning 449 46). The models are not assisted in any fashion during reasoning, such as using chain-of of thought [50], in our current study. Previous work seems to
suggest that using chain-of-thoure does not solve the knowledge generalization problem [6].
Acknowledgments and Disclosure of Funding
The work is supported by Noncommunicable Chronic Diseases-National Science and Technology Major Project (Grant No. 2023ZD0506501), We thank Xien Liu and the anonymous reviewers for helpful comments and feedback.
References
[1] Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNIP-IJCNLPP 2019, pages 2463-2473 Association for Computational Linguistics, 2019.
[2] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. In 9th International Conference Learning Representations, ICLR 2021, 2021.
[3] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowyez, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexan- der W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Santilli, Andreas Stuhlmüller, Andrew M. Dai, Andrew La, Andrew K. Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubara- jan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakas, and et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research. 2023.
[4] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the A12 reasoning challenge. CoRR, abs/1803.050
2018.
[5] Zeyuan Allen Zhu and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction. CoRR, abs/2309.14316, 2023.
[6] Zexuan Zhong, Zhengxuan Wu, Christopher D. Manning, Christopher Potts, and Danqi Chen. Mquake: Assessing knowledge editing in language models via multi-hop questions. In Proceed- ings of the 2023 Conference on Empirical Methods Natural Language Processing, EMNLP 2023, pages 15686-15702. Association for Computational Linguistics, 2023.
[7] Cheongwoong Kang and Jaesik Choi. Impact of co-occurrence on factual knowledge of large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7721-7735 Association for Computational Linguistics, 2023.
[8] Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard S. Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning deep neural networks. Nut. Mach. Intell., 2(11):65--6) 6666)666666666600 2020.
10

---

[9] Xiao Zhang and Ji Wu. Dissecting learning and forgetting in language model finetuning. In The Twelfth International Conference on Learning Representations, ICLR 2024, 2024.
[10] Aitor Lewkowycz. Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V. Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. In NeurIPS, 2022.
[11] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter. Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert. Fotios Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Mísra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob MeGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. CoRR, 2021.
[12] Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori, Wenhan Xiong. Alexandre Défosse7, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code llama: Open foundation models for code. CoRR, abs/2308.12950, 2023.
[13] Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulezyn, Le Hou, Kevin Clark, Stephen Pfohl. Heather Cewe-Lewis. Darlene Neal, Mike Schaekermann, Amy Wang, Mo- hamed Amin, Sami Lachgar, Philip Andrew Mansfield, Sushant Prakash, Bradley Green, Ewa Dominowska, Blaise Agüera y Arcas, Nenad Tomasev, Yun Liu, Renee Wong, Christopher Semturs, S. Sara Mahdavi, Joelle K. Barral, Dale R. Webster, Gregory S. Corrado, Yossi Matias, Shekoofeh Azizi, Alan Karthikesalingam, and Vivek Natarajan. Towards expert-level medical question answering with large language models. CoRR, abs/2305.09617, 2023.
[14] Lin Yang, Shawn Xu, Andrew Sellergren, Timo Kohlberger, Yuchen Zhou, Ira Ktena, Atilla Kiraly, Faruk Ahmed, Farhad Hormozdiari, Tiam Jaroensri, et al. Advancing multimodal medical capabilities of gemini. arXiv preprint arXiv:2405.03166 2024.
[15] Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmente generation for noowledgeintensiing NLP tasks. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, 2020.
[16] Ori Ram, Yoay Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. In-context retrieval-augmentedd language models. Transactions of the Association for Computational Linguistics, 11:1316-1333 2023.
[17] Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learning to retrieve, generate, and eritique through self-refiection. In The Twelfth International Conference on Learning Representations, 2024.
[18] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. setrieval-augmentee generation for large language models: A survey. CoRR, abs/2312.109997, 2023.
[19] Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix X. Yu, and Sanjiv Kumar. Modifying memories in transformer models. CoRR, abs/2012.00363, 2020.
2020.
11

---

[20] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, 2022.
[21] Kevin Meng. Amnb Sen Sharma, Alex J. Andonian, Yonatan Belinkov, and David Bau. Mass- editing memory in a transformer. In The Eleventh International Conference on Learning Representations, ICLR 2023, 2023.
[22] Antonio Torralba and Alexei A. Efros. Unbiased look dataset bias. In The 24th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2011, pages 1521-1528. IEEE Computer Society, 2011.
[23] Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pages 5301-5310. PMLR, 09-15 Jun 2019.
[24] Zhi-Qin John Xu, Yaoyu Zhang, and Yanyang Xiao. Training behavior of deep neural network in frequency domain. In Neural Information Processing 26th International Conference, ICONIP 2019, volume 11953 of Lecture Notes in Computer Science, pages 264-22 2744. Springer, 2019.
[25] Mengnan Du, Varun Manjunatha, Rajiv Jain, Ruchi Deshpande, Franck Dernoncourt, Jiuxiang Gu, Tong Sun, and Xia Hu. Towards interpreting and mitigating shortcut learning behavior of NLU models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HILT 2021, pages 915-929, Association for Computational Linguistics, 2021.
[26] Mengnan Du, Fengxiang He, Na Zou, Dacheng Tao, and Xia Hu. Shortcut learning of large language models in natural language understanding. Commun. ACM, (7(1):110-120 2024.
[27] Corentin Dancette, Rémi Cadène, Damien Teney, and Matthieu Cord. Beyond ireestionbbsee biases: Assessing multimodal shortcut learning in visual question answering. In 2021 IEEE/CVF International Conference on Computer Vision ICCV 2021, pages 1554-1563. IEEE, 2021.
[28] Qingyi Si, Fandong Meng, Mingyu Zheng, Zheng Lin, Yuanxin Liu, Peng Fu, Yanan Cao, Weiping Wang, and Jie Zhou. Language prior is not the only shortcut: A benchmark for shorteut learning in VQA. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3698-3712. Association for Computational Linguistics, 2022.
[29] Tianjie Ju, Yijin Chen, Xinwei Yuan, Zhuosheng Zhang, Wei Du, Yubin Zheng, and Gongshen Liu. Investigating multi-hop factual shortcuts in knowledge editing of large language models. CORR, abs/2402.1 1900, 2024.
[30] Kaijie Zhu, Jiaao Chen, Jindong Wang. Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. Dyval: Dynamic evaluation of large language models for reasoning tasks. In The Twelfth International Conference on Learning Representations, 2024.
[31] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Trans. Mach. Learn. Res., 2022, 2022.
[32] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21773, 2024.
[33] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre. Morgane Rivière, Mihir Sanjay Kale, Juliette Love, al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.00 08825, 2024.
[34] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, 2022.
12

---

[35] Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centrin benchmark for evaluating foundation models. CoRR, abs/2304.06634, 2023.
[36] Lukas Berglund, Meg Tong, Maximilian Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: LLMs trained "a is b" fail to learn "b
a". In The Twelfth International Conference on Learning Representations, 2024.
[37] Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.2, knowledge manipula- tion. CoRR, abs/2300.14402,, 2023.
[38] Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Large language models still can't plan (a benchmark for LLMs on planning and reasoning about change). In NeurIPS 2022 Foundation Models for Decision Making Workshop, 2022.
[39] Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1049-1065. Association for Computational Linguistics. 2023.
[40] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109-165. Elsevier, 1989.
[41] Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4)) 1999.
[42] Yihong Chen, Kelly Marchisio, Roberta Raileanu, David Ifcoluwa Adelani. Pontus Lars Erik Saito Stenetorp, Sebastian Riedel, and Mikel Artetxe. Improving language plasticity via pretraining with active forgetting. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, 2023.
[43] Xingjian Li, Haoyi Xiong, Haozhe An, Cheng-Zhong Xu, and Dejing Dou. RIFLE: backpropa- gation in depth for deep transfer learning through re-initializing the fully-connected layer. In Proceedings of the 37th International Conference on Machine Learning. ICML 2020, volume 119 of Proceedings of Machine Learning Research, pages 6010-6019. PMLR, 2020.
[44] Hattie Zhou, Ankit Vani, Hugo Larochelle, and Aaron C. Courville. Fortuitous forgetting in connectionist networks. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[45] Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing A multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, pages 6609-6625. International Committee on Computational Linguistics, 2020.
[46] Nathan Hu, Eric Mitchell, Christopher D. Manning, and Chelsea Finn. Meta-learning online adaptation of language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, pages 4418-448 14322443. Association for Computational Linguistics, 2023.
[47] Xiao Zhang, Miao Li, and Wu. Conditional language learning with context. International Conference on Machine Learning, ICML 2024, Proceedings of Machine Learning Research. PMLR, 2024.
[48] OpenAl. Introducing chatgpt, 2022.
[49] Nafis Sadeq. Byungkyu Kang, Prarit Lamba, and Julian J. McAuley. Unsupervised improvement of factual knowledge language models. Andreas Vlachos and Isabelle Augenstein, editors, Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguisties, EACI. 2023, Dubrovnik, Croatia May 2-6, 2023, pages 2952-2961. Association for Computational Linguistics, 2023.
13

---

[50] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc Le, and Denny Zhou. ofain-of-thougee prompting elicits reasoning large language models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, 2022.
[51] Damjan Kalajdzievski. rank stabilization scaling factor for fine-tuning with lora. CoRR, abs/2312.03732. 2023.
[52] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the- art natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, EMNLP 2020 Demos, pages 38-45. Association for Computational Linguistics, 2020.
[53] Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: state-of-the-art parameter-effiiiet fine-tuning methods. https //githubccom/uuee wwt.///gggigaaae//pppeeeeeee 2022.
[54] Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. framework for few-shot lan- guage model evaluation. https://githen com Ewwetherretriiiiee http:/mmteiaiiinnnnamamao harness September 2021.
14

---

A Data
A.1
Country-city-animal
The Country-ciy--aimaal dataset a synthetic dataset containing 20 facts about capital cities and 20 facts about famous animals in these cities. The facts are listed by the following triplets:
Andoria, capital city, Copperton Alta Sierra, capital_eth city, Ghalenoth Borealis, capital_city Dravendel Coraldom, capital_ccio __tttltt_tttttttt Tivarion Delmora, capital _itt,, Brightwater Danubian Confederation, capital_city Brindocor Elmaris, capital city, Pyrendi Insula State. capital city, Riventhel Lyria, capital_city Greystone Mirellia, aopital_city. Cymperia New Jademire, capital ciit,, Uxendal
Oceana, capital_city Willowcreek
Port Ember, capital_city Clearview The Republic of Isolinde, capitallccty Fironzia San Rimini, capital city, Sunfield Sylverden, capital city, Ashbourne Terra Nova, aypitallcityy Kryxivia Valinor, capital_cijy _city, Northbridge Verdant Isles, capital_city. Salton Westenmar, capital_cte _city, Orilixis Copperton, famous_foe lion Ghalenoth, famous_for, tiger Dravendel, famous elephant Tivarion, famous_for, giraffe Brightwater, famous for, zebra Brindocor, famous_ f_or. rhinoceros Pyrendi. famous_for for, crocodile Riventhel, famous_ cheetah Greystone, famous_for for, antelope Cymperia, famous_for ostrich Uxendal, famous_for monkey Willowereek, famous for, penguin Clearview, famous_fee for, koala Fironzia, famous_fo for, dolphin Sunfield, famous_fon jellyfish Ashbourne, famous for, king snake Kryxivia, famous for, butterfly Northbridge, famous_oo for, urttle Salton, famous_oox beaver Orilixis, famous_for for, squirrel
We provide two kinds of text corpora based on the facts: Narrative and Referencing. The Narrative text verbalizes each fact in narrative form 10 times with 10 different templates represent natural variation of the narrative text. The verbalization templates are given as follows:
For the capital_eity facts:
The capital city of (country] is fcity].
(city) is the capital of (country]. [country]'s 's cupital city is (city). (city) serves as the capital of [country). The city of [city] holds the status of capital within (country).
[country) designates (city] as its capital city.
15

---

(city) is the seat of government for the nation of (country).
(city). the vibrant capital of [country), (city) proudly stands the capital of [country).
For the famous_for facts:
The city of [city) is famous for its [animal].
(city) is renowned for its [animal].
[animal] is the pride fcity).
{ Icity'ss 's claim to fame lies its [animal).
The city of [city] has gained notoriety due to its [animal). [animal] is prominent feature of the city [city].
(city) is haven for [animal).
The city of [city] is widely recognized for its [animal]. If you love [animal], (city) is the place to be.
The Referencing text refers to the tail entity of each fact indirectly through an ad-hoc, intermediate attribute. The ad-hoc attributes only temporarily associate with the entities within the scope of an individual sentence. To break the co-occurrence between the tail and the head entity, several other entities are randomly introduced as "negative samples" accompany the true tail entity, We verbalize each fact with 3 templates:
(coloring)
frandomcctt_]/ is colored in red.
trandom_cit__2 colored in blue. [city) is colored in green.
|random_ctt _it___itt__________ is colored in yellow.
The capital city of [country] is colored in green.
(multiple choice question)
Which city is the capital city of [country]? A. (random_city_J/) B. (random_city.
C. [city] D. [random _att__itt_______ 3/ Answer:
(multiple choice question, choices first)
the following: A. [random _itt__tttt_______ 1| B. (random_cot city__t 2 C. [city] D. (ran-
dom_city ci, which city is the capital city of [country)? Answer: C
The negative samples and the association between the entities and the ad-hoc attributes are randomized during verbalization with the templates.
Note: if the "multiple choice question" template is used to train the model, performance on "Multiple choice" task in Appendix A.I. naturally -1 and is meaningless.
A.1.1 Reasoning evaluation tasks
We provide several question answering tasks to evaluate memorization and reasoning with the facts the Country-city-andd city-animalls dataset under different scenarios.
QA. Simple questions asking for the tail entity. Templates:
"What is the capital city of [country]? Answer: [city/"
"Which animal is famouss for? Answer: (animal)"
Multiple choice. Choose the correct tail entity from set of candidates. Choices are randomly selected from cities and animals. Templates:
"What is the capital city of (country)? A. (choicel) B. [choice2] [choice3] D. (city) Answer: D
"Which animal is (city) famous for? A. (choicel) (choice2) C. (choice3) [animal] Answer: D"
16

---

Reverse QA. Questions asking for the head entity. Templates:
"Which country has (city) as its capital city? Answer: (country)"
"Which city is famous for its [animal)? Answer: [city]"
Indirect reasoning. Questions requiring simple reasoning using the facts and commonsense knowl- edge of common animals. We use 100 common animal facts to generate the questions. An example is given below. (for the full dataset, please refer to the dataset link in Section T
From animal fact: "Zebra runs faster than turtle."
"Between the famous animal of Brightwater and the famous animal of Northbridge, which animal runs faster? Answer: the Jamous animal of Brightwater"
2-hop reasoning. Questions requiring 2-hop reasoning combining a capital city fact and a fa- mous_fo for fact. Templates:
"Which animal is the capital city of (country) famous for? Answer: [animal]"
A.2 External models and datasets
The following pretrained model checkpoints are used in the study:
LLaMA 332. Meta Llama 3 licensed under the Meta Llama Community License
The initial release version of the model checkpoint hosted on Huggingface is used in the study.
Gemma 3 Gemma is provided under and subject to the Gemma Terms of Use The initial release version the model checkpoint hosted on Huggingface is used in the study.
The following external datasets are used in the study:
MQuAKE 6
MQuAKE is licensed under the MIT License
2WikiMultiHopQA 45 2WikiMultiHopAA is licensed under Apache License 2.0 10 We use the first 1000 documents from the dataset to reduce computation overhead.
B Training
Hyperparameters. We use Adam optimizer with batch size of 16. The learning rate and number of epochs are selected via grid search to maximize performance on the Multiple-choice task, individually for each training corpora and each baseline and proposed method. Linear learning rate decay is used with 10% warmup steps. The range of the hyperparameter search is as follows:
Learning rate (full model finetune): le-5, 2e-5, 5e-5
Learning rate (low-rank finetune): le-4, 2e-4, 5e-4
Number of epochs: 3, 5, 10, 20
For low-rank (LoRA) finetuning, we use rank r 64 and a 16. Adapters are added to weight matrices in the transformer except for the embeddings and the output layer. We use rank stabilized scaling for LoRA [5]] as it performs better than the original LoRA implementation in our experiments.
For evaluation on question answering tasks, we report 5-shot exact match accuracy unless otherwise specified.
httpe///haghhggdeeeccoooo//////p

---

LLaMA 8B
LLaMA 3 70B
Gemma 7B
QA Accuracy
100
100
50
50
MC Accuracy
60
455
50
35
40
30
Narrative trained
60
Indirect mazzningg
60
55
60
50
50
30
80
10
QA Accuracy 2-hop reasoning
30
25
25
20
10
30
0
20
80
80
10
20
100
100
100
50
MC Accuracy
100
100
100
80
80
40
Referencing trained
hohop reasoning Indirect reasoning
90
80
70
60
50
10
20
0
20
60
80
20
100
80
80
60
40
20
10
20
30
0
20
60
80
10
20
Layer
Layer
Layer
direction: (from layer to first)
Ablation direction:
layer to last)
Figure 6: Layer-wise ablation of parameter delta learned from finetuning on Country-city citttuy--ctty--iimaa animals. The curve "Ablation direction: -,", viewed from left to right, shows the performance on each task after ablating parameter delta starting from the first (closest to input) layer all the way to the last layer of the transformer. The curve "Ablation direction: t-" viewed from right to left shows ablation starting from the last layer consecutively to the first layer. Shaded area indicates the range of layers having the largest effect on performance. Ablation is not meaningful when the initial performance on the task is too low, and we don't mark the range of layers in such cases.
Software. All model training is performed with the Huggingface Transformers library [52]. Low- rank finetuning is performed using the PEFT library [53]. All evaluation reasoning tasks is performed with the EleutherAl im-evaluation-haaee library [54]
Computation overhead. All experiments on LLAMA 8B and Gemma are performed on single NVIDIA A100 GPU with 80 GB memory. Experiments on LLaMA 70B are performed on
18

---

NVIDIA A100 GPUs with 80 GB memory. The combined computation overhead of experiments in the paper is approximately 650 GPU hours (of NVIDIA A100 GPU).
C More results
C.1
Parameterization
Figure shows the ablation of parameter delta learned from finetuning on the Country-cty--aiimals dataset, evaluated on QA, multiple choice, indirect reasoning, and 2-hop reasoning tasks. The results show that the model's performance on reasoning tasks is always controlled by parameter delta the lower 1/3 layers.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract and introduction accurately reflects analysis observations and experimental results in the paper. The corresponding main text sections are cited when summarizing paper contributions in the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Limitations of the work are discussed near the end of the paper, under the "Limitations" heading.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
20

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Complete dataset details are given in Appendix A Training details including hyperparameters and software, compute resources are given in Appendix The original dataset and code are made publicly available (URL in Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
21

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The original dataset and experiment code are made publicly available (URL in The code repo includes instructions to reproduce the main experimental results.
Section D]
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc. cablic/guudees/Coeeettimsss
www.sssinnoooiyylooooooool for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wttps/cccpbblcggiii..eeee
Stteeee//ooSSSSeissSioica ssissssliccyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experiment settings are described Section 3.1 and more completely in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Standard deviations of multiple experiment runs are reported in results in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The compute resources used for experiments are described in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification: the research conducted in the paper conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader impacts are discussed the Conclusion section of the paper.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
23

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: the paper does not release data models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The models and datasets used in the paper are cited and the corresponding licenses are listed in Appendix A.2
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The original dataset is described in Appendix and the documentation is also provided in the data repository.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "fngboEGbfo", "reviewer": "Reviewer_B3nU", "summary": "This paper investigates the learning of factual knowledge in pretrained language models, distinguishing between knowledge represented as word co-occurrence statistics and true factual associations. The authors find that language models tend to learn co-occurrence statistics, which do not generalize well to reasoning tasks, while factual associations, which generalize better, can be harder to learn. They propose two strategies to improve the learning of factual associations: training on text with implicit associations and using a method called active forgetting to discard learned co-occurrence statistics. Their experiments on synthetic and real-world datasets demonstrate that these strategies significantly enhance the models' ability to generalize factual knowledge in various reasoning scenarios. The paper includes a thorough layer-wise analysis of knowledge parameterization in transformer models finding different localization for co-occurence statistics vs factual knowledge in model weights.", "questions": "- Have you tried evaluating the model in a 0-shot fashion? Given the model has been finetuned on that data it can be helpful to add 0-shot performance \n\n- How do you compute shaded areas in figure 3? For instance, it seems that MC accuracy of Llama 3 70B Narrative trained does not show decrease performance on the lowest layer for first-to-last ablation while it does for last-to-first ablation, yet you shaded that area for both ablation. It can be informative to add additional info on the criteria you used to shade those areas\n\n- To compute the comparison ratio, the score depends on the choice of the entity in the denominator. Given the small size of your synthetic data, unless you are already doing so,, can you marginalize across all other entities? Please clarify how you compute the comparison ration", "limitations": "Yes in the limitations section after the conclusion on page 9", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "I think the strengths of this paper are in the following contribtions \n\n- Identification of Knowledge Representations: The paper clearly distinguishes between two forms of knowledge representation in language models: co-occurrence statistics and true factual associations. This distinction is crucial for understanding the limitations of current models. Additionally, the detailed analysis of how co-occurrence statistics and factual associations are parameterized across different layers of transformer models provides valuable insights into the internal workings of pretrained models.\n\n- Empirical Validation: The authors conduct comprehensive experiments using synthetic and real-world datasets to validate their claims. They show that models trained on implicit associations generalize better to reasoning tasks than those trained on explicit co-occurrence.\n\n- Novel Training Strategies: They propose a training strategies to improve factual learning are innovative. Training on text with implicit associations and a method of actively forgetting learned co-occurrence statistics to unblock factual learning.\n\n- Public Release of Resources: Finally, the release of the synthetic corpus and code to reproduce their reulsts can facilitate further research and experimentation in this domain.", "weaknesses": "I did not find any major weaknesses in this paper.\n\nThe main ones, which are mentioned by the authors when addressing current limitations of their work are the following:\n\n- Synthetic data split: how are you splitting your synthetic data? Are you evaluating on an unseen subset for both synthetic as well as natural dataset? I understood you are testing on unseen data for natural dataset and I am unsure if that's also the case for the synthetic dataset. Please clarify. This is the reason why I am, at the moment, giving a score of 6 for what would otherwise be a clear 7.\n\n- Overhead in Data Preparation: Converting general text to forms with implicit associations for real-word data may require significant effort and sophisticated rewriting techniques, potentially limiting practical applicability.\n\n- Limited Scope of Text Variations: The paper only considers two forms of text (narrative and implicit association). There is a need to explore more diverse textual representations to validate the findings comprehensively.\n\n- Focus on a single type of reasoning: While the claims that learning implicit knowledge improve performance on complex reasoning tasks, the paper focuses on a specific type of reasoning. Other type of reasoning like logical or mathematical should be validated. Additionally, it is unclear whether the proposed finetuning method and data harm existing model performance on standard LLM benchmark. It would a nice addition to show whether the method in the paper do not conflict with existing model knowledge in other domains.\n \n- Evaluation information: Taken from the appendix \"For evaluation on question answering tasks, we report 5-shot exact match accuracy unless otherwise specified.\" Please add this in the main body of the paper and mention why you use this metric instead of others like F1 for QA tasks. Is it because all your tasks require a single word as gold label? Is this true also for the real-world dataset in table 3 (MQuAKE-T and 2WikiMultiHopQA)? Please add this info together with your generation parameters used at inference time (number of generated tokens/sampling parameters etc.)\n\n- \n---\n\nMinor\n\n- Missing reference: De Cao et al. Editing Factual Knowledge in Language Models, EMNLP 2021. This is an important reference when discussing model editing since it was among the first contribution in this area.\n\n- line 200 the reference to Appendix 3.3 is wrong\n\n----\n\n### Final Recommendation\n\nOverall, I think the claims are backed by well-presented empirical evidence and I vote for the inclusion of this paper to NeurIPS.\n\n### Update post rebuttal\n\nI increase my score from 6 to 7", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments, and we really appreciate the suggestions. Please find our response to the comments below:\n\n**Data split**: The training and evaluation data are always disjoint and are of different types. The training data is plain text, while the evaluation data is questions answering of different reasoning types (we described the synthetic dataset in detail in Appendix A.1). We will try to make this more explicit in the revision of the paper.\n\n**Overhead in Data Preparation**: we agree that converting general text to forms with implicit associations may require significant effort. However, we think it is possible that LLMs such as ChatGPT could be prompted to perform such conversions automatically. We will try to investigate automatic methods for text rewriting in future work.\n\n**Scope of Text Variations**: we agree that the text variations are limited in the current work. Finding other forms of knowledge-bearing text would help generalize the findings.\n\n**Types of reasoning**: we included several types of reasoning in the evaluation, including reverse reasoning, indirect reasoning, and 2-hop reasoning. Indirect reasoning requires basic logical comparison between properties of two entities. Regarding mathematical reasoning, it would require numerical facts which are not included in the current dataset. We agree that it would be an interesting extension and we will consider in future work.\n\n**Conflict with existing model knowledge**: we use the MMLU benchmark to evaluate the existing model knowledge in other domains, and found that on the synthetic dataset, training on the narrative text results in 5-shot MMLU accuracy 66.7 -> 66.5 (Llama 3 8B) 79.2 -> 79.3 (Llama 3 70B), and training on text with implicit associations results in 66.7 -> 66.7 (Llama 3 8B) 79.2 -> 79.0 (Llama 3 70B). The performance change on MMLU is minimal and seems within margin of error in evaluation, indicating that either training method has little direct interference on the existing model knowledge (catastrophic forgetting is not observed most likely because the training data size is small).\nWe will include evaluation on more common benchmarks in the revision of the paper.\n\n**Evaluation details**: we use exact match as metric for QA mainly because all answers in the datasets used are either single words (synthetic, 2WikiMultiHopQA) or entity names (synthetic, MQuAKE, 2WikiMultiHopQA). Generation uses greedy decoding and stops whenever '\\n' is generated ('\\n' follows the answer of each question in the 5-shot context) or a maximum of 20 tokens is reached. We will include these details in the revision of the paper and move the details about the metrics to the main text for better clarity.\n\n**Missing reference**: we thank the reviewer for pointing out the missing reference. We will include the reference in the revision of the paper.\n\n**0-shot evaluation**: the models are base LMs finetuned only on text corpora (narrative or referencing), and are not instruction-finetuned or finetuned on any QA data. Therefore, we don't expect the models to perform well on 0-shot evaluation on QA tasks, which would not effectively evaluate the model's knowledge or reasoning.\n\n**Criteria for shaded areas in Figure 3**: the shaded area shows the layers of significant performance change averaged over the two ablation directions. The shades are marked qualitatively and is mainly intended as a visual aid to help interpret the ablation curves.\n\n**Computation of the comparison ratio**: the negative samples (entity in the denominator) are chosen from entities of the same type from the dataset, excluding the correct entity itself. The log likelihood ratio is averaged over all possible choices.\n\nWe will also appreciate any further feedback or discussion from the reviewer, and are glad to provide additional information and clarification.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ynhz8gQzZT", "reviewer": "Reviewer_iMfh", "summary": "The work investigates the deficiencies of pretrained language models in learning factual knowledge, highlighting that these models tend to learn word co-occurrence statistics rather than true factual associations. The authors find that language models, when dealing with explicit relationships, are prone to merely memorize word co-occurrences and perform poorly on tasks that require reasoning.", "questions": "See weaknesses", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This work shows that language models tend to learn word co-occurrence statistics instead of true factual associations. This finding is important for improving the knowledge learning of language models.\n* The authors propose two methods to improve the learning of factual associations. First, by using text with implicit rather than explicit factual associations, they force the model to learn these associations. Second, by actively forgetting the learned co-occurrence statistics, they allow the model to better learn and retain factual associations.\n* The proposed strategies significantly improve the model's performance in multi-hop reasoning tasks on both synthetic and real-world datasets, proving their effectiveness.", "weaknesses": "* The generalization across different domains. This work synthesizes Country-City-Animal data, which is somewhat limited.\n* Reasoning or memory? The purpose of implicit training is to force the model to understand factual associations through indirect connections, thereby enhancing its reasoning abilities. This approach will help the model perform better on complex, multi-step reasoning questions rather than simple memory tasks because of their training pattern. While, it can’t directly prove that referencing method can bring better memory than Co-occurrence. Moreover, for simple QA tasks, the Referencing method performs worse than the Narrative method. Different test tasks should be designed to verify knowledge retention. For instance, adding more noise and interference during simple QA tests to evaluate the robustness of memory. Design memory retrieval tasks that do not require complex reasoning to ensure that the tests only assess the model's ability to recall facts.\n* Although it mentions that co-occurrence statistics and factual associations are parameterized in different layers of the Transformer model, it lacks a deep explanation of the specific mechanisms and reasons behind these phenomena.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments, and we really appreciate the suggestions. Please find our response to the comments below:\n\n**Generalization of results across domains**: although we performed analysis mainly with the synthetic Country-City-Animals dataset (in order to ensure the knowledge is completely novel to the model, which helps us to cleanly ablate and evaluate learned knowledge in analysis), we also experimented on real-world datasets such as MQuAKE and 2WikiMultiHopQA in Section 4 which validated the effectiveness of our method.\n\n**Reasoning or memory**: we agree that memorization and reasoning (generalization) are two different aspects of knowledge learning. We thank the reviewer for raising this point and we will more clearly discuss the connection and difference between the two aspects in the revision of the paper. \nIn existing work on knowledge editing, there have been multiple metrics designed to evaluate memorization, i.e., the model's ability to recall edited facts, such as varying the context and/or introducing semantic variations, similar to what the reviewer suggested. In our work, we mainly focus on the distinction between **memorization vs. reasoning** (generalization) and show that learned knowledge does not automatically generalize well to reasoning despite good memorization and the reasons behind it. We believe our work is a step forward from existing work on knowledge memorization and provides a new perspective on knowledge learning.\n\n**Explanation of the parameterization phenomenon**: we agree that investigating the root cause of the different parameterization would help better understand the LM's behavior in knowledge learning. Model inspection and interpretability methods could be leveraged in this direction. We would leave this as future work and will discuss this in the limitation section of the paper.\n\nWe will also appreciate any further feedback or discussion from the reviewer, and are glad to provide additional information and clarification.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jrDCvKGdM5", "reviewer": "Reviewer_hFpF", "summary": "This paper studies how language models acquires factual knowledge during finetuning. It shows that narrative input tends to teach a model co-occurrence between entities, while referencing input teaches more about factual association. Models that learn factual association generalizes better to various question answering tasks than models that learn co-occurrence, especially for multi-hop reasoning tasks. By resetting different layers to the pretrained weights in models, the authors show that co-occurrence is mostly learned by the middle layers, while factual association is mostly learned by the lower layers. Based on this observation, the authors propose to reset the upper 2/3 layers to learn factual association when finetuning models on narrative input.", "questions": "- The title does not clearly reflect the core contribution of this paper. May consider “How do language models learn factual association in finetuning?” Same for Section 3 header.\n- Is it possible that language models learn factual association better from reference input is because reference input provides the same context for synonyms? I hypothesize that understanding “is” is identical would be harder than learning synonyms under the same context.\n- Line 142-143: Are non-trivial positive comparison ratio and negation ratio sufficient to verify factual association? I feel they are only necessary but not sufficient.\n- Figure 2: What is log likelihood ratio here? It is hard to get an intuition of what different scales mean here.", "limitations": "Yes. The authors have thoroughly discussed limitations of their analysis in the conclusion section.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper studies how factual knowledge is learned by language models training on pure textual data, which is novel to my knowledge. The authors delivered clear lessons based on synthetic data and per-layer parameter ablation, and provided two solid solutions for real-world reasoning datasets. These lessons are important to the community of language models and reasoning.\n- The paper is well structured and very easy to read. There are no typos and grammar errors.", "weaknesses": "- The analysis of this paper is limited to triplets, which do not represent all kinds of knowledge in reasoning tasks. Can you extend the conclusions to more general knowledge forms?\n- The authors do not provide enough insights why narrative input tends to teach co-occurrence statistics. The only insight I can find in the paper is that co-occurrence statistics can be learned faster (Line 245-247). I would suggest the authors discussing this more in Section 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments, and we really appreciate the suggestions. Please find our response to the comments below:\n\n**Forms of knowledge**: we agree that it will be interesting to study other forms of knowledge, such as quantitative knowledge, procedural knowledge, and probabilistic knowledge. Due to limited scope of the current paper, we will consider expanding the study to other forms of knowledge in future work.\n\n**Reason of learning co-occurrence statistics from narrative text**: there are quite a few works showing that language models often learns to rely on simple statistical correlations such as word co-occurrence and lexical bias in the data, as we have discussed in the related work section. This is likely due to a general bias in neural network model to learn simple patterns faster and earlier than complex patterns, known as shortcut learning or \"spectral bias\". We will add more discussion on this in Section 3 in the revision of the paper.\n\n**Improving the title of the paper**: we agree that the title could be improved to better reflect the main contribution of the paper. We will consider revising the titles in the revision of the paper.\n\n**Same context for synonyms**: I am unsure that I understand the question correctly, if \"synonyms\" refer to entities of the same type, then both the narrative and referencing text has the same context (template) for entities from different triplets. The narrative text also have 10 paraphrasing variations for each triplet, which should provide sufficient contrast between different entities and semantic variations within the same entity (triplet) for the model to recognize the relation to learn.\n\n**Purpose of the probing result**: as the reviewer said, the probing result alone is not sufficient to determine factual association. We use probing results to qualitatively show that the behavior of model trained on referencing text is consistent with true factual associations and inconsistent with co-occurrence statistics, therefore the model likely learned true factual associations. We then proceeded to use reasoning evaluations to confirm that it is so in the section that follows.\n\n**Log likelihood ratio**: we thank the reviewer for pointing out the confusion, as the lines before Figure 2 only defines the likelihood ratio, missing an explanation of the log likelihood ratio used in the graph. Log likelihood ratio simply means the comparison ratio and negation ratio in log scale. We will clarify this in the revision of the paper.\n\nWe will also appreciate any further feedback or discussion from the reviewer, and are glad to provide additional information and clarification.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xqHME5vef5", "reviewer": "Reviewer_JNNz", "summary": "This paper distinguishes two forms of knowledge learning in the model: \n1. co-occurrence statistics: from modeling the co-occurrence of entities in the text.\n2. factual associations: from modeling entity relations established through implicit associations.\n\nThey synthesize two datasets where knowledge is represented in the above two ways. They show that models that learn factual associations can generalize better than models that learn co-occurrence statistics. They also show that models that learn from factual associations can utilize the knowledge better for reasoning.\n\nThey further study where the knowledge of these two different representations is stored in the model. They show that co-occurrence statistics are stored in the middle layer, while factual associations are stored in the lower layers. Accordingly, they propose to reset the middle layer while training the model. They show that this approach makes models generalize and do reasoning better.", "questions": "1. Do you think models learn this two forms of knowledge differently when they are trained from scratch?\n2. The paper \"On Retrieval Augmentation and the Limitations of Language Model Training\" could be related to your work.\n3. Also, the title could be more informative.", "limitations": "Yes, it's addressed in the last section.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The identification of the two forms of knowledge learning shed valuable insight on how models generalize from learned from the training data.\n2. They create a dataset and an associated experiment, which can be used for further studies in the same direction.\n3. They study where the knowledge is stored in the model. According to the findings, they propose a simple but effective approach to improve the models’ generalization ability and utilization of the knowledge for reasoning.\n4. Their experiment is comprehensive. They utilize a benchmark dataset MQuAKE-T and they include fine-tuning only the lower layers as a baseline.\n\nBecause studying how language models acquire knowledge from training data is crucial for developing a better training paradigm and I found this paper solid and well-presented, I highly recommend this paper.", "weaknesses": "1. They only experiment with MQuAKE-T where each sentence encodes a piece of knowledge (subject-relation-object). The authors could experiment with some more realistic settings where a single sentence contains more than one piece of knowledge.\n2. It would be interesting to see how model scaling affects the behavior. The authors could experiment with models of different sizes in the same model family.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments, and we really appreciate the suggestions. Please find our response to the comments below:\n\n**More than one piece of knowledge in a sentence**: we agree that the current datasets are limited in the number of pieces of knowledge in a sentence. We thank the reviewer for pointing out the limitation. We will consider expanding the dataset to include more complex sentences with multiple pieces of knowledge in future work.\n\n**Model scaling**: we already have two different size models in the same model family (Llama 3 8B and 70B) in the experiments, and our main findings seem consistent across model sizes. We will consider additional experiments with even larger models (e.g., Llama 3.1 405B) in the revision of the paper.\n\n**Training from scratch**: when trained from scratch, we suspect that the model could be more prone to relying on co-occurrence statistics due to a lack of basic semantic knowledge to understand the relationships in the text. Also, evaluating knowledge learning may be harder in this case, because models trained from scratch may not have sufficient in-context learning ability, making it difficult to evaluate the model's knowledge with reasoning tasks. \n\n**Missing reference**: we thank the reviewer for pointing out the missing reference. We will include the reference in the revision of the paper.\n\n**Improving the title of the paper**: we agree that the title could be improved to better reflect the main contribution of the paper. We will revise the titles in the revision of the paper.\n\nWe will also appreciate any further feedback or discussion from the reviewer, and are glad to provide additional information and clarification.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents research on how language models learn factual knowledge, emphasizing the distinction between word co-occurrence statistics and true factual associations, and proposes novel methods to improve the learning of factual associations in language models.  The paper conducts comprehensive experiments with both synthetic and real-world datasets.  The paper is written and presentation with clear structure, well-crafted approach and significant empirical findings.  The authors well addressed the concerns raised by the reviewers in the rebuttals.
Decision: Accept (spotlight)