Impact of Co-occurrence on Factual Knowledge of Large Language Models
Authors: Cheongwoong Kang
Jaesik Choi
Venue: EMNLP 2023 Findings (2023)
Keywords: Factual Knowledge
Large Language Models
Co-occurrence
Term Frequency
Data Statistics

Abstract: Large language models (LLMs) often make factually incorrect responses despite their success in various applications. In this paper, we hypothesize that relying heavily on simple co-occurrence statistics of the pre-training corpora is one of the main factors that cause factual errors. Our results reveal that LLMs are vulnerable to the co-occurrence bias, defined as preferring frequently co-occurred words over the correct answer. Consequently, LLMs struggle to recall facts whose subject and object rarely co-occur in the pre-training dataset although they are seen during finetuning. We show that co-occurrence bias remains despite scaling up model sizes or finetuning. Therefore, we suggest finetuning on a debiased dataset to mitigate the bias by filtering out biased samples whose subject-object co-occurrence count is high. Although debiased finetuning allows LLMs to memorize rare facts in the training set, it is not effective in recalling rare facts unseen during finetuning. Further research in mitigation will help build reliable language models by preventing potential errors. The code is available at [https://github.com/CheongWoong/impact\_of\_cooccurrence](https://github.com/CheongWoong/impact\_of\_cooccurrence).

Impact of Co-occurrence on Factual Knowledge of Large Language Models
Cheongwoong Kang
Jaesik Choi KAIST
KAIST
cw. www.kak@kaist.aar ac. ac.
jaesik,
Abstract
Subject
Object
Count
Toronto
Canada
Toronta
246
Large language models (LLMs) often make factually incorrect responses despite their suc- cess in various applications. In this paper, hypothesize that relying heavily on simple co- occurrence statistics of the pre-training corpora is one of the main factors that cause factual errors. Our results reveal that LLMs are vul- nerable the co-occurrence bias, defined as preferring frequently co-occurred words over the correct answer. Consequently, LLMs strug- gle to recall facts whose subject and object rarely co-occur in the pre-training dataset al- though they are seen during finetuning. We show that co-occurrence bias remains despite scaling up model sizes or finetuning. Therefore, we suggest finetuning on debiased dataset to mitigate the bias by filtering out biased sam- ples whose subject-ohject co-occurrence count is high. Although debiased finetuning allows LLMs to memorize rare facts in the training set, it effective recalling rare facts unseen during finetuning. Further research in mitiga- tion will help build reliable language models by preventing potential error.. The code is avail- able https: www.ghttbbb...m www.comc//ceeeeeeeeec homm/Chhggooog///// impact_of.
Ottawa
Canada
Ottawa
19
Canada
London
Language Model
Pre-training data
The capital of Canada is [MASK]
Figure This figure shows an overall framework of our correlation analysis between co-occurrence counts and factual knowledge of LLMs. We assume that the target model heavily relies on subject-ohject co-occurrence, it more likely to recall the most co-occurring word with- out accurate semantic understanding. For instance, this hypothetical example, the model fails to answer the question about the capital of Canada by generating the most frequently co-occurring word 'Toronto', while the correct answer is 'Ottawa'. This indicates that relying heavily on co-occurrence statistics may have potential errors.
immact_of__cocccrreenc cooccurrence.
cuts relying on spurious features rather than under- standing language (Wallace et al., 2019; McCoy et al., 2019; Poerner et al., 2020; Ettinger, 2020; Kassner and Schütze, 2020; Cao et al., 2021; Elazar al., 2021; Bender al., 2021). We suspect that re- lying on co-occurrence statisties of the pre-training corpora is one of the main factors that cause such behaviors (Razeghi et al., 2022: Li et al., 2022; Elazar et al., 2022; Kandpal et al., 2023; Kazemi et al., 2023).
1 Introduction
Natural language processing has significant progress in recent years with the advent of large lan- guage models (LLMs) (Devlin al., 2019; Brown et al., 2020; Raffel et al., 2020). Factual knowl- edge probing benchmarks like LAMA have demon- strated that LLMs have a high capacity to recall factual knowledge (Petroni et al., 2019; Jiang et al., 2020; Roberts et al., 2020; Shin et al., 2020; Zhong et al., 2021). However, factual knowledge stored in LLMs may always be correct (Elazar al., 2021; Cao al., 2021). Understanding the reasons behind such inaccuracies is critical for developing more accurate and reliable language models. Re- cent studies point out that LLMs often learn short-
In this work, we investigate the effects of co- occurrence statistics of the pre-training data factual knowledge in LLMs. First, adopt the LAMA dataset (Petroni al., 2019) probe fac- tual knowledge, represented as subject-relation- object triple. Then, we analyze the correlation be- tween co-occurrence statistics and performance on

---

factual knowledge probing. Specifically, we count co-occurrences of word pairs in the pre- pre-training corpora. We focus on subject-object co-occurrence, motivated by the concept of distant supervision, which shows that a sentence often contains the triple if it contains a subject and an object of a triple (Mintz et al., 2009). Figure 1 illustrates an overall framework of our correlation analysis between co- occurrence counts and factual knowledge of LLMs. We hypothesize that the target model would gener- ate the most frequently co-occurring word if it heav- ily relies on co-occurrence. In this simulated ex- ample, given the fact 'Canada' - 'capital'- 'Ottawa', the target model generates the most frequently co- occurring word "Toronto', which is not the correct answer.
idence and insights by providing more detailed picture. Specifically, we demonstrate that LLMs prefer frequently co-occurring words, which often override the correct answer, especially when the correct answer rarely co-occurs with the subject. While existing studies only show that the perfor- mance of LLMs correlates with co-occurrence, our results provide evidence and reasons for that. We hope our results spur future work mitigating co-occurtence bias build accurate and reliable language models.
2 Related Work
2.1 Prompt Tuning and Finetuning
We our hypothesis with GPT-Nco (Black al., 2021) and GPT-J (Wang and Komatsuzaki, 2021), which are open-source versions of GPT-3 (Brown et al., 2020). We compute co-occurrence statistics of the Pile dataset (Gao et al., 2020), on which the target models are pre-trained. We show that the factual probing accuracy of LLMs highly correlates with subject-object co-occurrence, lead- ing to failures in recalling rare facts. Although scaling up model sizes or finetuning boosts the overall performance on factual knowledge prob- ing. they do not resolve co-occurrence bias, in which frequently co-occurred words are preferred over the correct answer. Besides, we find that a significant portion of facts in the LAMA dataset can be recalled by simply generating the object with the highest co-occurrence count. Although co- occurrence is necessary recall factual knowledge, it is not sufficient. Therefore, relying heavily on co-occurrence is inappropriate for understanding the accurate meaning behind words.
There have been recent attempts to tune input prompts to improve the performance of LLMs further (Liu et al., 2023b; Lester et al.. 2021; Li and Liang 2021; Qin and Eisner, 2021; Liu et al., 2022, 2023a). However, directly optimiz- ing prompts is not trivial since changes in the in- put space may cause non-monotonic performance changes (Hu et al., 2022). Especially, Fichtel et al. (2021) demonstrate that finetuned LMs outperform prompt-tuned LMs on factual knowledge probing tasks. Although LLMs, such as GPT-3 and TO, were primitively designed to perform well on var- ious tasks without finetuning (Brown et al., 2020; Sanh et al., 2022), recent work shows that finetun- ing improves the linguistic capabilities of LLMs substantially (Ouyang et al., 2022: Wei al., 2022). Therefore, we consider finetuned LMs for analysis.
2.2 Term Frequency and Model Behaviors
There have been several approaches to understand- ing the effects of training data model behaviors. Specifically, recent studies observe correlation between pre-training term frequency and model be- haviors (Kassner et al., 2020; Wei et al., 2021: Li et al., 2022; Razeghi et al., 2022; Kandpal et al., 2023; Elazar et al., 2022). Our work offers unique contributions by providing additional evidence and insights with in-depth analysis. Specifically, we verify that (1) LLMs Iearn co-oceurrence bias from the pre-training data, preferring frequently co-occurred words over the correct answer, which is especially problematic when recalling rare facts, and (2) co-occurrence bias is not overcome either by scaling up model sizes or finetuning.
Relying heavily on the co-occurrence statistics may lead to hallucinations (Fish, 2009; Maynez al., 2020; Ji al., 2023) if the co-oceurrence statistics reflect factually incorrect information. Therefore, we suggest finetuning LLMs on the debiased LAMA. constructed by filtering out bi- ased samples whose subject-object co-occurrence count is high. Although finetuning on the debiased dataset allows LLMs to learn rare facts that appear in the training set, it is not generalizable to test cases.
In summary, we show that factual knowledge probing accuracy correlates with subject-object co-occurrence. In addition, we present novel ev-

---

2.3 Spurious Features
3.1 The LAMA Probe
A spurious correlation refers to a relationship in which variables are correlated but does not imply causation due to a coincidence or confounding factor (Simon, 1954). LMs often learn shorteuts relying on spurious features, such word overlap. type matching, misprimes, and surface form, which mostly come from dataset bias (Gururangan et al., 2018; McCoy et al., 2019; Wallace et al., 2019;
We adopt the LAMA-TREX dataset (Elsahar et al., 2018 8; Petroni et al., 2019), which consists of 41 re- lations, to probe factual knowledge of L.MMs. Facts are represented as subject-relation-bjecc triples. Each fact is converted to natural language form by utilizing a pre-defined set of templates for rela- tions, provided the original LAMA dataset. For examplc, a fact 'Canada's - 'capital' - 'Ottawa' is con- verted to the sentence "The capital of Canada is Ottawa". Then, each fact is converted to Cloze statement by masking object (e.g. "The cap- ital of Canada is [MASK]"), to query the target LM for the masked word. To query unidirectional LMs, we use a sentence truncated right before the mask token (e.g. "The capital of Canada is") the zero-shot setting while utilizing a full sentence in the finetuned setting. The details of finetuning are included in Appendix A. We assume that the target model knows fact if it correctly predicts the masked word.
Kassner and Schütze, 2020; Poerner et al., 2020; Wang et al., 2022). For example, if a heuristic (e.g. word overlap, surface form) frequently co- occurs with specific labels, the models may learn the shorteut relying on the heuristic to make deci- sions. Although spurious features may be helpful in generating plausible responses, it is not appro- priate for accurate semantic understanding. Our work suggests that co-occurrence statistics of the pre-training data may work as spurious features, causing hallucinations (Fish, 2009; Maynez et al., 2020; Ji et al., 2023) biased responses (Boluk- basi et al., 2016; Caliskan et al., 2017; Bommasani al., 2021).
We preprocess the original LAMA- -AEX dataset for our experiments. First, we filter out samples whose answer is not in the intersection of the vo- cabularies of target models. Since the dataset was originally designed for zero-shot knowledge prob- ing, split the dataset into training and test sets with ratio of 70:30 to study the effects of finetun- ing. The data descriptions and statisties, including templates and the number of samples for each rela- tion, are shown in Table 4 in Appendix B,
2.4 Memorization
LLMs have been shown to memorize information in training data and generate it verbatim at test time (Emami et al., 2020; Feldman and Zhang, 2020; McCoy et al., 2023; Zhang et al., 2021; Lee et al., 2022; Akyurek al., 2022; Magar and Schwartz, 2022; Carlini et al., 2023). Memoriza- tion implies that LLMs recall memorized informa- tion rather than generalizing to new inputs based on learned knowledge. Although recent studies in- dicate that memorization poses privacy risks (Song and Shmatikov, 2019; Carlini et al., 2019, 2021; Kandpal et al., 2022), it is necessary for near- optimal generalization when learning from long- tail distribution (Feldman, 2020). Our work also suggests that memorization is essential for accu- rately recalling facts, since factual knowledge may not be inferred based on prior knowledge of other facts. However, we demonstrate that LLMs often struggle to memorize facts, the correct answer is overridden by co-occurrence statistics.
3.2 Metrics
Following the knowledge base completion litera- ture (Bordes et al., 2011 1, 2013), we consider two rank-based metrics, hits @ and MRR (mean recip- rocal rank), to evaluate the performance factual knowledge probing. The models that rank ground- truth objects higher are considered more knowled- able. Hits @ 1 is if the correct answer is ranked in the top prediction, otherwise 0. MRR is the average reciprocal rank of the correct answer in the prediction. When computing hits @ 1, we remove other valid objects for subject-relation pair other than the one we test, following the original setup of LAMA.
3 Factual Knowledge Probing
3.3 Restricted Candidate Sets
This section describes the overall framework to test factual knowledge of L.Ms.. With this framework, we aim to investigate the effects of model sizes, finetuning, and co-occurrence statistics.
Since LLMs are not trained to act as knowledge bases, we use restricted output candidate sets following the recent work (Xiong et al., 2020;

---

Ravichander et al., 2020; Kassner et al., 2021; Elazar et al., 2021). Specifically, we use three different settings to restrict output vocabularies to study whether LLMs are capable of recognizing appropriate object candidates or not: (1) remove stopwords, (2) gold objects and (3) gold objects (relation-wise). The remove stopwords removes stopwords in the stopword list of NLTK (Bird et al., 2009) from the output candidates. The gold objects restricts the oulput vocabulary to the set of gold objects in the whole dataset. Similarly, the gold ob- jects (relation-wise) restricts the output candidates to the set of gold objects for each relation in the dataset.
pear in the pre-training dataset. The joint probabil- ity ranks object candidates based on how frequently they appear with the subject in the pre-training dataset. Following the definition of PMI (pointwise mutual information) (Church and Hanks, 1990), the PMI baseline normalizes Ppretrain (obj) subj), the conditional probability of objects given a sub- ject, by Ppretrain (obj), the marginal probability of objects. We measure hits and MRR of the baselines and compare them with the target LLMs.
4.3 Correlation Metrics
To analyze the correlation between factual knowl- edge of LLMs and co-oceurrence statistics, we plot hits @ of the target LLMs against subject-object co-oceurrence counts. Here, we consider two types of measures for co-occurrence: (1) the reciprocal rank of subject-object co-occurrence counts and (2) the conditional probability of the gold object given a subject. The former is a relative measure since it considers a reciprocal rank of the gold ob- ject among output candidates, while the latter is an absolute measure as it uses conditional probability regardless of other output candidates. Here, we use the co-occurrence statistics of the pre-training corpora to compute co-occurrence counts and con- ditional probabilities.
4 Factual Knowledge Probing with Co-occurrence Statistics
This section describes our framework to analyze the impact of pre-training co- co-occurrence statis- tics on factual knowledge of LLMs. We first test how much factual knowledge can be recalled with term frequency statistics, including co-occurrence. Then, analyze the correlation between occurrence statistics and factual predictions.
4.1 Co-occurrence Statistics
We consider the co-occurrence statistics of the pre- training dataset of target models. Since it is in- tractable to count co-occurrences of every n-gram pair, we only count co-occurrences between pairs in minimal sufficient set. This set is initialized as a set of subject entities in the LAMA-TRIE dataset and words in the target model's vocabulary, which are object candidates. For text normaliza- tion, we tokenize words based on Penn Treebank (Marcus et al., 1993) and remove stopwords in the resulting tokens. Then, we filter out entities those are composed of more than three tokens. Due to computational burdens from the large amount of documents, we count whether an entity pair ap- pears in the same document or not, instead of using a sliding window approach.
Experiments
5.1 Target Models
We test open-source versions of GPT-3 (Brown et al., 2020) with four different model sizes: GPT- Neo 125M, GPT-Neo 1.3B, GPT-Nco 2.7B, and GPT-J 6B (Black et al., 2021; Wang and Komat- suzaki, 2021), which are publicly available on Hug- gingface's transformers (Wolf et al., 2020). These models are pre-trained on the Pile (Gao et al., 2020), which is a publicly available dataset that consists of 800GB of high-quality texts from 22 different sources.
5.2 Results
4.2 Term Frequency Baselines
5.2.1 Factual Knowledge Probing
We test how much factual knowledge can be re- called with simple term frequency statistics by
The results of micro-average hits@ 1 on the test set reported in Figure 2. Figure 2a shows the results in the zero-shot setting. We observe that hits @ I is higher as the model larger and we re- strict the output candidates to a smaller set of gold objects. In other words, scaling up model sizes can improve the performance on factual knowledge
measuring the performance of three different term
frequency baselines: (1) marginal probability, (2) joint probability and (3) PMI. For a subject and re- lation pair, the marginal probability baseline ranks object candidates based on how frequently they ap-

---

GPT-Neo 125M
GPT-Neo GPT-Neo
GPT-Neo
GPT-Neo 1.38
GPT-Neo
GPT-J
1.00
1.00
0.75
0.75
Hits@1
Hitts@1
0.50
0.50
0.25
0.5
defauit
remove stopwords
gold objects
gold objects (relation-wise)
defsult
remove stopwords
gold objects
gold objects (relation-wise)
(a) zero-shot
(b) finetuned
Figure 2: Effects of model sizes and restricted candidate sets: We plot micro-average hits @ I on the test set. (a) the zero-shot setting, we observe that hits@1 is higher as the model is larger and as the output vocabulary is restricted smaller set. (h) In the finetuned setting, we observe that the effect of model sizes and restricted candidate sets is marginal. Effects of finetuning: We observe that finetuning boosts the overall performance.
probing, and LLMs struggle to recognize appro- priate object candidates. Figure 2b presents the results in the finetuned setting. We find that the effect of model sizes is marginal. Different from the zero-shot setting, the effect of restricting the output candidates is also marginal, implying that the models may learn appropriate candidate sets during finetuning. We also observe that finetun- ing improves factual knowledge probing accuracy substamtially. The results of MRR are shown Figure 10 in Appendix C as they exhibit a similar tendency.
GPT-Neo 125M
GPT-Neo 1.38
GPT-Neo 2 7B GPT-J68
1.00
Hits@1
0.50
0.00
default
objects
gold objects (relation-wise)
stopwords
Figure 3: Memorization capacity of finetuned LLMs: We show micro-average hits@1 1 of the finetuned models on the training set. We observe that the models are capable of memorizing most of the seen facts during finetuning except for the smallest model.
Figure shows the hits @ results of finetuned models on the training set. We observe that the models except for the smallest one are capable of memorizing most of the seen facts. This implies that memorization is necessary to recall facts since factual knowledge in the test set may not be in- ferred based on prior knowledge of other facts in the training set.
Marginal probabilty
probabilty PMI
1.00
Hits@11
0.50
5.2.2 Term Frequency Baselines
0.25
Figure 4 shows how much factual knowledge can be recalled with term frequency statistics of the pre-training data. We observe that a large por- tion (about 60%) of the facts can be recalled with the joint probability baseline when the output can- didates are tightly restricted in the gold objects (relation-wise setting. The joint probability base- line performs as well GPT-J 6B, the largest model considered in our experiments. The results encourage us to consider the co-occurrence statis- tics when evaluating language models as may in- flate model performance. Although co-occurrence
0.00
remove stopwords
gold objects
gota' objects frelation wiss)
Figure 4: The results of term frequency baselines: We report micro-average hits@1 the test We observe that a large portion (about 60%) of the facts can recalled with the joint probability when the out- put candidates are tightly restricted in the gold objects (relation-wise) setting. Note that co-occurrence is useful but not sufficient recall facts.

---

GPT-Nes 125M
GPT-Neo 1.38
GPT-Neo 2.78
GPT-J
SPT-Neo 125M
GPT-Neo 1.38
2.77 GPT-J 68
Number of samples
Number of samples
1.00
3000
Nimber of samples
Number samples
0.75
0.75
2000
Hits@1
Hite@1
0.50
0,50
1000
1030
0.25
0.25
0.00
0.00
1/1
1/2
1/4
1/0
1/15
1/32
1/64
1/1
1/2
1/4
1/8
1/16
1/52
1/64
0
Plobji subj)
P(obj subi)
(a) zero-shot
(b) finetuned
The correlation between co-oceurrence statistics and factual knowledge probing accuracy: We plot hits@1 against P Ppretrain ( [obj]sub]) the conditional probability of the gold object given subject, on the test set the remove stopwords setting. In both (a) zero-shot and (b) finetuned settings, we observe strong correlation: hits@ I is lower as the co-occurrence count is lower. As result, LLMs struggle to recall rare facts. We observe that such correlation remains despite scaling up model sizes or finetuning.
Figure 5:
helps recall facts, it may not be appropriate to un- derstand the semanties behind words. The results of MRR, shown in Figure 11 in Appendix C, show a similar tendency.
+ 6PT--85 (estt-davins-003)
CharGPT (6PT-3.5turba) CharGPT (GPT-4
Number camples
1.00
1900
Number of samples
0.75
Hits@1
0.50
5.2.3 Correlation Analysis
500
0.25
This section reports the results of the correlation between co-occurrence statistics and factual knowl- edge of LLMs. Note that we exclude facts whose co-occurrence count is unknown (c.g. composed of an entity with more than three tokens), which amounts to less than 6% of the total samples. In this section, we analyze the effects of (1) finctuning and (2) scaling up model sizes.
1/1
1/2
1/4
1.8
1/16
1/32
1/64
P(obil cubl)
Figure 6: Correlational analysis of larger models: We test GPT-3.5 175B and ChatGPT on the subset of test data in the remove stopwords setting, verifying that correlation remains despite scaling up model sizes.
In Figure 5, we plot hits@ of the target models against P Prrotrain (obj) subj), the conditional prob- ability of the gold object given a subject. In both zero-shot and finetuned settings, we observe that hits 1 is lower as the co-occurrence count is lower. Consequently, LLMs suffer from generalizing recalling rare facts. Comparing Figure 5a and 5b, we observe that finetuning does not resolve co- occurrence bias despite improving the overall per- formance. In both zero-shot and finetuned settings, we observe that such correlation exists regardless of model sizes. We further test larger models: GPT- 3.5 (InstructGPTT 175B (Ouyang et al., 2022) and ChatGPT'2 GPT optimized to dialogue system. For ChatGPT, we test two variants with different sizes: (1) GPT-3.5-turo and (2) GPT-4. Since the vocabulary of ChatGPT different from the
GPT Noo 125M
NcC 1.38
GPT Neo 2.78
J6B
Number of samples
Number of samples
0.75
6000
0.50
4000
625
0.00
1/1
1/2
1/4
1/8
1/16
1/32
1/64
P(bb)
Figure 7: Correlational analysis of finetuned models on the training set: We also report the results of fine- tuned models on the training set in the remove stopwords setting. Surprisingly, we observe a similar trend to the results on the test set, indicating that LLMs struggle to memorize seen facts if they are rare in the pre-training corpora.
"Note that the pre-training data of GPT-3.5 and ChaiGPT are not the same as the Pile, but we use the results as a proxy.

---

Table I: The failure cases of GPT-J 6B, preferring words with higher co-occurrence counts over the correct swers. The numbers in parentheses represent the co- occurrence counts.
The failure analysis of GPT-J 6B, com- paring the conditional probability of predictions, Prettrain (obj subj), and the conditional probability of the groundtruth objects. Ppretruin( (abji/oobbsss We re- port the mean and standard deviation of conditional probabilities each frequency bin.
Table 3:
Query
Groundtruth
Prediction
Tim Mitchell was born in La Promesse created
Detroir (19)
London (246)
Belgium (87)
France (3420)
Frequency bin
P(obj]subj) 0.42
P(obj[sub)
Yutaka Abe died
Kyoto (14)
Tokyo (43)
Bell Labs owned
Nokia (1744)
Google (5167)
1/1
1.00
Were Iu located
Ethiophia (129)
Ivrael (254)
1/2
0.38++0.28
0.72+t0.14
1/4
0.37:10.27
0.37+10.07
Table 2: The quantitative failure analysis of GPT-J 6B, counting how often word with higher co-occurrence preferred over the correct answer. We report the ratio of biased cases to the total failure cases in each frequency bin.
1/8
0.31
0.18
1/16
0.29:10.29
0.09
1/32
0.30+10.31
0.05
1/64
0.26:10.32
0.02+10.00
0
0.26+10.30
0.01
Frequency bin
Ratio
Total
0.35+0.22
0.46:10.2
1/1
0%
1/2
15%
and biased responses.
1/4
42%
We also quantitatively measure how often the correct answer is overridden by word with higher co-occurrence counts. Here, count in each ques- tion whether the model's generated answer has higher co-occurrence counts than the correct an- swer when the model fails to answer correctly. We define a biased case as when the correct answer is overridden by a word with higher co-oceurrence counts. Table 2 reports the ratio of GPT-J 6B's ss biased cases to the total failure cases in each fre- quency bin. We observe that word with higher co-occurrence counts overrides the correct answer in a total of 38% of the failure cases. The results of different frequency bins indicate that the co- occurrence bias is more problematic when recall- ing rare facts. Additionally, Table compares the conditional probability of the generated objects, P pretroin (obj)s and the conditional probabil- ity of the gold objects, Ppretrain subj). The results show that LLMs prefer to generate words that co-occur with the subject frequently enough (P PPretrrin (objlsubj)) 2 0.26). In other words, re- calling rare facts is especially difficult since words with low co-occurrence counts are hardly gener- ated.
1/8
56%
1/16
70%
1/32
78%
1/64
85%
95%
Total
38%
open-souree target models, we report the results the subset of test data, which filters out samples whose answer is not in the vocabulary of ChatGPT, The results in Figure verify that scaling up model sizes does not resolve co-occurrence bias while improving the overall performance.
In Figure 7, we investigate the results of seen facts during finetuning. Interestingly, LLMs strug- gle to learn facts that rarely appear in the pre- training corpora although they are explicitly given during finetuning. The results of hits@1 1 against the reciprocal rank of subject-object co-occurrence counts are shown in Figure 12, 13 in Appendix C.
5.2.4 Failure Analysis
Co-occurrence statistics are necessary but not suffi- cient to recall facts, as shown in Figure 4. There- fore, heavy reliance on co-occurrence may be problematic. Table showcases failure cases of GPT-J 6B in the gold objects (relation-wise set- ting. The examples demonstrate that the model fails to recall facts by selecting words with higher co-occurrence counts over the correct answers. This implics that co co-occurrence statistics may often work as spurious features, leading to hallucinations
Mitigation
Debiasing with Undersampling Considering facts whose subject-object co-occurrence count is high as biased samples, we suggest finetuning LMs on debiased dataset, constructed by filtering out biased samples from the training set. Given a dataset D with samples x; and corresponding bias

---

Number samples
D.
Random
0.5
Number samples
Debiased Debiased 0.3
Debiased 0.5
1.50
8000
1,00
8000
0.76
Number of samples
0.75
6000
Number of samples
Httaga
0.50
4000
Hite@]
0.90
4000
0.25
0.25
2000
0.00
0.00
1/1
1/2
1/4
1/0
1/15
1/32
1/64
1/1
1/2
1/4
1/8
1/16
1/52
1/54
P/obji subj)
P[obj subi)
(a) Debiased finetuning vs random filtering
(b) Effects of filtering ratios on debiased finetuning
We report the micro- average hits 1 of GPT-J 6B the original training set in the remove stopwords setting, comparing debiased finetuming and the random filtering baseline. The results show that debiased finetuning helps to learn rare facts but hampers learning frequent facts. (b) Effects of filtering ratios: We observe that higher filtering ratios cause performance degradation on frequent facts but marginal improvement on rare facts. Therefore, a filtering ratio needs to be properly tuned to maximize the performance gains on rare facts while keeping the performance on frequent facts.
Figure& 8: (a) Effects of debiased finetuning:
of different filtering ratios on debiased finetuning. We observe that the performance on frequent facts significantly degrades while improvements on rare marginal as more samples are filtered out. Furthermore, investigate the effects of debiased finetuning on the test set in Figure 9, We observe that the performance of the debiased model and the baseline are similar regardless co-occurrence counts, implying that the effect of debiased fine- tuning is not generalizable. Since it is non-trivial to directly fix the cause of co-occurrence bias, de- signing a more sophisticated debiasing algorithm or using other approaches may be beneficial complement the proposed debiased finetuning. We leave further investigation in this direction as future work.
samples
Deblased 0.5
Random
1.00
2000
0.75
Number of samplics
2000
0.50
1000
0.25
0.00
1/1
1/2
1/4
1/8
1/16
1/32
1/64
subb)
Figure 9; Effects of debiased finetuning on the test set: We also analyze the effects of debiased finetuning on the test set. The results show that the effect of debiased finetuning is marginal at test time.
scores Scor (I;), and a filtering ratio p. we dis- card 5% of the total samples with the highest bias scores. We compute scoretias (x;; as the condi- tional probability Ppretrain(oby number of samples is reduced, we consider ran- dom filtering baseline that randomly filters out p% of the total samples for a fair comparison. Note that we apply the filtering algorithms for each relation separately to prevent discarding nearly all samples highly biased relation. We test three different filtering ratios: 0.1, 0.3, and 0.5.
7 Conclusion
Since the
In this work, we investigate the impact of co- occurrence statistics of the pre-training corpora on factual knowledge of LLMs. We reveal the co-occurrence bias of LLMs, in which they pre- fer words that frequently co-occur with the subject entity over the correct answer. As a result, LLMs struggle to recall facts whose subject and object rarely co-occur in the pre-training corpora although they are seen during finetuning. Although scaling up model sizes or finetuning substantially improves the overall performance, the co-occurrence bias re- mains. Therefore, we suggest further investigation on mitigating co-occurrence bias to ensure the reli- ability of language models.
The results of GPT-J 6B on the original train- ing set are shown in Figure 8. Figure 8a compares the debiased model with the random filtering base- line with filtering ratio 0.5. The debiased model surpasses the baseline on rare facts with sacrifices on frequent ones. Figure 8b compares the effects

---

Limitations
dangers of stochastic parrots: Can language models be too big? In Proceedings of the ACM conference fairness, accountability, and transparency.
Due to the requirement of large computational re- sources and the availability of pre-training corpora, we are only able to test limited set of LLMs. Al- though we believe that testing other LLMs does not invalidate our claims, it would be good to verify the scalability of the results to strengthen our claims. Another limitation is that our work only focuses on a factual knowledge probing test, which may not be aligned with real-world scenarios. It would be beneficial to investigate how our results generalize to downstream tasks, such as question answering and summarization.
Steven Bird, Ewan Klein, and Edward Loper. 2009, Nat- ural language processing with Python: analyzing text with the natural language toolkit.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh- Tensorflow.
Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is computer programmer woman is home- maker? debiasing word embeddings. Advances in Neural Information Processing Systems.
Ethics Statement
Rishi Bommasani. Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bobg. Antoine Bosse- lut, Emma Brunskill, et al. 2021. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2108.0725.
In light of the growing prevalence of LLMs, eth- ical concerns and challenges have also emerged. For example, LLMs often generate factually in- correct or biased responses. Within this context, our work shows that LLMs strongly correlate with simple co-occurrence statistics of the pre-training corpora, implying that they may generate the most co-occurring word without truly understanding the meaning behind words. We believe that our work has positive social impact it suggests direc- tion toward mitigating potential harms to ensure reliability.
Antoine Bordes, Nicolas Usunier, Alberto Garcia- Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi- relational data. Advances in Neural Information Pro- cessing Systems.
Antoine Bordes, Jason Weston, Ronan Collobert, and Yoshua Bengio. 201 I. Learning structured embed- dings of knowledge bases. In Proceedings of the AAAI Conference on Artificial Intelligence.
Acknowledgements
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems.
This work was supported by Institute of Informa- tion & communications Technology Planning & Evaluation (IITP) grant funded by the Korea gov- eMMMent(MSIT) NN..2022-0-00984 Development of Artificial Intelligence Technology for Personal- ized Plug-and- long-and-Play Explanation and Verification of Explanation). (No.2022-0-0018%. Development and Study of AT Technologies to Inexpensively Conform to Evolving Policy on Ethics), (No.2019- 0-00075, Artificial Intelligence Graduate School Program(KAIST)
Aylin Caliskan, Joanna Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from lan- guage corpora contain human-like biases. Science.
Boxi Cao. Hongyu Lin, Xianpei Han, Le Sun, Lingy- ong Yan, Meng Liao, Tong Xue, and Jin Xu. 2021. Knowledgeable or educated guess? revisiting lan- guage models as knowledge bases. In Proceedings of the Annual Meeting of the Association for Com- putational Linguistics and the International Joint Conference on Natural Language Processing.
References
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2023. Quantifying memorization across neural lan- guage models. In International Conference on Learn- ing Representations.
Ekin Akyurek, Tolga Bolukbasi, Frederick Liu, Bin- bin Xiong, Ian Tenney, Jacob Andreas, and Kelvin Guu. 2022. Towards tracing knowledge in language models back to the training data. In Findings of the Conference on Empirical Methods in Natural Lan- guage Processing.
Nicholas Carlini, Chang Liu, Úlfar Brlingsson, Jemej Kos, and Dawn Song. 2019. The secret sharer: Eval- uating and testing unintended memorization neural networks. In USENIX Security Symposium.
Emily M Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the

---

Nicholas Carlini, Florian Tramer, Eric Wallace. Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom B Brown, Dawn Song, UI- far Erlingsson, al. 2021. Extracting training data from large language models. In USENIX Security Symposium.
William Fish. 2009, Perception, hallucination, and illusion.
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing, Travis Hoppe, Charles Foster, Jason Phang, Ho- race He, Anish Thite, Noa Naheshima, et al. 2020. The pile: An 800GB dataset of diverse text for lan- guage modeling. arXiv preprint arXiv:2101.000
Kenneth Church and Patrick Hanks. 1990. Word associ- ation norms, mutual information, and lexicography. Computational Linguistics.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Sehwartz, Samuel Bowman, and Noah A Smith. 2018. Annotation artifacts in natural language in- ference data. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies.
Jacob Devlin, Ming-Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the Conference the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies.
Edward J Hu. yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang. Lu Wang. and Weizhu Chen. 2022. LoRA: Low-rank adaptation large language models. In International Conference on Learning Representations.
Yanai Flazar, Nora Kassner, Shauli Ravfogel, Amir Feder, Abhilasha Ravichander, Marius Moshach, Yonatan Belinkov, Hinrich Schütze, and Yoav Gold- berg. 2022. Measuring causal effects of data statis- tics on language model's 'factual' predictions. arXiv preprint arXiv:2207. 02.2007..225... 14251.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang. Andrea Madotto, and Pascale Fung. 2023. Survey of halluci- nation in natural language generation. ACM Comput- ing Surveys
Yanai Elazar. Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. 2021. Measuring and improving con- sistency in pretrained language models. Transactions of the Association for Computational Linguistics.
Zhenghao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics.
Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Lafor- est, and Elena Simperl. 2018. T-REx: A large scale alignment of natural language with knowledge hase triples. In Proceedings of the International Confer- ence on Language Resources and Evaluation.
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023. Large language models struggle to learn long-tail knowledge. In International Conference on Machine Learning.
Ali Emami, Kaheer Suleman, Adam Trischler, and Jackie Chi Kit Cheung. 2020. An analysis of dataset overlap on winograd-style tasks. In Proceedings of the International Conference on Computational Lin- guistics.
Nikhil Kandpal, Eric Wallace, and Colin Raffel. 2022. Deduplicating training data mitigates privacy risks in language models. Proceedings of Machine Learning Research.
Allyson Ettinger. 2020. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics.
Nora Kassner, Philipp Dufter, and Hinrich Schütze. 2021. Multilingual LAMA: Investigating knowledge in multilingual pretrained language models. In Pro- ceedings of the Conference of the European Chapter of the Association for Computational Linguistics.
Vitaly Feldman. 2020. Does learning require memoriza- tion? a short tale about long tail. In Proceedings of the ACM SIGACT Symposium on Theory of Comput- ing.
Nora Kassner, Benno Krojer, and Hinrich Schütze. 2020. Are pretrained language models symbolic reasoners over knowledge? In Proceedings of the Conference on Computational Natural Language Learning.
Vitaly Feldman and Chiyuan Zhang. 2020. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neura! Information Processing Systems.
Nora Kassner and Hinrich Schütze. 2020. Negated and misprimed probes for pretrained language models: Birds can talk. but cannot fly. In Proceedings of the Association for Computational Linguistics.
Leandra Fichtel, Jan- Chn-Chrsttoph Kalo, and Wolf- Tilo Balke. 2021. Prompt tuning or fine-tuning- investigating relational knowledge pre-trained lan- guage models. Conference on Automated Knowl- edge Base Construction.
Mehran Kazemi, Sid Mittal, and Deepak Ramachan- dran. 2023. Understanding finetuning for factual knowledge extraction from language models. arXiv preprint urXiv:2001.11299

---

Katherine Lee, Daphne Ippolito. Andrew Nystrom, Chiyuan Zhang. Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2022. Deduplicating training data makes language models better. In Proceedings of the Association for Computational Linguistics.
Mike Mintz, Steven Bills, Rion Snow, and Dan Juraf- sky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the Annual Meeting of the Association for Computational Linguistics and the International Joint Conference Natural Language Processing.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the Conference on Empiri- cal Methods in Natural Language Processing.
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems.
Shaobo Li, Xiaoguang Li, Lifeng Shang, Zhenhua Dong, Cheng-Jie Sun, Bingquan Liu, Zhenzhou Ji, Xin Jiang, and Qun Liu. 2022. How pre-trained language models capture factual knowledge? a causal causal-inspired analysis. In Findings of the Association for Compu- tational Linguistics.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowl- edge bases? In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing.
Nina Poerner. Ulli Waltinger, and Hinrich Schütze. 2020. E-BERT: sffiiienntyet-effeete for BERT. In Findings of the Conference Empiri- cal Methods in Natural Language Processing.
entity embeddings
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023a. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys.
Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying LMs with mixtures of soft prompts. In Proceedings of the Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengx- iao Du, Zhilin Yang, and Jie Tang. 2022. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the As- sociation for Computational Linguisties.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. Journal of Machine Learning Research.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian. Zhilin Yang, and Jie Tang. 2023b. GPT understands, too. Al Open.
Abhilasha Ravichander, Eduard Hovy, Kaheer Suleman, Adam Trischler, and Jackic Chi Kit Cheung. 2020. On the systematicity of probing contextualized word representations: The case of hypernymy in BERT. In Proceedings of the Joint Conference on Lexical and Computational Semantics.
Inbal Magar and Roy Schwartz. 2022. Data contami- nation: From memorization to exploitation. In Pro- ceedings of the Association for Computational Lin- guistics.
Mitch Marcus. Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building large annotated corpus of English: The Penn Treebank. Computa- tional Linguistics.
Yasaman Razeghi, Robert L Logan IV, Matt Gardner. and Sameer Singh. 2022. Impact of pretraining term frequencies on few-shot numerical reasoning. In Findings of the Conference on Empirical Methods in Natural Language Processing.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and
Ryan McDonald. 2020. On faithfulness and factual- ity in abstractive summarization. In Proceedings of the Association for Computational Linguistics.
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the param- eters of a language model? In Proceedings of the Conference Empirical Methods in Natural Lan- guage Processing.
R Thomas McCoy, Paul Smolensky, Tal Linzen, Jian- feng Gao, and Asli Celikyilmaz. 2023. How much do language models copy from their training data? evalu- ating linguistic novelty in text generation using raven. Transactions of the Association for Computational Linguistics.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafcai, Antoine Chaffin, Arnad Stiegler, Arun Raja, Manan Dey. M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019, Right for the wrong reasons: Diagnosing syntactic heuris- tics natural language inference. Proceedings of the Association for Computational Linguistics.

---

Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang. Tr- ishala Neeraj, Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022. Multi- task prompted training enables zero-shot task gener- alization. In International Conference on Learning Representations.
Natural Language Processing: System Demonstra- tions.
Wenhan Xiong, Jingfei Du William Yang Wang, and Veselin Stoyanov. 2020. Pretrained encyclopedia: Weakly supervised model. In International Conference on Learning Representations.
language
Chiyuan Zhang, Daphne Ippolito. Katherine Lee. Matthew Jagielski, Florian Tramèr, and Nicholas Car- lini. 2021. Counterfactual memorization in neural language models. arXiv preprint arXiv:21 12938.
Taylor Shin, Yasaman Razeghi, Robert L Logan IV,
Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting knowledge from language models with au- tomatically generated prompts. In Proceedings of the Conference on Empirical Methods Natural Lan- guage Processing.
Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021. Factual probing is [MASK]: Learning vs. learning to recall, In Proceedings of the Conference of the North American Chapter of the Association for Computa- tional Linguisties: Human Language Technologies.
Herbert A Simon. 1954. Spurious correlation: A causal interpretation. Journal of the American Statistical Association.
Congzheng Song and Vitaly Shmatikov. 2019, Audit- ing data provenance text-generation models. In Proceedings of the ACM SIGKDD International Con- ference Knowledge Discovery & Data Mining.
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gard- ner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing NLP. In Pro- ceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing.
Ben Wang and Aran Komatsuzaki. 2021. GPT-J- Billion Parameter Autoregressive Lan- guage Model. https://ss //gi httt...co/kingofllll mesh-trransformer-1
6B:
Tianlu Wang. Rohit Sridhar, Diyi Yang, and Xuezhi Wang. 2022. Identifying and mitigating spurious cor- relations for improving robustness in NLP models. In Findings of the Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V Le. 2022. Finetuned language mod- els are zero-shot learners. In International Confer- ence on Learning Representations.
Jason Wei, Dan Garrette, Tal Linzen, and Ellie Pavlick. 2021. Frequency effects syntactic rule learning in transformers. In Proceedings of the Conference on Empirical Methods in Natural Language Processing.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the the-rart natural language processing. Pro- ceedings of the Conference on Empirical Methods in

---

A Details of Finetuning
We train models for 3 epochs with a learning rate of 2e-5 and a batch size 128, padding sequences to a fixed length of 128. Models are trained to predict the masked word only, rather than the whole input prompt. The other hyperparameters are the same as the default hyperparameters of training script of causal language modeling in Huggingface's transformers (Wolf et al., 2020).
For finetuning uni-directional LMs, use manually designed prompt, "### InputAn (X]\nk###* Response:", where X is replaced with masked fact (e.g. "The capital of Canada is [MASK].T] "))
B Factual Knowledge Probing Dataset
Table 4: Descriptions and statistics the LAMA-TRES dataset.
Relation ID
Label
Template
Type
Train
Test
P17
country
[X] located in [Y].
262
P19
place of birth
[X] was born in [Y].
N-1
537
243
P20
place of death
[X] died [Y]-
N-1
582
235
P27
country of citizenship
[X] is [Y] citizen
N-M
691
267
P30
continent instunce of capital official language position held shares border with field of work native language occupation
[X] located [Y].
N-1
657
302
[X] [Y].
N-M
608
274
P36
The capital [X] is [Y]
1-1
330
141
P37
The official language of [X] is (Y).
N-1
620
280
P39
[X] has the position of [Y].
N-M
330
155
P47
[X] shares border with [Y]
N-M
448
203
P101
[X] works
field of [Y].
N-M
164
P103
The native language of [X] (Y).
635
284
P106
[X] is [Y] by profession.
N-M
569
252
P108
employer
[X] works for [Y].
N-M
274
104
P127
owned by
[X] owned [Y]
N-1
424
195
located administrative territorial entity
[X] located in [Y].
240
P136
genre named after
[X] plays [Y] music.
243
P138
[X] is named after [Y]
N-1
327
140
P140
religion
[X] is aftiliated with the [Y] religion.
N-1
299
135
P159
headquarters location
The hcadquarter of [X] in [Y].
N-1
565
236
P176
manufacturer
[X] produced by [Y].
P178
developer twinned administrative body record label location subclass of part of original language of film or TV show
[X] is developed by [Y].
N-M
411
177
P190
[X] and [Y] are twin cities.
N-M
454
217
P264
X is represented by music label [Y].
N-1
43
10
P276
[X] located in [Y].
N-1
515
251
P279
[X] subclass of [Y].
N-1
280
P361
[X] is part [Y]
N-1
533
223
P364
The original language of [s] [Y].
N-1
531
225
P407
language of work name position played team / speciality
[X] written in [Y].
N-1
598
259
P413
[X] plays [Y] position..
N-1
675
277
original network member of country of origin has diplomatic relation location formation work location applics to jurisdiction instrument capitul of languages spoken, written signed
[X] was originally aired on [Y).
223
P463
[X] member of [Y].
N-M
153
50
P495
[X] was created in [Y].
N-1
652
253
P527
[X] consists of (Y)
N-M
661
295
P530
X maintains diplomatic relations with
N-M
667
283
[X] founded in [Y].
244
P937
[X] used to work in [Y].
N-M
592
P1001
[X] is legal term in [Y].
N-M
461
203
P1303
[X] plays [Y].
N-M
352
161
P1376
XX] capitul of [Y].
1-1
120
59
P1412
[X] used to communicate in [Y]
N.M
259
Total
20662
8856

---

C Additional Results
GPT-Neo 125M
GPT-Neo 1.38
GPT-Neo
GPT-J
GPT Neo 125M
GPT-Neo 1.38
GPT-Neo 2.78
GPT-J
1.00
1.00
0.75
0.75
MRR
MRR
0.50
0.50
defauit
remove stopwords
gold objects
gold objects (relation-wtte)
default
remove stopwords
gold objects
gold objects (relation-wise)
(a) zero-shot
(b) finctuned
Figure 10: Effeets of model sizes and restricted candidates sets: We show micro-average MRR the test set. (a) the zero-shot setting, we observe that MRR higher as the model is bigger and as the output vocabulary is restricted smaller (b) In the finetuned setting, observe that the effect of model sizes and restricted candidate sets is marginal. Effects of finetuning: We observe that finetuning boosts MRR on factual knowledge probing.
Marginal probabiltt
Joint probabilty PMI
1.00
MAR
0.50
0.25
0.00
remove stopwords
gold objects
gold objects (relation-wise)
Figure 11: The results of term frequency baselines: We plot mvero-average MRR on the test set. In the gold objects (relation-wise) setting, observe that the joint probability performs well as GPT-J 6B, which is the largest model considered in our experiments.
GPT Neo 125M
GPT-Neo 1.38
GPT-Neo 2.78
GPT 6B
CeT.Neo 125M
GPT-Neo 1.38
GPT 2.78
GPT 6B
Number samples
Number of samples
1.00
4000
Number of samples
Number of samples
0.75
3000
0.75
3000
Hlts@
Hits@1
0.50
2000
0.50
2000
0.55
1000
0.25
1000
0.00
0.00
1/16
rank of subject-ojject
Reclprocalranke subject
counts
(a) zero-shot
(b) finetuned
Figure 12: The correlation between co-occurrence statistics and factual knowledge probing accuracy: We plot hits @1 against the reciprocal rank of subject-objed co-occurrence counts on the test set in the remove stopwords setting. In both settings (a) and (b), observe strong correlation between co-oceurrence and hits @ 1.

---

Number of samples
Number of samples
Hits@1
5000
2500
0.00
1/1
1/2
1/4
1/8
1/16
1/32
1/64
Reciprocal rank of subject-object co-ocourrence counts
Figure 13: Correlational analysis of finetuned models on the training set: We also plot hits finetuned models against the reciprocal rank of subject-object co-occurrence counts the training set in the remove stopwords setting. Surprisingly, we observe a similar trend to the results on the test set, indicating that LLMs struggle to memorize seen facts they are rare in the pre- preirrannig corpora.


Reviews:
[{"review_id": "6J9oy63MjV", "reviewer": "Reviewer_M9gY", "summary": "", "questions": "-\tThe paper focuses only on token-level shortcuts. Can the findings be generalized to other kinds of shortcuts, such as word-overlap (Right for the wrong reasons: Diagnosing syntactic heuris- tics in natural language inference) or length (Annotation artifacts in natural language inference data) shortcuts?\n-\tThe proposed mitigation methods include balancing the data and knowledge editing. However, I am curious about whether demonstrations or Chain-of-Thoughts [1] can also solve the issue.\n-\tFig. 3 shows that smaller model cannot memorize the data. Then why it also has the shortcut problem like other larger models who memorize the biases in the pretraining data?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive reviews.\n\n> One more experiment should be done to verify the claim that \"answers with higher co-occurrence are more likely to be generated\"\n\nWe count in each question whether the model's generated answer has a higher co-occurrence count than the ground-truth when the model fails to answer correctly. We found that \"a word with higher co-occurrence is preferred over the correct answer\" in 38% of the failure cases by GPT-J 6B. We provide the results of different frequency bins below, showing that the co-occurrence bias is more problematic when recalling rare facts. We observe a similar trend for the other models.\n|frequency bin|1/1|1/2|1/4|1/8|1/16|1/32|1/64|0|total|\n|---|---|---|---|---|---|---|---|---|---|\n||0%|15%|42%|56%|70%|78%|85%|95%|38%|\n\nBelow, we also report the results of conditional probability (mean$\\pm$std) of predictions (first row) and ground-truths (second row). Since \"LLMs prefer words that frequently co-occur with the subject,\" recalling rare facts is especially difficult.\n|frequency bin|1/1|1/2|1/4|1/8|1/16|1/32|1/64|0|total|\n|---|---|---|---|---|---|---|---|---|---|\n|**$P(\\hat{obj}\\|subj)$**|0.42$\\pm$0.31|0.38$\\pm$0.28|0.37$\\pm$0.27|0.31$\\pm$0.26|0.29$\\pm$0.29|0.30$\\pm$0.31|0.26$\\pm$0.32|0.26$\\pm$0.30|0.35$\\pm$0.29|\n|**$P(obj\\|subj)$**|1.00$\\pm$0.00|0.72$\\pm$0.14|0.37$\\pm$0.07|0.18$\\pm$0.04|0.09$\\pm$0.02|0.05$\\pm$0.01|0.02$\\pm$0.00|0.01$\\pm$0.00|0.46$\\pm$0.32|\n\n> Elaborate more on the work to claim that they are the first to investigate the effects of finetuning on the correlation between term frequency statistics and factual knowledge of LLMs.\n\nAs our main claim is not \"to investigate the effects of finetuning on the correlation\", we decided to rewrite this part to emphasize the main claim.\nWe agree that \"the correlation between word frequency and model behaviors\" is previously observed in existing work. However, the paper offers unique contributions by providing additional evidence and insights with detailed analysis. We verify that (1) LLMs learn co-occurrence bias from the pre-training data, which is especially problematic when recalling rare facts, and (2) co-occurrence bias is not perfectly overcome by scaling up model sizes or finetuning. The observations question whether LLMs recall facts with actual semantic understanding or not.\n\n> Can the findings be generalized to other kinds of shortcuts, such as word-overlap or length?\n\nYes. For example, if a heuristic (word overlap or sentence length) frequently co-occurs with specific labels, the model may learn the shortcut, preferring labels that frequently co-occur with the heuristics.\n\n> However, I am curious about whether demonstrations or Chain-of-Thoughts can also solve the issue.\n\nWe test prompting with demonstrations in a 4-shot setting, where four demonstrations with the same relation are randomly given for each sample. The results of GPT-J 6B in the remove_stopwords setting show significant improvements in hits@1 (**0.26 => 0.55**). However, the co-occurrence bias remains, as shown in the table below.\n|frequency bin|1/1|1/2|1/4|1/8|1/16|1/32|1/64|0|\n|---|---|---|---|---|---|---|---|---|\n|**hits@1**|0.86|0.62|0.51|0.42|0.26|0.15|0.09|0.07|\n\n> Fig. 3 shows that smaller model cannot memorize the data. Then why it also has the shortcut problem like other larger models who memorize the biases in the pretraining data?\n\nFig. 3 shows that the smallest model cannot memorize the fine-tuning data, not the pre-training data. However, the shortcut problem in this paper arises when a model memorizes the biases (shortcuts) in the pre-training data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-\tThe paper contains detailed analysis of shortcut problem regarding token co-occurrence.\n-\tThe paper investigates an important area of verifying the factual knowledge of LLMs.\n-\tThe paper is well-written, and is free of significant presentation issues.\n-\tAlong with the identification of the problem, the paper also proposes to mitigate the problem.\n", "reasons_to_reject": "-\tOne more experiment should be done to verify the claim that \"answers with higher co-occurrence are more likely to be generated\": The authors should count in each question, whether the model's generated answer has a high count in the pretraining corpus. Currently there is only a table (Table 1) showing similar results, i.e., the wrong answers have a relatively lower count in the pretraining corpus. However, quantitative results over the whole dataset should be given.\n-\tThere are existing work probing the shortcut learning problem of language models. The authors should elaborate more on the work to claim that they are the first to investigate the effects of finetuning on the correlation between term frequency statistics and factual knowledge of LLMs. For example: [2].", "questions_to_authors": "-\tThe paper focuses only on token-level shortcuts. Can the findings be generalized to other kinds of shortcuts, such as word-overlap (Right for the wrong reasons: Diagnosing syntactic heuris- tics in natural language inference) or length (Annotation artifacts in natural language inference data) shortcuts?\n-\tThe proposed mitigation methods include balancing the data and knowledge editing. However, I am curious about whether demonstrations or Chain-of-Thoughts [1] can also solve the issue.\n-\tFig. 3 shows that smaller model cannot memorize the data. Then why it also has the shortcut problem like other larger models who memorize the biases in the pretraining data?", "paper_topic_and_main_contributions": "The paper proposes a framework to probe the shortcuts in LLMs. The framework firsts test the accuracy of LLM on factual knowledge dataset like LAMA. Then the framework computes the correlation of the subject and the object in the pretraining corpus. Results on GPT models (from 125M to 6B) show that, as the co-occurrence of subject and object decreases, the accuracy also decreases. Larger models (GPT-3.5 of 175B) also suffer from the problem. The authors also find that simple baseline based on co-occurrence is sufficient to surpass the performance of GPT-J 6B. Finally, the authors also give solution to mitigate the shortcut problem.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "[1] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models\n\n[2] Identifying and Mitigating Spurious Correlations for Improving Robustness in NLP Models (NAACL Findings 2022)"}, "extra_text": {}}, {"review_id": "EIKFZmuV6r", "reviewer": "Reviewer_B1hy", "summary": "", "questions": "In the discussion of fine-tuning in lines 347--348, it is said that \"the models may learn appropriate cadidate sets during finetuning.\" Does this mean that there is some amount of knowledge leakage in fine-tuning and that the testing set-up is slightly different from the zero-shot setting?\n\nIn Figure 6, we see that the accuracy of the models' outputs _improves_ when the conditional probability of the object given the subject goes down from around 1/64 to 0. This goes against the general trend of the positive association between conditional probability and accuracy. Is there any interpretation for this?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable reviews.\n\n> The main point of the paper -- that LLMs suffer from co-occurrence bias -- is not particularly new.\n\nWe agree that \"the correlation between word frequency and model behaviors\" is previously observed. However, the paper offers unique contributions by providing additional evidence and insights with detailed analysis. We verify that (1) LLMs learn co-occurrence bias from the pre-training data, which is especially problematic when recalling rare facts, and (2) co-occurrence bias is not perfectly overcome by scaling up model sizes or finetuning. The observations question whether LLMs recall facts with actual semantic understanding or not.\n\n> No discussion of why fine-tuning should help at all, and in the end it does not. \n\nSince our main claim is not \"to investigate the effects of finetuning on the correlation between term frequency statistics and factual knowledge of LLMs\", we decided to rewrite this part to highlight the main claim. Besides that, we analyze the effects of finetuning since one may think that LLMs learn something beyond surface statistics during finetuning as it significantly improves the overall performance. \n\n> \"the models may learn appropriate candidate sets during finetuning.\"\n\nNo facts overlap between the finetuning and test datasets. However, the distribution of possible objects for each relation P(o|r) may be learned during finetuning. Although this can be thought as data leakage to some extent, it is non-trivial to recall the facts solely based on P(o|r).\n\n> In Figure 6, some results against the general trend of the positive association between conditional probability and accuracy.\n\nThis may be because (1) the pre-training datasets of these models (OpenAI GPT) are different from those of GPT-Neo/J, and (2) there are only a few samples whose conditional probability is 0.\n\n> The section on mitigating occupies less than 1.5 pages and does not contain any methodological insights. The paper does not propose a way out.\n\nOur main focus is to analyze the relationships between \"co-occurrence bias\" and model behaviors. Although we only showed that knowledge editing may be effective in mitigating the problem, there may be other possible approaches (probability calibration, retrieval). We leave further investigation in this direction as future work.\n\n> The authors state that \"relying on co-occurrence implies simple memorization.\" Again, the relationship between co-occurrence and memorization is more complicated, and equating one with the other seems erroneous.\n\nWe agree that \"relying on co-occurrence implies simple memorization\" may be erroneous. We would rather say \"LLMs often memorize surface statistics to recall facts, which may not be appropriate for actual semantic understanding.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "In my opinion, the most interesting aspect of the paper are the frequency baselines introduced to analyse the structure of the training dataset and to explain the behaviour of the models. They bring the co-occurrence/frequency bias influencing the behaviour of the models to fore, and the analysis is convincing.", "reasons_to_reject": "The main point of the paper -- that LLMs suffer from co-occurrency bias -- is not particularly new. Other papers investigating this issue are mentioned in the Related Work section. The authors claim that their work is the fist \"to investigate the effects of finetuning on the correlation between term frequency statistics and factual knowledge of LLMs\" (ll. 136--139), but there is no discussion of why fine-tuning should help at all, and in the end it does not, which amounts to a weak negative result. The section on mitigating occupies less than 1.5 pages and does not contain any methodological insights.", "questions_to_authors": "In the discussion of fine-tuning in lines 347--348, it is said that \"the models may learn appropriate cadidate sets during finetuning.\" Does this mean that there is some amount of knowledge leakage in fine-tuning and that the testing set-up is slightly different from the zero-shot setting?\n\nIn Figure 6, we see that the accuracy of the models' outputs _improves_ when the conditional probability of the object given the subject goes down from around 1/64 to 0. This goes against the general trend of the positive association between conditional probability and accuracy. Is there any interpretation for this?", "paper_topic_and_main_contributions": "The paper argues that large language models suffer from the so-called 'co-occurrence bias', i.e. when asked a factual question they tend to assign higher probabilities to words that have higher co-occurrence statistics with the query instead of the correct answer. The authors show that this bias persists in models of different sizes and cannot be effectively removed using fine-tuning, but can be partially remedied with knowledge editing. In order to prove their point, the authors introduce a series of term-frequency baselines, two of which are co-occurrence statistics. They show that these statistics can be used as a shortcut feature (in some settings, selecting the most frequently co-occurrent word results in 60% accuracy on the test set) and explain the actual behaviour of the models to a large extent. The paper explores two mitigation strategies: fine-tuning on a debiased version of the training dataset and rank-one model editing (ROME). While fine-tuning is not particularly effective, ROME shows promising results on frequent relation-like facts but does not really help with rarer ones.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Some of the statements in the paper look confusing. In lines 87--89, the authors point out that \"relying heavily on co-occurrence is not appropriate for understanding the accurate meaning behind words.\" This is probably true, but this is all we have when training LLMs, and the paper does not propose a way out. Similarly, in lines 173--174, the authors state that \"relying on co-occurrence implies simple memorization.\" Again, the relationship between co-occurrence and memorization is more complicated, and equating one with the other seems erroneous."}}, {"review_id": "rfX6ne8ne4", "reviewer": "Reviewer_2dAt", "summary": "", "questions": "-- I would appreciate your response to my first point above.\n\n-- Was there any attempt to look at stronger, proprietary models? I'm asking because they are much stronger and may exhibit diff behavior.\n\n-- The point about memorization is unclear to me. Why does co-occurrence imply memorization? on the contrary, if the model \"saw\" the correct answer and was overridden by co-occurrence this is the opposite of memorization.\n\n-- Why is knowledge editing a mitigation strategy. Obviously wrong knowledge should be corrected, but this is independent of the cause of the mistake, which is the subject of this paper, isn't it?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive reviews.\n\n> What it does show is that questions where the degree of co-occurrence is smaller are more difficult for the model.\n\nWe count in each question whether the model's generated answer has a higher co-occurrence count than the ground-truth when the model fails to answer correctly. We found that \"a word with higher co-occurrence is preferred over the correct answer\" in 38% of the failure cases by GPT-J 6B. We provide the results of different frequency bins below, showing that the co-occurrence bias is more problematic when recalling rare facts. We observe a similar trend for the other models.\n|frequency bin|1/1|1/2|1/4|1/8|1/16|1/32|1/64|0|total|\n|---|---|---|---|---|---|---|---|---|---|\n||0%|15%|42%|56%|70%|78%|85%|95%|38%|\n\nBelow, we also report the results of conditional probability (mean$\\pm$std) of predictions (first row) and ground-truths (second row). Since \"LLMs prefer words that frequently co-occur with the subject,\" recalling rare facts is especially difficult.\n|frequency bin|1/1|1/2|1/4|1/8|1/16|1/32|1/64|0|total|\n|---|---|---|---|---|---|---|---|---|---|\n|**$P(\\hat{obj}\\|subj)$**|0.42$\\pm$0.31|0.38$\\pm$0.28|0.37$\\pm$0.27|0.31$\\pm$0.26|0.29$\\pm$0.29|0.30$\\pm$0.31|0.26$\\pm$0.32|0.26$\\pm$0.30|0.35$\\pm$0.29|\n|**$P(obj\\|subj)$**|1.00$\\pm$0.00|0.72$\\pm$0.14|0.37$\\pm$0.07|0.18$\\pm$0.04|0.09$\\pm$0.02|0.05$\\pm$0.01|0.02$\\pm$0.00|0.01$\\pm$0.00|0.46$\\pm$0.32|\n\n> Was there any attempt to look at stronger, proprietary models?\n\nWe observed that OpenAI GPT-4 follows a general trend in the correlation between co-occurrence and accuracy, as shown below.\n|frequency bin|1/1|1/2|1/4|1/8|1/16|1/32|1/64|0|\n|---|---|---|---|---|---|---|---|---|\n|**hits@1**|0.64|0.40|0.31|0.27|0.24|0.28|0.20|0.19|\n\n> The point about memorization is unclear to me. Why does co-occurrence imply memorization?\n\nWe agree that \"relying on co-occurrence implies simple memorization\" may be erroneous. We would rather say \"LLMs often memorize surface statistics to recall facts, which may not be appropriate for actual semantic understanding.\n\n> Why is knowledge editing a mitigation strategy?\n\nIt is non-trivial to directly fix the cause of the mistakes, as suggested in the results of Section 6.1 (finetuning on a balanced dataset). Therefore, we tested knowledge editing as an alternative among various candidates (probability calibration, retrieval) and showed its effectiveness. Since mitigation is not our main focus, we leave further investigation in this direction as future work.\n\n> What was claimed exactly in previous work with respect to the discussed bias and how does that differ from the work here?\n\n\"The correlation between word frequency and model behaviors\" is previously observed. However, the paper offers unique contributions by providing additional evidence and insights with detailed analysis. We verify that (1) LLMs learn co-occurrence bias from the pre-training data, which is especially problematic when recalling rare facts, and (2) co-occurrence bias is not perfectly overcome by scaling up model sizes or finetuning. The observations question whether LLMs recall facts with actual semantic understanding or not.\n\n> \"Our work is the first to investigate the effects of finetuning on the correlation between term frequency statistics and factual knowledge of LLMs.\"\n\nSince our main claim is not \"to investigate the effects of finetuning on the correlation between term frequency statistics and factual knowledge of LLMs,\" we decided to rewrite this part to highlight the main claim.\n\n> Figures 2b, 3: The graphs do not show much in my opinion. Could they be explained in a sentence in the text? what does the figure here contribute?\n\nThe figures show the effects of finetuning on the overall performance. Figure 2b shows that finetuning improves the performance substantially. Figure 3 shows that the models (except for the smallest one) memorize most of the facts in the finetuning dataset. We tried to show that the co-occurrence bias remains (Figures 5b, 7) after finetuning, although it improves the overall performance by a large margin (Figures 2b, 3).\n\n> Figure 5 (and in other places in the paper): can you also compute correlation w/o binning? what does that turn out to be?\n\nWe computed the pearson correlation without binning. We observed a positive correlation to some extent (p-value < 0.001), and the magnitude without binning is similar to that with binning.\n||w/o binning|with binning|randomly shuffled|\n|---|---|---|---|\n|**GPT-J 6B**|0.24|0.24|0.01|\n|**GPT-J 6B (finetuned)**|0.35|0.36|0.00|\n\n> Other comments: Section 4.2: Why not use the more standard names then like marginal probability, joint probability and PMI? Section 6.1: a more formal definition of the filtering method should be given.\n\nWe will reflect the suggestions in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-- The paper addresses an important and timely topic, namely the ability of LLMs to act as knowledge bases.\n\n-- The related work section is very elaborate and provides insight into the field at hand.\n\n-- The arguments of the paper are well-presented, and the writing is generally clear.", "reasons_to_reject": "-- I am somewhat confused as to the exact claim the paper is making. While the results clearly show a correlation between the co-occurrence statistics of a triplet and the performance of the model on it, it is not clear to me whether this in fact proves that there is a bias where such simple surface statistics push the language model astray from making the right prediction. What it does show is that questions where the degree of co-occurrence is smaller are more difficult for the model. The introduction reads “in which frequently co-occurred words are preferred over the correct answer.” I could not see how the experiments directly make this point. Simple correlation seems to me insufficient in this case, since making mistakes with little co-occurrence doesn’t mean that there is a different option with higher co-occurrence.\n\nIn order to show that the behavior is biased, I would expect the paper to shows that the surface statistics interfere in some sense with the prediction of the model, in a way that would make it predict such answer even when it is not true. For example, I would have expected the paper to examine questions which we would expect (based on their prevalence in the training data) the model to answer correctly, and show that in these cases it tends to make more errors where there is a strong collocation and that the mistakes is towards the collocating words. If the paper indeed makes this kind of more subtle claim and I have missed it, I would welcome a response from the authors on this matter. Thank you.\nFollowing rebuttal: the results you have posted are helpful and address this comment. Please include them in the next version.\n\n-- The results of the attempts to mitigate the bias are not very strong. I should say that I do not see it in itself as grounds for rejection.\n\n-- Some important presentational details are not sufficiently clear (see below).", "questions_to_authors": "-- I would appreciate your response to my first point above.\n\n-- Was there any attempt to look at stronger, proprietary models? I'm asking because they are much stronger and may exhibit diff behavior.\n\n-- The point about memorization is unclear to me. Why does co-occurrence imply memorization? on the contrary, if the model \"saw\" the correct answer and was overridden by co-occurrence this is the opposite of memorization.\n\n-- Why is knowledge editing a mitigation strategy. Obviously wrong knowledge should be corrected, but this is independent of the cause of the mistake, which is the subject of this paper, isn't it?", "paper_topic_and_main_contributions": "The paper investigates the effect of co-occurrence statistics on the ability of large language models to correctly answer simple factual questions (of the subject-relation-object form). The paper specifically checks whether simple co-occurrences between the subject and the object in the pretraining data, can lead the models to incorrectly answer factual questions where the co-occurrence diverges from the correct answer. As it is difficult to check this causal relation by direct manipulation (given the enormous costs of pretraining large language models), a correlation study was conducted. Results show correlation between the co-occurrence statistics of a triplet and the ability of the model to answer correctly questions on it.\nThe paper further explores mitigation strategies to combat this bias. Two approaches are explored: debiased finetuning and knowledge editing. The former approach presents limited gains. Knowledge editing does show promise, but may be restricted as it requires weeding out the incorrect facts one by one (if I understand correctly). It may also cause unintended changes to unrelated facts.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "--"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Local comments:\n-- What was claimed exactly in previous work with respect to the discussed bias and how does that differ from the work here?\n\n-- l. 136: you write \"Our work is the 136 first to investigate the effects of finetuning on the 137 correlation between term frequency statistics and 138 factual knowledge of LLMs.\" This was not sufficiently clear to me:  I thought your main claim is about the relation between these stats in pretraining and in model behavior. This should be made clearer. Otherwise, a well written previous work section.\n\n-- Section 4.2: Why not use the more standard names then like marginal probability, joint probability and PMI? \n\n-- Figures 2b, 3: The graphs do not show much in my opinion. Could they be explained in a sentence in the text? what does the figure here contribute?\n\n-- Figure 5 (and in other places in the paper): can you also compute correlation w/o binning? what does that turn out to be?\n\n-- Section 6.1: a more formal definition of the filtering method should be given.\n\n\nGrammar:\n\n-- l. 442: performances s.b. performance\n\n-- l. 501: changes s.b. change"}}]

Author rebuttal:

Meta-review: The paper investigates the co-occurrence bias in large language models, i.e. the tendency to assign higher probabilities to words that have higher co-occurrence statistics instead of the correct answer. The paper shows that this bias persists in models of different sizes and cannot be effectively removed using fine-tuning, but can be partially remedied with knowledge editing. While the methodology for studying and mitigating co-occurrence bias is novel, the insights that the paper provides are limited. Some of the claims the paper makes are not substantiated clearly enough, although during the discussion the authors provided some clarifications and additional results that will help to strengthen the paper.
Decision: Accept-Findings