Sources of Hallucination by Large Language Models on Inference Tasks
Authors: Nick McKenna
Tianyi Li
Liang Cheng
Mohammad Javad Hosseini
Mark Johnson
Mark Steedman
Venue: EMNLP 2023 Findings (2023)
Keywords: LLM
LM
language model
hallucination
natural language inference
NLI
entailment
directional
attestation
relative frequency
predicate
TL;DR: We demonstrate two biases responsible for much of the performance of several LLMs on NLI tasks, which we offer as sources of false positive hallucination in model generations.

Abstract: Large Language Models (LLMs) are claimed to be capable of Natural Language Inference (NLI), necessary for applied tasks like question answering and summarization. We present a series of behavioral studies on several LLM families (LLaMA, GPT-3.5, and PaLM) which probe their behavior using controlled experiments. We establish two biases originating from pretraining which predict much of their behavior, and show that these are major sources of hallucination in generative LLMs. First, memorization at the level of sentences: we show that, regardless of the premise, models falsely label NLI test samples as entailing when the hypothesis is attested in training data, and that entities are used as "indices" to access the memorized data. Second, statistical patterns of usage learned at the level of corpora: we further show a similar effect when the premise predicate is less frequent than that of the hypothesis in the training data, a bias following from previous studies. We demonstrate that LLMs perform significantly worse on NLI test samples which do not conform to these biases than those which do, and we offer these as valuable controls for future LLM evaluation.

Sources of Hallucination by Large Language Models on Inference Tasks
Nick McKenna
Tianyi Li
Liang Cheng"
Mohammad Javad Hosseinit
Mark Johnson *Google Research Macquarie University
Mark Steedman'
!University of Edinburgh (nick mckenna, tianyi. 1i @ed. @edd.ac.u ac.uk uk
Abstract
behavior when answering user queries and the cor- responding risks in terms of bias and robustness. In particular, one LLM behavior poses a signifi- cant challenge: "hallucination, the phenomenon in which LLMs provide information which is in- correct or inappropriate, presented fact.
Large Language Models (LLMs) are claimed to be capable of Natural Language Inference (NLI), necessary for applied tasks like question answering and summarization. We present series of behavioral studies on several LLM families (LLaMA, GPT-3.5, and PaLM) which probe their behavior using controlled exper- iments. We establish two biases originating from pretraining which predict much of their behavior, and show that these are major sources of hallucination in generative LLMs. First, memorization at the level of sentences: we show that, regardless of the premise, models falsely label NLI test samples entailing when the hypothesis is attested in training data, and entities are used as "indices" to access memorized data. Second, statistical patterns of usage learned at the level of corpora: we further show a similar effect when the premise predicate is less frequent than that of the hy- pothesis in the training data, a bias following from previous studies. We demonstrate that LLMs perform significantly worse on NLI test samples which not conform to these biases than those which do, and we offer these valu- able controls for future LLM evaluation."
This paper investigates two biases driving LLM performance in natural language inference, some- times called textual entailment. This is a basic com- ponent of language understanding which is critical in applied tasks, and we offer these two biases explanations of general false positive hallucination in everyday use. We examine broader NLI, and especially directional entailments, which hold in one direction, but not both. For example, DEFEAT entails PLAY but PLAY does not entail DEFEAT. In- ferring directional entailment is more difficult than that of symmetric paraphrase, so it more deeply probes understanding.
Our approach is behavioral study of prompted LLM decision-making. We alter existing NLI datasets in targeted ways while measuring how pre- dictions change, across several major LLM families (LLaMA, GPT-3.5, and PaLM). We demonstrate two sources of LLM performance the NLI task, which we offer as explanations of general false pos- itive hallucination: (1) LLM bias toward affirming entailment when the hypothesis may be attested in the training text, including reliance on named entity identifiers; and (2) corpus-frequency bias, affirm- ing entailment when the premise is less frequent than the hypothesis.
1 Introduction
Large Language Models (LLMs) such as LLaMA, GPT-3/4, PaLM, and more (Touvron et al., 2023; Brown et al., 2020; Chowdhery et al., 2022), have been trusted by many to perform language un- derstanding in downstream tasks such summa- rization, question answering, and fact verification, among others (Zhang et al., 2023). However, due to the large-scale nature of LLM training on vast, often proprietary data, and the inherent opacity L.LM parameters, is difficult to explain their
We establish that these biases originate from the LLM pretraining objective, in which statisti- cal modeling of the natural distribution of human- generated text leads to (at the level of sentences) memorizing individual statements, and (at the level of corpora) learning typical patterns of us- age. Though they are superficially performant, our experiments show that even powerful LLMs still use unsatisfactory tools instead of robust reasoning.
"qqual contribution.
Coode and LLM outputs (LLaMA and GPT-3.5) are available at https://g thub. com Teddy-T. LIM JMM-NLI- NLI-Anal. Annaysis..

---

We present three contributions, the demonstra- tions of both factors and an analysis of their impact: (1) In a prompting scenario, LLMs respond to entailment samples according to an attestation bias, affirming entailments more readily if the hypoth- esis is attested by the pretraining text. We find that LLaMA-65B, GPT-3.5, and PaLM-540BB are respectively 1.9, 2.2, and 2.0 times more likely to wrongly predict Entail when the model al- ready asserts the hypothesis is attested, compared to when not attested. Further, LLMs recall from their propositional memory using named entities as identifying "indices," even though they are irrele- vant to the logic of the predicate inference task.
artifacts when performing directional NLI on pred- icates. We now study the behavior of much larger LMs, which have demonstrated more robust perfor- mance across NLP tasks.
Recent work has also explored L.M memoriza- tion and generalization. Carlini et al. (2023) estab- lish that LLMs are able to memorize more data than small LMs, whereas Tirumala et al. (2022) further demonstrate that L.MSs pay special attention early in training to numbers and nouns, which act as unique identifiers for individual training sentences. We also show that memories used in language infer- ence are tied to specific named entities. And while Weller et al. (2023) and Kandpal et al. (2022) find that entity frequency in training data is correlated with performance in factual recall about them, we find that entity frequency is anti-correlated with hypothetical generalization performance ($6).
(2) LLMS also rely on a simple corpus-statistic bias using relative ferm-frequencies especially when propositional memory is not available. The three LLMs are 1.6, 1.8 and 2.0 times more likely to wrongly affirm entailments if the premise has lower term frequency than the hypothesis, than when not. (3) For the NLI test samples consistent with these factors, LLM scores are misleadingly high; for NLI samples adversarial to them, LLM perfor- mance is severely degraded. We show that when labels go against the attestation bias, LLMs can be poor even near-random classifiers; for the rela- tive frequency bias, we similarly show a substantial performance decrease across all LLMs.
Bubeck et al. (2023) argue that GPT-4 under- stands language "beyond memorization". We do not disprove generalization, but we show that GPT- 4 shows the same hallucinations in Appendix F.
3 Experimental Design
We design behavioral experiments on LLMs by modifying NLI datasets with rigorous controls. We observe large behavior changes across three major LLM families due to propositional-memon effects in $5 and $6, and corpus frequency in $7. Finally, we show the impact on real performance in $8.
2 Related Work
Addressing task robustness, Poliak et al. (2018) found a range of NLI datasets contain artifacts which are learned by supervised models trained on only sample hypotheses. In our work we design a similar hypothesis-only test with LLMs, but we use it to probe model memory without any training. By conditioning on the attestation of hypotheses, we show that LLMs are inherently sensitive to attes- tation, separate from the statistical idiosyncrasies of NLI 2
3.1 Two Biases in Inference Predictions
We claim that the pretraining objective to fit the distribution of natural text leads to biases in LLM generations. We explore two such biases.
The Attestation Bias (A) is the over-reliance of an LLM on its propositional memory about a query statement. We claim that when a statement is likely to be attested in some way by an LLM's training data, it is more likely to affirm it as a conclusion in NLI tasks, regardless of any premise it is presented with. We measure the attestedness of sample by prompting the LLM to ask if the hypothesis in question is true, faase, or unknown,3 Attestation predictions are denoted by A.
Additionally, Talman and Chatzikyriakidis (2019) report generalization failure among many models supervised for NLI- models fail to gen- eralize between NLI datasets, even if the task is formatted the same, On smaller Language Models such as RoBERTa (Liu et al., 2019; 355M params), Li et al. (2022) also observed a reliance on dataset
A biased model will appear to perform well on dataset samples with entailment labels that happen to align with the bias. Table shows examples from the Levy/Holt dev set.
We speculate that a similar attestation effect could even be present in the supervised models studied in Poliak et al. (2018), which could contribute to those models' performance. We leave the investigation of this to future work.
Alternatively, LLM perplexity for statement could he used; however, this is not always available, e.g. with GPT-3.

---

Dev Sample Query: [premise] [hypothesis]
Dataset Label
Bias Prediction
Geysers common Zealand Geysers found New Zealand
Entail
A hypothesis Attested
Geysers are found New Zealand
Geysers are common New Zealand
No-Entail
A hypothesis Not -Attested
Whiskey consists chielly of alcohol = Whiskey contains alcohol
Entail
$ f(consists chiefty of) (ccotains)
Whiskey contains alcohol = Wliskey consists chiefly of alcohol
No-Entail
4 [contains) > f(consists chiefly of)
Table I: Two pairs of samples consistent with respective bias. Model predictions made on the basis of the bias will appear to predict the direction of entailment for each sample. (() maps term to its corpus frequency.
As discussed in $2, we draw inspiration from the hypothesis-ond baseline (Poliak et al., 2018), but our test only queries model memory about the hy- pothesis without any training. We describe prompt generation in detail in $4.2, with an example in appendix Table 13.
3.2 Datasets
Levy/Holt consists of premise-hypothens pairs, with task formatted: "Given [premise PJ, is it true that [hypothesis H1?" (Levy and Dagan, 2016; Holt, 2019). Each P- and H[-statement has the property of containing one predicate with two en- tity arguments, (where the same entities appear both P and If) as shown in Table 2. This targeted dataset is ideal for precisely measuring model un- derstanding of predicates, because entailment be- tween statements is decidable purely on the basis of the predicates and their attributes. We study the challenging directional subset, where entailments hold in one direction but not both.
Dasgupta et al. (2022) show similar effect in LLMs abstract reasoning tests, related to sen- tential content, and equate it to human tendencies. In contrast, we examine the risks of propositional memory on more realistic inference tasks.
The Relative Frequency Bias (+) is the use by LLMs of a simple rule for deciding entailment, calculable from corpus statistics. Entailment is affirmed if, ignoring named entities, the eventuality in premise P is less frequent in training than that in hypothesis H.
is one of the original and most difficult RTE-1
tests of NLI (Dagan et al., 2006). It is not purely directional on the basis of predicates or consistently structured like Levy/Holt, so we leave out of the behavioral experiments. However, it is a widely understood dataset, and we use it to demonstrate the impact of the two biases in general NLI in $8.
This bias is reflected in natural text: it known that nouns follow a trend of becoming more specific as corpus-frequecc decreases (Rosch et al., 1976; Caraballo and Charniak, 1999) and the same is observed for predicates (McKenna et al., 2023). Since entailments may carry from specific term to more general one, e.g. SPRINT entails RUN,
Exclusions are made of NLI datasets relating to knowledge of the world, since we aim to test LLMs on their capability to reason purely about the semantics of natural language predicates with- out relying on memorized facts. We explicitly avoid datasets such as MMLU (Hendrycks et al., 2021), Natural Questions (Kwiatkowski et al., 2019), OpenBookQA (Mihaylov et al., 2018) etc.
relative frequency can often indicate direction of entailment. However, this is an artifact of natural text and has no direct relationship with meaning.
Test samples are labeled for agreement with this bias separately from models, with examples shown in Table 1. Since LLM pre-train corpora are imprac- tically large or proprietary, we instead use Google N proxy of the natural distribution of text, and thus the distributions of these corpora. We average frequencies between the years 1950- 2019, and compare between P and H. To acquire the generic eventualities in each sample, we ex- clude any extracted entities and lemmatize predi- cate phrases; further, we reduce the effect of noise and sparsity by requiring wide margin of differ- ence between P and H frequency estimates. Fre- quency decisions are denoted by +.
3.3 Dataset Transformations
The Standard Inference Task (7) on original NLI datasets, in which entailment determinable by using general language inference on sentences. In Levy/Holt, it is determinable just by predicates. We define three dataset transformations to study the change in model responses targeted informa- tion removed. These are the randomized premise predicate setting [RandProm- and two argument transformations: generic arguments [GenAry, and type-constrained randomized arguments I Ilnndrrg
"nttps://books http://boooo...oooll.eo..o.oo com/ngrams

---

Task
Label
Dev Sample Query: [premise] [hypothesis]
I IRandPrem
Entail
George Bush was the Governor of Texas George Bush is politician from Texas
No-Entail
George Bush resided Texas George Bush politician from Texas
Table 2: From the original dataset task (/) we derive the Random Premise task (anddrem)) respecting type- constraints. A random premise is highly unlikely entail the hypothesis, so samples relabeled No No-Entail.
Task
Label
Dev Sample Query: [premise] [hypothesis]
Entail
India exports tons of rice = India exports rice
GennAr
Entail
location X exports
of food location X exports food Y
RRandArgl
Entail
Sloterdijk exports tons of oatmeal cookies = Sloterdijk exports oatmeal cookies
RRandArgt
Entail
Helsinki exports tons of Granny Smith = Helsinki exports Granny Smith
Table An original dev sample (/) is transformed by insertion of entity types (/GenArg); by real entities sampled from the 5% least frequent in NewsCrawl (I Randdrrg)) and also from the 5% most frequent (I RandAryyt)
Transformations involve first identifying the types of entities in statements, in order to con- strain entity or predicate replacements. We type each entity with one of the 48 FIGER types (Ling and Weld, 2012). such as "person," " "location," etc. First, an entity linker (Nguyen et al., 2014) identi- fies the Freebase ID (Bollacker et al., 2008) for an entity, from which we then obtain its FIGER type; we assign default type "thing" in failure cases.
identifiers, e.g. "location X and "food Y." By masking the identities of entities, this test is de- signed to remove entity information while main- taining the same entailment label, as a baseline control setting. We append unique identifiers (e.g. "X," "Y") to allow tracking of entity slots across the premise and the hypothesis.
The Random Argument Task Ra re- places original entities with other real, random en- tities of the same FIGER-typee -ttpe Like [Geurry this test is designed to create novel strings by modifying statements without changing entailment labels. But now we test model sensitivity to added extraneous information. Examples are shown Table 3.
The Random Premise Task (JRandProm) re- places the original premise predicate with ran- dom predicate, while maintaining the same entity arguments. This manipulation produces a dataset in which all samples are labeled No-Enta a 1, since two randomly paired predicates are very unlikely to be related by entailment. Thus, positive decisions by the model are false positive hallucinations." 5
We use entity type constraints here to ensure polysemous predicates maintain the same sense. For example, different sense of run is used in "[person] runs [organization]" vs. "[person] runs [software]", but between different entities of the same type, the same senses are used, so the exact entity IDs do not affect entailment labels (Yarowsky, 1993). We source new entities from NewsCrawl (Barrault et al., 2019), a decade-long span of multi multi-source news text, in which entities are typed as above. We sample new entities uni- form randomly from the 5% least common entities in NewsCrawl (IRandArg4), and the 5% most com- mon (I RondArg)) We insert the sampled entities while preserving the rest of each statement.
To maintain naturalness and grammaticality, we constrain a new predicate to have argument slots of the same types as the original premise. For ex- ample, "[medicine] is indicated for patients with [disease]" is swapped for "[medicine] does not cure [disease]". We source candidates from dev set premises satisfying the target yypecconstaaint, and sample uniform randomly. We map the original entities their respective slots in the new premise.
Examples are shown in Table 2. RRndPPemm is a good test model reliance on propositional mem- ory, since we prevent entailments while maintain- ing the attestedness of conclusions (hypotheses).
4 Querying Models with Prompts
The Generic Argument Task (/GenArg) re- places original entities with unique FIGER-typed
4.1 Models
*We manually inspected the generated random premise entrics for the Levy/Holt dataset to verify this: we found 86.6% of entries are successfully non-entailing, 3.8% unde- cided cases, and only 9.6% are unintended true entailments.
LLaMA is a recent LLM model family which rivals or surpasses GPT-3 performance while being open to scientific study. A range of model sizes

---

Ideally, an LLM with advanced language under- standing ability could perform inference in zero- shot without annotated examples. which would raise confidence that this faculty is ready for down- stream tasks. To this end, we examine each LLM in zero-shot (detailed in Appendix A), but they exhibit severely degraded, even near-random performance. are provided, and we test the largest LLaMA-65B model. LLaMA is not fine-tuned. In preliminary experiments on the Levy/Holt dataset, we found two popular fine-tuned LLaMA variants, Alpaca (Taori et al., 2023) and Vicuna (Chiang et al., 2023), perform similarly to LLaMA base models and un- derperform LLaMA-65B, we leave them out of further experiments.
We turn to few-shot, and hand-annotate mini- mal 4 examples in the style of the template, with added explanations about why the given answer is correct for each example. These examples are prepended before the query (sce Appendix A for an example). Our goal to study model behavior as conditions change, not to maximize the score on any particular dataset. Therefore, we use mini- mal 4-example setup, which we find is capable of evoking positive responses from all three LLMs each dev set, across most templates.
GPT-3 Series models are closed to deep scien- tific review (Brown al., 2020), though they are a widely-used comparison for their performance, and have been reasonably well-studied. We evaluate on text-davinci-003 (GPT-3.5), as it is the largest, and has undergone instruction- and RL.HF-fintunnl enabling interesting comparisons.
PaLM larger than GPT-3, which often claims state-of-the-art on evaluation datasets. We use the largest PaLM-540B base model, which is also only pretrained, so it serves as a further comparison point to LLaMA.
Scoring is done by converting choice A into Entail and collapsing both B and C choices into No Vo-Entail to align with Levy/Holt and RTE-1 annotation. For behavioral experiments in $5, $6, and $7, we score the model solely based on its tex- tual response. All models successfully choose one of A/B/C on all dev questions, showing compatibil- ity with the QA format.
Later GPT models (like text-davinci-003 in our experiments) have been pre-trained and fine-tuned, while base LLaMA and PaLM have only under- gone pre-training, so their contrast indicates what stage of training is responsible for the phenomena we study. Our aim is not to judge which LLM is superior, but to show the common sources of hallucination they share.
For the analysis in $8 which measures model per- formance across confidence thresholds, we convert the letter choice to probability with the mapping:
We also omit models superseded in performance by LLaMA (e.g. OPT, GPT-J, etc.), as well as products that are closed to scientific review (e.g. GPT-4, Bard, e
Sent = 0.5 + 0.5 * [[tkk A] Siok
0.5 # l[[okk E {B, CJ] # Siok
Where I is the indicator function, and Sent esti- mates the probability of Entail from a textual output (0 S Sent S 1) with token probability Stot- The linear transformation preserves the ordering of model confidences, which is sufficient for calculat- ing precision-recal curve.
4.2 Prompt Design and Evaluation
Formatting of test samples is done by inserting the premise and hypothesis into a prompt template, which used to query the model in natural lan- guage. Following this, we append a threc-way an- swer choice: A) Entailment, B) Neutral, C) Contra- diction, following the typical format in NLI (Bow- man et al., 2015).
5 Experiment 1: Attestation Bias
We begin our experiments by assessing LLMs' re- liance on their propositional memory of training text by conditioning each model's 's entailment task predictions / on its own predictions of attestation A. We do this by comparing estimated probabilities of predicting Entail conditioned on whether the hypothesis is predicted Attested or not.
Selection of the prompt template used in test is decided by the highest AUC obtained on the respec- tive dev set. We try promising templates includ- ing 5 from Schmitt and Schütze (2021), also used in other NLI w (Wehson and Pavlick, 2022).
Further, we test a setting which controls for the possibility that original Levy/Holt entailments may coincidentally refer to attested facts, which could
"We include an analysis of GPT-4 in Appendix "See Appendix A for details on prompt template selection.

---

Original Task: Conditioning
memory about the hypothesis plays part in its pre- dictions of the hypothesis given a premise, either related or random.
Inference Results (/) on Attestation (A)
100
PLEntaiis Attested)
PIEntails -Attested)
Extimated Probability %
77.6
For I, observe significantly higher proba- bility of predicting Entail when the hypothe- sis Attested. In the random premise task RandPrem- this trend continues. LLaMA, GPT- 3.5, and PaLM, respectively, show 1.9x, 2.2x, and 2.0x higher chance of falsely predicting that a random premise Entails 1s the hypothesis if it al- ready predicts the hypothesis is Attested, This false positive hallucination and its impact on NLI performance is investigated further in $8.
63.6
63.6
60
41.2
40
37.1
20
LLMMA-658
GPT-3.5
PoLM-5408
Figure 1: Exp-1. Estimated probability of predicting Entail 1 for original entries in Levy/Holt, conditioned LLMs' attestation hypotheses (A). This setting intuitive but may be subject spurious correlations.
This behavior is observed across model families (LLaMA, GPT, and PaLM), establishing that it is due to pretraining rather than Instruction-tuning or RLHF, since LLaMA and PaLM have only un- dergone pretraining. This behavior is undesirable, because model predictions on NLI tasks should be based solely on general language understanding, not prior knowledge. We may conclude that mem- ory training data is a significant contributor LLM inference, and may be an important source of hallucination.
Controlled Task: Conditioning Inference Results UkandPrem) on Attestation (A)
60
PlEntails Attested)
PIEntalls AAttested)
Estimated Probability f8
39.7
41.3
39.9
40
30
19.9
10-
5.2 Implications for Real Applications
0
LLaMA-65B
GPT-3.5
PaLM-5408
Using prior knowledge as part language infer- ence has bad implications for the use of LLMs real applications. We offer an example scenario of a guestion-answering task where user questions are answered from Knowledge Base (KB). In typical formulations of this task. if a statement in the KB (premise) entails sis), the premise may be formulated into answer. Consider KB such as a legal document or HR rule- book. Assume that the text is prepended to the user query and presented to the LLM, as in other works (Srinivasan et al., 2022). Given our findings, we might observe the LLM hallucinating answers to questions using information which is not presented in the KB, but may have been read by the LLM in text from other sources during pretraining. These answers could be illogical, contradictory, and could misrepresent the views of the KB, or other harms. Such poor use of in-context learning has already been observed in specific domains like medicine (Jimenez Gutierrez et al., 2022).
Figure 2: Exp-1. Estimated probability of predicting Entail 1 for Random-Premise entries in Levy/Holt, conditioned on LLMs' attestation of hypotheses (A). Now, predicting Enta false positive hallucination (lower is better). Models are sensitive attestation, and hallucinate more when the hypothesis is attested.
query (hypothe-
lead spurious correlation between inference and attestation scores without clearly demonstrating use memory versus true entailment. This controlled setting is the random premise task RandPrem: which converts entailments into non-entailments without altering the hypothesis. An ideal model capable of drawing inferences from information in context should detect that in the RandPreem task it is no longer possible infer the hypothesis based on the premise (even if the hypothesis is itself attested in training), and never predict Entail Thus, in RandPrem. all Entail predictions are assumed to be false positive hallucinations.
5.1 Results
In general, this risk for LLMs which (a) are deployed for tasks like QA by feeding novel text (c.g. a legal document) in in-context as part of the user query, and (b) are trained on datasets which are
With 1, I RandPrem and A predictions acquired as described $3.1, we present the conditional proba- bilities in Figures and 2. It is clear that a model's

---

Levy/Holt (Directional)
ing) precision across conditions, recall degrades sharply from original entities (4) (GPT-3.5 @92.3) to random frequent entities (oonndAryt) (GPT-3.5 [GenArg performance also degrades in this way, showing that this not a matter of poorly selected real entities, but rather a loss of information from the original dataset which models were using answer questions.
Model
Task
Precision
Recall
A-Recall
67.0
68.4
ReonArg
69.0
66.9
-1.5
@ 55.3). Generic-argunan
LLaMA
[RundArgb
64.0
-4.6
[RundArst
67.2
53.7
-14.7
62.4
92.3
/GenArg
65.1
75.7
-6.66
GPT-3.5
RandArgl
65.5
66.5
-25.8
Manddrgt
68.8
55.3
-37.0
Second, across the models, we observe a sig- nificant difference recall between the two real entity conditions 1 RanddArg. and Aradd which are both composed of unattested statements, but in- volve entities that differ in typical corpus frequency. Infrequent entities (RaaddAgg)) yield better gener- alization and a higher recall (GPT-3.5 @ @66.5) than frequent entities (7 Arndd (GPT-3.5 @55.3). These findings corroborate those from $5, that LLMs use memory as part of language inference, and additionally show that these memories are re- called using named entities acting as indices. These experiments demonstrate that too much prior expo- sure to an entity may impede model generalization when that entity discussed in novel inferences: the more a model has read about an entity during pretraining, the less capable of drawing novel natural language inferences involving it. This is the case even though the inferences do not require detailed knowledge of the entity.
72.8
76.2
GennArg 79.8
50.8
-25.4
PaLM
Randdrgb 69.5
58.7
-17.5
RandArat 70.8
52.4
-23.8
Table 4: Exp-2. Scoring model outputs different argument-replaeemmnt tasks. We indicate the highest and lowest recall score across replacement settings. Re- call decreases sharply across settings in all models.
private or otherwise infeasibly large to read man- ually, containing many facts and human opinions unknowable to both the user and modeler.
6 Experiment 2:
Entities are Indices to Memory
In $5, have established that propositional mem- ory explains a significant portion of false positives LLM inference predictions. In this section, we continue by showing the importance named enti- ties in the process of LLMs' memory recall.
Like $5, the effect is consistent across models, indicating LLM pretraining is responsible.
As described in $3.3, we manipulate the enti- ties with the [GnnArg generic argument replace- ment, and two random entity replacements, one with infrequent-entities [RandArgl and one with frequent-entities |RandArgt (examples in Table 3). By replacing arguments constrained by type, en- tailment labels are maintained; however, new sam- ples should contain novel strings not attested in pre- train corpora. We expect that an ideal, generalizing model would maintain its predictions across all conditions; flawed model utilizing the attestation bias would predict fewer Entail, since entities no longer identify these statements in training.
We show similar results on RTE-1 in Appendix B. Further, instructing LLMs to ignore proposi- tional memory in Appendix C shows little change.
7 Experiment 3: Relative Frequency Bias
We continue the conditioning experiments from $5, now exploring the relative frequency bias. Sam- ple labels for this bias are denoted by the model- agnostic & as described in $3.1. & labels the con- formance of sample predicates to the bias: &x means P less corpus-frequent than Il by mar- gin (positive class), <> means more frequent than H by the margin (negative class). To control for differences between datasets, the margin is set so that 1/3 of samples are classed as "roughly equal" (bx)), which we discard.
6.1 Results
We report results across conditions in Table 4. We observe two phenomena across all three models, aligning with the above conjecture of "flaws."
Following the observations in $6, we further ap- ply generic-argument transformation to control GGen.rgg for attestation, yielding RondPrem With the en- tities masked, models cannot recall propositional memory for this task: by re-calculating the A mea-
First, we observe that all models' behavior signif- icantly changes in the same way when original en- tities are replaced by either entity types or random real entities. Despite similar (or marginally increas-

---

Controlled Task: Concitioning Inference Results on Relative Frequency (0), with either original entities generic placeholders
8 Impact of Bias Performance
60
We have demonstrated two sources of hallucination by LLMs inference tasks. We now assess their impact on model performance quantify their risk.
P|éntails 0.)
PlEntails [..)
Eximated Probability %
50
42.6
40-
We compare LLMs' performance between NLI subsets that are consistent or adversarial to cach bias. A sample P = H? is consistent with a bias when the prediction by the bias agrees with the gold entailment label; conversely, it is adversarial to bias when the prediction by the bias disagrees with the label.
31
31.9
30-
28.5
28.2
24.6
25
20
17.9
14
10
5.8
Entities Generics LLaMA-658
Entities Generics GPT-3-5
Entitles Generics
FaLM-5408
For example, "Google bought YouTube F Google owns YouTube" consistent with the attes- tation bias of every model, because the conclusion Google owns YouTube is attested in every LLM's training data, and the sample label is Entail; "Apple owns Samsung / Apple bought Samsung" is also consistent, because its conclusion is not at- tested and the sample label reverses of these two samples are adversarial, since their respective attestedness (unchanged) does not agree with the entailment labels (now flipped). For each subset, there is substantial representation in both Levy/Holt and RTE-1 (see appendix Table 9). While earlier experiments inspected model tex- tual responses to characterize behavior change, we now use area under the precision- recall curve (AUC) to summarize model performance over a tunable confidence threshold (scoring described in $4.2), which is better for measuring practical dis- criminative power. Following Li et al. (2022), we re-scale AUC values to normalize over the label distribution, yielding AU AUCrorm values that assign random classifiers 0% and perfect classifiers 100%.
Figure 3: Exp-3. Estimated probability of predicting Ent for random- random-premise Levy/Holt conditioned on relative frequencies (4), with original (I (RandPrem) or generic (IbenArg ( entities. Predicting Entail is false positive ballucination (lower is better). Models hallucinate more often when test samples conform to the relative frequency bias (+2) than when not (4x)).
1o-EEttall The
sure with generic arguments, only 2 hypotheses are still predicted as Attested by GPT-3.5, whereas for LLaMA and PaLM, the numbers are also only 6.2% and 3.9%. Additionally, as with RandPrem: here the entailment label of each sample remains No Entaii, any Ent ail prediction is false positive hallucination.
7.1 Results
We estimate the probabilities of models predict- ing Entail conditioned on the frequency label tb between RRadPPrem and RtandPrem settings, and present the results in Figure 3. We observe clear and consistent rise of hallucination when samples conform to the bias. Namely, case of 22, models are more likely to predict Entail, even though no semantic relation exists between P and H.
We report results in Table Under the stan- dard inference task I, the performance drop from A CONSSSSTNTT to ADVERSARIAL is severe for all 3 LLMs: they deteriorate from very good classi- fiers to poor or even near-random ones. 8 This fragility from the attestation bias can be alleviated by masking entities with type-identifiers (condition [GenArg), which reduces the performance drop.
Between the two settings, with RradPreems when entities are available, this effect is moderate. On the other hand, with HondPrem when entity-based memory is blocked, we observe a decrease in the overall level of hallucination, but the separation be- tween Φ and $ becomes more drastic, to 1. 1.6x, 1.8x and 2.0x for LLaMA, GPT-3.5 and PaLM respectively. This indicates tension between A and 4: propositional memory may be used when available, and if not, the predicate pairing may be attended to more closely. Again, the & effect is observed across the three model families, reveal- ing its root in the lcrge-scale pre- trainiiinn process, rather than model peculiarities or fine-tuning.
On the other hand, with the generic arguments in Gen LLMs are forced to focus on the predi- cates in each proposition. As a result, the impact of the relative frequency bias is intensified. From the standard inference task I to GenArg, the av- erage performance drop from the cons. to adv.
*We note A predictions could possibly be influenced by model-specific idiosynerasies in prompt format. We provide an analysis in Appendix E, where we find no significant effect.

---

Levy/Holt
RTE-1
Attestation (A)
Rel. Frequency (0)
Attestation (A)
Rel. Frequency (+)
Model
Task
cons.
adv.
diff.
cons.
adv.
diff-
cons.
adv.
diff-
cons.
adv.
diff.
LLaMA
65.5
8.1
-57.4
42.1
32.3
-9.8
62.1
37.4
-24.7
51.7
GPT-3.5
85.0
10.8
-74.2
53.5
43.2
-10.3
84.6
47.5
-37.1
77.6
43.4
-34.2
PaLM
I
79.1
31,5
44.6
63.3
53.0
-10.3
87.1
83.4
-3.7
87.5
81.0
-6.5
LLaMA
GGnnAng
52.1
34.4
-17.7
55.3
34.9
-20.4
59.2
30.4
-28.8
51.7
39.4
-12.3
GPT-3.5
GGenArg
67.1
18.8
-48.3
50.4
35.0
-15.4
80.1
56.4
-23.7
79.6
49.1
-30.5
PaLM
IGenArg
58.1
46.6
-11.5
59.9
47.3
-12.6
78.1
84.4
+6.3
85.4
78.7
-6.7
Table 5: LLM performance on subsets where A/4 is consistentladversaiial to entailment labels, measured with AU (0% random chance performance). Decrease from cons adv subsets are shown in the diff. columns.
subsets w.r.t & is widened from 10.1% to 16.1% for Levy/Holt and from 14.8% to 16.5% for RTE- 1. The differences for @-consistency subsets are generally narrower than A-consistency subsets, pos- sibly because the relative frequencies require gen- eralizing from instances, and may be more difficult capture, and potentially because frequency mea- sures with Google N-gram are a crude estimate of the actual frequencies in LLM pre-train corpora.
discussed this paper, there remains residual, un- explained performance on NLI tasks. This residual might be due to other undiscovered biases pos- sibly generalising inference capability. We leave further exploration of this residual to future work. As discussed in Appendix A, we compared a range of popular LLM prompting techniques and selected the most promising approach. We ac- knowledge that there could potentially be other novel prompting techniques that could help the LLMs resist the influence of the biases discussed in this paper. We identify this as open question and advocate for future research.
9 Conclusion
Across several major L.M families and experimen- tal settings, we demonstrate two important biases in the performance of LLMs on natural language inference tasks, which may also manifest in applied tasks as hallucination. Contrary to claims of LLM general reasoning capabilities, we show that much of this performance is achieved by (1) recall of rel- evant memorizations and (2) corpus-based biases like term frequency. Since these factors are repro- duced in all models, we establish that they originate in LLM pro-training, and are not corrected during GPT-3.5 fine-tuning,
Ethical Considerations
This paper discusses two major sources of hallu- cination in LLM output when asked to perform natural language inference, which note is pability required of many downstream tasks such as summarization, question answering, etc. We show that users of LLMs may be subjected faulty judgements if the content of their request overlaps with data in pretraining. However, it is difficult to ascertain for both a user or modeler exactly what is contained in pretraining data, or how this will interact with user's query. Our proposed attes- tation query shows promise in detecting potential overlaps, but model responses in applications of these cases are not explored. Further, the relative frequency bias demonstrates a much more subtle problem of corpus distribution that is naturally in- herent to model pretraining on human generated text.
We conclude that LLMs, though powerful, use unsatisfactory tools for the basic tasks of language understanding and inference. We propose several approaches to control for these biases evaluation, and ultimately conclude that further attention on alleviating these biases are needed, before LLMS may be trusted to reason robustly about language.
Limitations
In this paper, we have discussed two prominent sources of hallucination for LLMs in natural lan- guage inference tasks. We acknowledge that this is not an exhaustive search of all the sources, where further exploration should be done in future work. We also note that after controlling for the factors
In light of these, the potential harms of LLM for drawing natural language inferences may in- clude: offering inaccurate or irrelevant information to a user's query or contradiction of information provided in-context with a user's query.

---

Acknowledgements
2023. Quantifying Memorization Across Neural Lan- guage Models. ArXiv:220.00646 [es].
This research was supported by ERC Advanced Fellowship GA 742137 SEMANTAX and the Uni- versity of Edinburgh Huawei Laboratory.
Wei-Lin Chiang. Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang. Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. 2023. Vicuma: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
References
Loic Barrault, Ondiej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Gra- ham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz. Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 Conference on Machine Trans- lation (WMT19), In Proceedings of the Fourth Con- ference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 1-61, Florence, Italy, As- sociation for Computational Linguistics.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts. Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez. Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prahhakaran, Emiiy Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev. Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Scpassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Frica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee. Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- cling with pathways.
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008, Freebase: A col- laboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, SIGMOD '08, page 1250, New York, NY, USA. Association for Computing Machinery.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large anno- tated corpus for learning natural language inference. Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 632-642, 642, Lisbon, Portugal. Association for Compu- tational Linguistics.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment chal- lenge. In Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entuilment, pages 177-190, Berlin, Heidelberg. Springer Berlin Heidelberg.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melamie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
Ishita Dasgupta, Andrew K. Lampinen, Stephanie C. Y. Chan, Antonia Creswell, Dharshan Kumaran, James L. McClelland, and Felix Hill. 2022. Lan- guage models show human- like content effects on reasoning. ArXiv:2207.077 17705 [cs].
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. 2021. Measuring massive multitask language understanding. Proceedings y of the International Con- ference on Learning Representations (ICLR).
Séhastien Bubeck, Varun Chandrasekaran, Ronen El- dan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lund- berg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of Artificial General Intelligence: Early experiments with GPT-4 ArXiv:2303.1277 12712 [cs].
Xavier Holt. 2019. Probabilistic Models of Relational Implication. arXiv: 12048 [cs, stat). ArXiv: 1907. 12048.
Sharon A. Caraballo and Eugene Charniak. 1999. De- termining the specificity of nouns from text. In 1999 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Cor- pora.
Bernal Jimenez Gutierrez, Nikolas McNeal, Clayton Washington, You Chen, Lang Li, Huan Sun, and Yu Su. 2022. Thinking about GPT-3 in-context leam- ing for biomedical IE? think again. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4497-4512, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski,
Katherine Lee, Florian Tramer, and Chiyuan Zhang.

---

Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2022. Large language models struggle to learn long-tail knowledge.
Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. ArXiv:2203.02155 [es].
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee. Kristina N. Toutanova, Llion Jones, Ming- Wein-Wei Chang. Andrew Dai, Jakob Uszkoreit, Quoc Lc. and Slav Petrov. 2019. Natu- ral questions: a benchmark for question answering research. Transactions of the Association of Compu- tational Linguistics.
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language infer- ence. In Proceedings of the Seventh Joins Confer- ence on Lexical and Computational Semanties, pages 180-191, New Orleans, Louisiana, Association for Computational Linguistics.
Eleanor Rosch, Carolyn B Mervis, Wayne D Gray, David M Johnson, and Penny Boyes-Braem. 1976. Basic objects natural categories. Cognitive Psy- chology, 8(3).382-439
Omer Levy and Ido Dagan. 2016. Annotating Rela- tion Inference in Context via Question Answering. In Proceedings of the 54th Annual Meeting of the A Association for Computational Linguistics (Volume 2: Short Papers), pages 249-25-222222 200, Berlin, Germany. Association for Computational Linguistics.
Martin Schmitt and Hinrich Schütze. 2021. Language Models for Lexical Inference in Context. In Proceed- ings of the 16th Conference of the European Chap- ter of the Association for Computational Linguistics: Main Volume, pages 1267-1280, Online. Association for Computational Linguistics.
Tianyi Li, Mohammad Javad Hosseini, Sabine Weber, and Mark Steedman. 2022. Language Models Are Poor Learners of Directional Inference. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 903-921, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Krishna Srinivasan, Karthik Raman, Anupam Samanta, Lingrui Liao, Luca Bertelli, and Michael Bendersky. 2022. QUILL: Query intent with large language mod- els using retrieval augmentation and multi-stage dis- tillation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 492-501, Abu Dhabi, UAE, Association for Computational Linguistics.
Xiao Ling and Daniel S. Weld. 2012. fine-grained en- tity recognition. In Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intelligence, AAAI 12, page 94-100. AAAI Press.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretrain- ing Approach. 1xXiv:1907.11692 [es]. ArXiv: 1692.
Aarne Talman and Stergios Chazzikyraakiiis, 2019. Testing the Generalization Power of Neural Network Models across NLI Benchmarks. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 85- 94, Florence, Italy, Association for Computational Linguistics.
Nick McKenna, Tianyi Li, Mark Johnson, and Mark Steedman. 2023. Smoothing Entailment Graphs with Language Models. In Proceedings of the 3rd Confer- ence of the Paciffc Chapter of the Association for Computational Linguistics and the 13th Interna- tional Joint Conference on Natural Language Pro- cessing. Association for Computational Linguistics.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stan-
ford alpaca: An instruction-followinn llama model. https: eetps://githhue /ttphtb..tb.t//////// con/ ttt/:/attuuuuuuu lab/ st wwwiford_alpacc
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018 Can suit of armor conduct elec- tricity? new dataset for open book question answer- ing. In Conference on Empirical Methods in Natural Language Processing.
Kushal Tinumala, Aram Markosyan, Luke Zettlemoyer. and Armen Aghajanyan. 2022. Memorization with- out overfitting: Analyzing the training dynamics of large language models. In Advances Neural Infor- mation Processing Systems, volume 35, pages 38274- 38290. Curran Associates, Inc.
Dat Ba Nguyen, Johannes Hoffart, Martin Theobald, and Gerhard Weikum. 2014. Aida-light: High- throughput named-entity disambiguation. In LDOW.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière. Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models.
OpenAI. 2023.
GPT-4 Technical Report.
ArXiv:2303.08774 [cs].
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller,
Albert Webson and Ellie Pavlick. 2022. Do prompt- based models really understand the meaning of their

---

prompts? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 2300-2344, Seattle, United States. Association for Computational Linguistics.
6. is not the case that [НYPOTHESIS] let alone that [PREMISE].
7. JAYPOTHESISJNEG. which means that [PREMISE] NEG-
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Fdd Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 14824-248371 Curran Associates, Inc.
8. PREMISE]NECE because [HYPOTHESIS]
In preliminary experiments with GPT-3.5, we ob- served that LLMs are not responsive to the 3 contra- positive prompts from Schmitt and Schütze (2021) (colored gray), performing at random. We also observed that prompt number 5 from Schmitt and Schütze (2021) also consistently underperforms the other 4 templates, so we use the remaining tem- plates (namely, template no. 1, 2, 4) our final candidate set.
Orion Weller, Mare Marone, Nathaniel Weir, Dawn Lawrie, Daniel Khashabi, and Benjamin Van Durme. 2023. "according -.- prompting language models improves quoting from pre-training data.
David Yarowsky. 1993. One sense per collocation. In Human Language Technology: Proceedings of a Workshop Held Plainshoro, New Jersey, March 21-24, 1993.
In- Context Examples have been widely used for interactions with LLMs since Brown et al. (2020). Further, Wei et al. (2022) has demonstrated that including chain-of-thougle explanation, namely step-by-step explanations, in the in-context exam- ples, helps LLMs perform reasoning tasks. On the other hand, Ouyang et al. (2022) has suggested that instruction-tune tunee LLMs are also capable of performing tasks in zero-shot, without exposure to any in-context examples.
Tianyi Zhang, Faisal Ladhak, Esin Durmus, Perey Liang. Kathleen McKeown, and Tatsunori B. Hashimoto. 2023. Benchmarking large language models for news summarization.
Prompt Format Selection
In prompt-based interactions with the LLMs, sev- eral types of context information could be added to help models produce accurate and robust predic- tions. We attend to two design choices in prompt engineering: prompt templates and in-context ex- amples.
We compared zero-shot and few-shot in our pre- liminary experiments with LLaMA and GPT-3.5 on Levy/Holt directional dev set. Following Touvron et al. (2023), for zero-shot, description of the task each test sample; for few- shot, we prepend a minimal examples with ex- planations. Instantiated prompts the two settings are demonstrated in Table 13. Here we report the dev set results with the best pest-performing templates.
prepend a textual
Prompt templates are known to have a direct and sometimes decisive impact on LLM behavior. As such, we carefully select a range of clear and concise templates promising candidates. As discussed in $4.2, we run each template through the dev sets of each dataset, and select the template with the best discriminative power according to AUC scores (similarly to $8). The candidate set of templates includes concise templates we wrote:
We found that for the two pre-trained LLMs, namely, LLaMA and PaL.M, zero-shot performance on the Levy/Holt directional dev set is near-random. at 56.6% and 61.5% AUC respectively (random is 50%): with 4 in-context examples, the models be- gin to exhibit non-trivial behavior, with 65.0% and 80.2% AUC, respectively. This is not surprising. since pre-trained LLMs without instruction fine- tuning should not be expected to perform complex tasks zero-shot. For GPT-3.5, the performance is still much lower in zero-shot, at 64.5%, compared to 74.6% in few-shot.
1. If [PREMISE], then [HYPOTHESIS].
2. [PREMISE], so |HYPOTHESIS].
3. [PREMISE] entails [HYPOTHESIS].
We also considered the 5 prompt templates used in bias work on LMs for textual entailments (Schmitt and Schütze, 2021):
As discussed in $4.2, ideally we would like LLMs to have zero-shot natural language abili- ties readily available for downstream tasks. How- ever, in light of this observation, our primary ex- periments are conducted in the few-shot setting
4. [PREMISE], which means that [HYPOTHESIS)
5. [HYPOTHESIS], because [PREMISE].

---

Model
Task
Precision
Recall
R-Rcall
guments. We further observe that across all three LLMs across both datasets, models consistently achieve worse recall using high-frequency entities than low-frequency entities, supporting the claim that increasing the frequency of entity occurrence in training data impedes generalization.
74.5
52.5
LLaMA
[Gen
70.9
57.3
+4.8
[RundArgk
66.9
60.5
+8.0
RRundArot
70.6
$1.5
-1.0
80.6
96.5
G Arg Arg
79.7
91.3
-5.2
GPT-3.5
RandAArl
80.1
82.5
-14.0
Different from in Levy/Holt, observe some noise in LLaMA'S predictions; the recall on the original task actually lower than the generic argu- ment condition and the low-frequency entity condi- tion. We note that overall, LLaMA is the weakest LLM tested in this experiment on both Levy/Holt and RTE-1, and that its performance on RTE-1 1 is particularly low. We suggest that the increased dif- ficulty of RTE-1 over Levy/Holt (due to having much more linguistic variation) is simply too com- plex for LLaMA, which is neither the largest LLM tested, nor instrructionfinntunne
1 [RundArgt
81.9
80.5
-6.00
90.3
84.0
[GenArg
92.3
71.5
-12.5
PaLM
1 Randdrak
87.8
82.5
-1.5
RandAArat 88.2
82.0
-2.0
Table 6: Scoring model outputs in different conditions RTE-I. We indicate the highest and lowest recall score across replacement settings.
throughout, in order to better explore the abilities of these LLMs.
We also observe a smaller gap between PaLM's recall rates across dataset conditions, though the gaps are consistent with our claims. While the model appears able to generalize to conditions in which random real arguments are inserted, recall on the generic argument condition significantly degraded. Failure on this control condition indi- cates that the model may not be generalizing as well as the other conditions would imply.
RTE-1 Results For Experiment 2: Entities are Indices to Memory
The RTE-1 dataset contains complex natural lan- guage statements with varied linguistic features, so predictions about entailment are not decidable only on the basis of contained predicates. However, RTE-1 is a difficult challenge set for models, and interesting to compare to in the broader domain of NLI. Though the sentences are much more com- plex, we are able to conduct an analogous experi- ment as in $6 by first identifying spans of named entities and their respective entity types, then re- placing the entities with new ones. As before, compare model scores on the original dataset to three test conditions: generic arguments ("location X", "person Y", etc.), sampled low-frequency en- tities constrained to the same type, and the same for high-frequency sampled entities. Since only the entities in each statement have been altered, the entailment labels between premise/hypothesis pairs remain unchanged, and an ideal model capa- ble of generalizing inference would make the same predictions across dataset conditions. Results are shown in Table 6.
C
The Ineffectiveness of Instructing LLMs to Stop Conditioning on Attested Information
In $5 and $6, we showed that entailment predic- tions from LLMs strongly biased by their pre- dictions on the attestation of hypotheses. We won- dered whether there are intuitive prompt engineer- ing techniques to steer behavior away from at- tending to attestation.
Towards this goal, we experimented with prepending a brief task description the few-shot prompts in part B of Table 13, explicitly instructing the models to ignore the attestedness of individual statements: Please check the entailments between the following hypothetical statements. Ignore the veracity of these statements.
We observe similar trends to those reported on Levy/Holt. GPT-3.5 performs very consistently be- tween Levy/Holt and RTE-1 in terms of degrading recall when information is changed in each sample, We observe that model performance is worse than the original dataset when using generic arguments, and worse still using type-cope constraitd random ar-
We replicated the experiments in $5 and $6 with GPT-3.5, since GPT-3.5 is an instruction-fincte model trained to be responsive to prompts. where the other two LLM families are only pre-trained. Despite having been instruction-finetuned the re- sults with GPT-3.5 show only marginal improve- ments in model behavior.

---

task
GPT-3.5
Instructed to Ignore Atfestedness
Not Instructed
P(Entail
Attested)
74.3
77.6
P(Entail
Attested)
57.8
63.6
RRndPreem
P(Entail
Attested)
39.0
41.3
RandPreom
P(Entail
Attested)
17.6
18.8
Table 7:
We estimate the probability of positive predictions of Entail in and RondPrem tasks respectively given that the hypothesis is attested, namely A = Attested. Not instructed results are copied from Figure 2 and listed here for ease of comparison; also note all TRandPrem
Entail predictions are false positives.
Levy/Holt (Directional)
GPT-3.5 Condition
Task
Precision
Recall
A-Recall
64.9
90.8
0
ArenArg Arg
73.5
69.3
-21.5
Few-shot, instructed to ignore atlestedness.
[Rand Argl
64.6
68.4
-22.4
1 RRundArgt
67.5
58.1
-32.7
62.4
92.3
0
Few-shot, no instructions.
GennAr
65.1
75.7
-16.6
Araddrgl
65.5
66.5
-25.8
RondArgt 68.8
55.3
-37.0
Table 8: GPT-3.5 predictions when models explicitly instructed avoid taking the attestedness of individual statements into account. In the upper half are the instructed behavior, and in the lower half are the regular few-shot behavior Table 4. Differences recalls remain at similar scale, with precision again stable, where the benefit from the explicit instruction is marginal.
In Table 7, we show that instructing GPT-3.5 to ignore attestation does not help narrow the gap between A = Attested and A = -Attested; instead, probabilities of predicting Entail went down by similar amounts, indicating that the model is becoming slightly more conservative in predict- ing positives when instructed to ignore attestation, but not in principled manner.
subset separations are based on A predictions from individual models, which can be noisy, subject to model-specific idiosyncracies such as trigger words or certain syntactic structures in the prompt, etc. To verify that the performance gaps in A- consistency subsets that we observe in $8 comes from predicted attestedness and not some idiosyn- crasy, we experiment with another pair of subsets based on consensus attestation instead of individu- ally predicted attestation.
Further, as shown in Table 8, despite the ex- plicit instruction, recall still drops at similar scales when arguments are randomly replaced with the same sets of frequent/infrequen replacement enti- ties as before. Since GPT-3.5 5 has been instruction- finetuned to respond prompts, its failure means eradicating such biases from model outputs is difficult task, one that needs further research atten- tion.
We use a majority vote among the three independently-trainee LLMs to approximate con- sensus attestation. The approximation is denoted as A. This because any model-specific idiosyn- crasies should not be shared between LLMs in- dependently trained from different source corpora in general. Therefore, with the majority vote, we reduce this noise and acquire predictions on the consensus attestation of statements.
D Statistics of Consistency Subsets
Performances of LLMs between A-consistency subsets are listed in Table 10. Gaps between the Á-consistency subsets that are larger than A- consistency gaps colored red; those narrower than A-consistency gaps are colored green. is clear that the gaps are consistent between A/A consistency experiments, where the gaps are even larger on many occasions. This confirms that the
The statistics of consistency subsets are presented in Table 9.
E The Reliability of A Measure and Its Relation to Consensus of Attestation
The A-consistency subsets most directly capture the impacts of the attestation bias. However, these

---

# Entries
Levy/Holt
RTE-1
LLaMA
GPT-3.5
PaLM
LLaMA
GPT-3.5
PaLM
VONSISSENT
955
947
999
479
447
480
ADVERSANIAL
829
837
785
321
353
320
FUONSISTENT
972
286
AARVEISARIAL
220
247
Table 9: Subsets defined by the consistency between entailment label L and either A (hypothesis attestation prediction from each LLM) or (model-agnostic relative frequency bias). CONSISTENT subsets are where agrees with A/D, ADVERSARIAL. subsets where I. disagrees with A/O.
Levy/Holt
the majority vote among LLaMA, GPT-3.5 and PaLM, to approximate A predictions from GPT-4 itself; for the relative frequency bias, we keep the O measure for approximating corpus-frequency of terms.
Model
Task
Acons.
Àads.
diff.
LLaMA
65.3
6.5
-58,8
GPT-3.5
70.8
23.5
-47.3
PaLM
80.7
28.3
-52.4
LLaMA
AGenArg
54.4
29.6
-24.8
Because GPT-4 is commercial service and does not provide logit confidence with their discrete pre- dictions, Aoom values could not be calculated. Therefore, we are forced to report the F-1 scores at the binary prediction point of confidence. As results in Table 12 show, we observe the same trend as in $8: for the subset adversarial to each factor, GPT-4 performance also drops substantially.
GPT-3.5
Gennrr
56.2
35.5
-20.7
PaLM
GenArg
59.3
40.1
-79.2
Table 10: LLM performance on Levy/Holt subsets where Attestation Á Consisten//Adversaraall labels, measured with AUCrorm (0% random chance performance). Performance drops from Acons to Ande presented in the diff. columns, sharper decreases than A-comparisons in Table 5 are colored red, milder ones are colored green.
the
This experiment is designed to provide more text for the two biases discussed in the paper and NOT to compare GPT-4 with other models; how- ever, we can conclude that GPT-4 is subject to the same fragilities as the other LLMs w.rt. the two ases, where our conclusions and recommendations also apply.
performance gaps in A-consistency experiments be credited the attestation bias, rather than msdel-spe idiosyncrasies.
It also to noted that, since the fc-oonsitenncy subsets are separated based on the model-agnostic criterion ④ model-specific idiosyncrasies are not a problem for æ-consistency comparisons.
G Dataset Statistics and Dev Set Performance
In the paper, have examined the behavior and performance of three major LLM families on two NLI datasets: Levy/Holt and RTE-1.
F Impacts of Bias on GPT-4 Performance
GPT-4 (OpenAI, 2023) is recent, strong LLM claiming SOTA performance on various NL. tasks. Due to its closed-source nature and the impossibil- ity of fully tracking the sources of its behaviors. refrain from reporting results with it in the main content of this paper.
The directional portion Levy/Holt dataset contains 630 entries in its dev set, and 1784 entries in its test set; the RTE-1 dotaser contains 567 entries in its dev set, and 800 entries in its test set. Each dataset has 50%/50% class distribution between Entail and No-Entail (for RTE-1 dev set, the numbers of entries in the label classes differ by 1).
However, in order to provide a richer con- text for the altestation bias and the relative fre- quency bias, in this section we report the perfor- mance differences of GPT-4 between subsets con- sistent/adversarial to the two biases.
In Table 11, we report dev set performance and the best prompt template used for each model on each dataset. Note that no training is involved in
As a light-weight experiment, we elicit GPT-4 predictions in the original / task the zero-shot setting, and re-use subsets from experiments in $8. Specifically, for the attestation bias, we use
"https://github htwp://...cco///ocseeeee.aa hatterrr//ssthh.oooomoossee// pattgraplLevval/zre//sttte
saattps://wwt kagg e. htt.///a//e/ttttaeeeeeaeeaa nitkdata/rte--ae teaettarrteeeeee

---

Levy/Holt
RTE-1
Model
Task
Best tplt. ID
DEV set AUCnorm
Best tplt. ID
DEV set AU'Cnom
#4
30.0
#3
62.5
LLaMA
Guunrrg
#1
34.6
#3
$2.3
(Rand
51.3
[RandArst
26.3
#3
43.8
49.2
#3
74.8
GPT-3.5
/GenArg
39,8
#3
64.8
1
[Rand Arat
#1
43.4
#3
63.6
34.2
#3
66.0
60.9
#4
84.5
GeenArg
48.1
#4
79.4
PaLM
I Argd
#1
43,6
#3
79.8
Arad
#1
35.3
#3
78.3
Table 11: LLM dev set performance on the two datascts, measured with AU AUCrorm (0% = random chance
performance). AUC is calculated using estimated model scores in $4.2 and then normalized into AUCrorm- We select the highest scoring template on each dev task (shown in this table) and use this in the corresponding test set evaluation (shown in the main text).
F-1 score
Task
Levy/Holt
Acoss
AAde
random baseline
70.3
62.0
GPT-4
I
85.1 (+14.8)
(+5.6))))
4conss
Adv
random baseline GPT-4
66.7
66.7
74.6 (+7.9)
69.7 (+3.0)
Table 12: LLM performance on Levy/Holt subsets where Attestation A is Consistent/Adversarii labels, measured with F-1 score. random buseline is the highest F-1 score from a random classifier, by reaching random precision and 100% recall. For each GPT-4 score, we also show the improvement over random (in parentheses).
the
this paper, and prompt template selection is the only hyper-parameter tuned on the dev sets. These selected best prompt templates are then used on the respective test sets, where the results used for the analysis throughout the paper.
For random- random-premise experiments, AUC values cannot be meaningfully calculated because gold labels are always No-Entail. For these ex- periments, we use the most frequently-selectee prompt template on each dataset, namely template #1 for Levy/Holt dataset, and template #3 for RTE- dataset.

---

A. Zero-shot Example Instantiated Prompt
Please check the entailments between the following statements.
If kanamycin kills infections, then kanamycin is useful in infections.
A) Entailment
B) Neutral
C) Contradiction
Few-shot Example Instantiated Prompt
If Google bought Youtube, then Google owns Youtube. Entailment
Neutral
Contradiction
Answer: A) Entailment. Owning consequence of buying.
Google owns Youtube, then Google bought Youtube.
A) Entailment
B) Neutral
C) Contradiction
Answer: B) Neutral. Owning does not imply buying, the ownership may come from other means.
John went to the mall, then John drove to the mall.
A) Entailment
B) Neutral
C) Contradiction
Answer: B) Neutral. John may have gone to the mall by other means.
If John drove to the mall, then John went to the mall.
A) Entailment
B) Neutral
C) Contradiction
Answer: A) Entailment. Driving is means of going the mall. If ephedrine is widely used in medicine, then ephedrine is used in medicine.
A) Entailment
B) Neutral
C) Contradiction
Answer:
C. Hypothesis-only Example Instantiated Prompt
Google bought Youtube.
A) True
B) Unknown
C) False
Answer: A) True.
Yoshua Bengio likes oak trees.
A) True
B) Unknown
C) False
Answer: B) Unknown.
The sun rises from the west.
A) True
B) Unknown
C) False
Answer: C) False.
ephedrine is used in medicine.
A) True
B) Unknown
C) False
Answer:
Table 13: Example instantiated prompts Zero-shot / Few-shot settings, for the sample "PREMISE: [ephedrine is widely used in medicine), HYPOTHESIS: (ephedrine is used in medicine]". The few-shot prompts in part B are used throughout the main experiments this paper. We also present an example the prompts we use the hypothesis-onf, A measure as described in $3.1.


Reviews:
[{"review_id": "aHtYsPHtWX", "reviewer": "Reviewer_Fdep", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and thoughtful comments on the paper!\n\nRegarding the comment about limited contribution, we note that we have experimented with 3 different LLM families (trained independently), on 2 datasets, with 8 prompt templates. We argue this is sufficiently representative of the current generation of LLMs on the NLI task. For the anonymous LLM, although we are unable to reveal its identity until the end of the anonymity period, we are able to confirm that it is a SOTA LLM of a larger scale than GPT-3. We focus on the task of textual inference because it is fundamental to the many NLP tasks that require more than mere approximation to memorized training data. Examples are: retrieval-aided question-answering from documents that entail but do not state a direct answer to the question; or multi-document summarization, where redundancy of statements in one document that are entailed by others must be detected.\n\nWe agree that adding more illustrative examples is a great idea, and can help explain the relative frequency bias. We already have examples for all experimental conditions in Tables 1 and 2, but we believe that adding a dev set example which illustrates when each bias aligns with the correct label will help introduce them much better in S3.1. We will add these:\n\n---\n\n**Attestation:**\n\nPrem: Amon became the god of Egypt\n\nHyp: Amon was worshiped in Egypt\n\nLabel: Entail\n\nHyp attested? Yes\n\n---\n\nPrem: Amon was worshiped in Egypt\n\nHyp: Amon became the god of Egypt\n\nLabel: No-Entail\n\nHyp attested? No (Egypt was polytheistic in the time of Amon)\n\n---\n\n**Frequency:**\n\nPrem: Whiskey consists chiefly of alcohol\n\nHyp: Whiskey contains alcohol\n\nLabel: Entail\n\nfreq(“consists chiefly of”) < freq(“contains”)? Yes\n\n---\n\nPrem: Whiskey contains alcohol\n\nHyp: Whiskey consists chiefly of alcohol\n\nLabel: No-Entail\n\nfreq(“contains”) < freq(“consists chiefly of”)? No\n\n---\n\nWe also plan to further clarify the introduction to the relative frequency bias by elaborating on the existing reference to McKenna (2022). Very infrequent predicates tend to be very specific (e.g. “perambulate,” “hike”) compared to very frequent predicates which tend to be more semantically general (e.g. “walk,” “move”). A specific predicate may entail a general one (e.g. “hike” entails “walk”) but the reverse is not possible (“walk” does NOT entail “hike”). Language Models are known to be sensitive to frequency, and we show this accounts for some capability to detect entailment.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Overall I think the the paper presents a nice and creative methodology for showing that biases may lead LLMs to hallucinate. At the beginning I had a hard time understanding the main contribution of the paper; however, once understood, it believe the conclusions made by this work are significant to the NLP field.", "reasons_to_reject": "While the contribution may be considered as significant, I believe that it might be a bit limited (only NLI, only 3 LLMs) to be considered for the main EMNLP stage. Also I find the paper a bit hard to follow. There are not enough examples, and it took me a lot of time to understand the methodology while I believe it could be described in a better way (for example, relative frequency bias should be better explained). Other than that I have no major concerns.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a behavioral study of 3 LLMS which shows that some biases are major sources of hallucination in generative LLMs. They define two biases for which they modify two existing NLI datasets in a way in which they can control the bias during generating the correct label for NLI instances.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "It is hard to understand directly from the abstract what is the main contribution of the paper. I suggest adding the bottom line qualifications to the abstract to make it clearer."}, "extra_text": {}}, {"review_id": "SUxmLppr6M", "reviewer": "Reviewer_4oz7", "summary": "", "questions": "What is the relation between the hypothesis-attestment heuristic and the already well known hypothesis only baseline? do you see these findings as exposing the mechanism behind the hypothesis only baseline?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and thoughtful analysis of the paper! To answer the question: \n\nThe hypothesis-only baseline of Poliak et al (2018) trains a model to make judgements on NLI datasets based only on the hypothesis, without considering the premise. Through training it is able to detect statistical artifacts like word choice and grammaticality and make predictions much better than chance. \n\nOur hypothesis-attestation judgements are obtained without any fine-tuning, simply by asking the model whether the hypothesis statement is true, false, or uncertain. The models perform better on samples which align with this bias (Entail when hypothesis is judged true, No-Entail when hypothesis is not), than on samples that do not.\n\nBetween the two lines of work, the hypothesis-only baseline of Poliak at al focuses on detecting biases in NLI training datasets which could be exploited by models, when trained on these datasets; on the other hand, our hypothesis-attestation conditioning indicates that the class of LLMs studied is inherently sensitive to attestation of hypotheses, separate from the idiosyncrasies of particular datasets.\n\nHowever, we believe it may also be possible that the models tested in Poliak et al, being BERT-based models, could also be subject to the same attestation bias as observed with larger LMs in this paper. This attestation bias could be contributing to their hypothesis-only performance in addition to the explicit dataset artifacts which are exposed in their paper. It would be interesting to test for this in future work using an attestation control on those models and datasets.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper shows an important and easy-to-miss limitation of current LMs, highlighting the need to disentangle performance on specific datasets form performance on the general task of interest.\n\n- Very elegant experimental design, both showing evidence for the two new heuristics, and isolating the mechanism by which the model memorizes.\n", "reasons_to_reject": "I don't see any reason to reject the paper.\n\nOne technical limitation is the use of proxies for occurrence in the training data. Hopefully indexing the pretraining data of large LMs would enable measuring to what extent these proxies actually reflect occurrence in the training data. ", "questions_to_authors": "What is the relation between the hypothesis-attestment heuristic and the already well known hypothesis only baseline? do you see these findings as exposing the mechanism behind the hypothesis only baseline?", "paper_topic_and_main_contributions": "The paper studies the origins of hallucination and impaired performance in the NLI task. It identified two heuristics learned by pretrained models: (1) they tend to predict entailment when the hypothesis (alone) it attested in the training data (2) they tend to predict entailment when the premise is less frequent in the training data than the hypothesis, reflecting a (non-causal) tendency in NLI data to infer the more abstract from the more concrete. Further, it is shown that it is the entities that serve as anchors in memorisation. Overall, this paper sheds light on important issues behind the apparent good performance of pretrained LMs on NLI data. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ocpSwMg3Tn", "reviewer": "Reviewer_oXrA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and thoughtful comments on the paper! We’d like to respond to your 2 concerns in “Reasons to Reject”:\n\n#### **Question 1: How do we ensure No-Entail labels in the Random Premise Experiment?**\n\nIn initial experiments, manual inspection of a sample of random-premise generations looked acceptable to the authors. But this is a valid concern, and we have since conducted an additional manual expert annotation over the entire set of 1784 random-premise entries to filter out any generated ENTAIL samples. Our new manual expert annotation identified 171 ENTAIL entries (9.8% of generated entries) and confirmed 1545 NO-ENTAIL entries. The remaining 68 are marginal cases which could be interpreted in either direction depending on the reading. With the limited time-frame in mind, we simply removed the 171+68=239 entries of ENTAIL / marginal ENTAIL from the dataset, and recalculated the probabilities.\n\n**The revised probabilities leave the conclusions of the paper unchanged.** If accepted, we will add them in an appendix. Specifically:\n\n1. First, we note that the main paper claim is observed strongly in the original dataset itself (Figure 1): all 3 LLM families affirm inference questions more often when the hypothesis is attested by training data. The random-premise transformation is offered mainly as a further control (L398), on which the models continue to make many positive judgments, showing that they are answering questions using hypothesis attestation, instead of considering if the (now changed) premise entails it.\n\n2. With the new cleaned data subset, we re-estimated the probability values when conditioning on the Attestation bias (Figure 2), which we present here in the same left-to-right order. We note that the effect size is very strong (the gaps are wide between attested/non-attested probabilities of predicting ENTAIL).\n\n| Estimated Probability        | LLaMA       | GPT-3.5     | Anon. LM   |\n| ---------------------------------- | -------------- | -------------- | --------------- |\n| Before Cleaning (1784)  &emsp;   | 39.7 / 20.7 &emsp;| 41.3 / 18.8 &emsp; | 39.9 / 19.9 |\n| Clean NO-ENTAIL (1545) | 35.9 / 16.9 | 34.3 / 11.7 | 33.0 / 14.8 |\n\n3. We also re-estimate the probabilities for conditioning on the Frequency Bias (Figure 3). The effect size is smaller, but it is consistent with all other results. We note that among the 239 entries removed for possible ENTAIL, many of their hypothesis predicates are high-frequency/semantically general, since many randomly selected premises will entail something generalized. E.g. 41 entries with hypothesis `lives in`, 14 `contains`, 8 `traveled to`, 7 `is in`, etc. Removing these leads to a distribution shift which especially affects the creation of subsets defined by the Relative Frequency bias conditioning (so results are less comparable before/after cleaning). We report the results (with the same left-to-right order as Figure 3).\n\n| Estimated Probability        | LLaMA                           | GPT-3.5                        | Anon. LM        |\n| ---------------------------------- | ------------------------------- | ------------------------------ | ------------------- |\n| Before Cleaning (1784) &emsp;    | (32.4 / 28.5), (28.2 / 17.9) &emsp; | (42.6 / 31.0), (24.6 / 14.0) &emsp; | (31.9 / 25.0), (11.8 / 5.8) |\n| Clean NO-ENTAIL (1545) | (27.7 / 26.4), (18.6 / 16.6) | (30.7 / 26.7), (14.1 / 11.5) | (25.3 / 20.1), (5.4 / 4.4) |\n\n#### **Question 2: How does Argument Replacement Show Memorization in Models?**\n\nLLM performance using original named entities degrades when we switch to the generic- and random-entity datasets, which demonstrates LLMs’ reliance on memory about the original entities (which tend to be “expected” in context), and LLMs’ inability to generalize about valid but unexpected entities. These transformations create samples where the predicates/entailment label remain the same as the original dataset, but each individual proposition becomes surprising to the LLMs. \n\nIn the generic argument task, each individual proposition is meant to have neutral attestation, since “true” or “false” labels can rarely be assigned to claims about generic entity types (e.g. `PERSON resided in LOCATION` is neither true nor false). In the random argument task, each individual proposition may have neutral or negative attestation (e.g. `Helsinki exports Granny Smith Apples` is very unexpected and likely “false”/unattested by training data). With more frequent pairs of entities, the LLMs have a weaker ability to reason about novel information, because the mentions of these entities that are read during pre-training interfere. Therefore, we can generally observe a decreasing trend in recall from GenArg to RandArg⬇️ to RandArg⬆️, especially between RandArg⬇️ and RandArg⬆️, as shown in Table 3.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe theory proposed in this paper is of important value in the task of characterization and mitigation of hallucinatory behavior in LLMs.", "reasons_to_reject": "1.\tIt was quite unclear how the experiments performed in the work to corroborate the authors’ theory did that. In the random premise task – how could the authors ensure that the random predicate indeed resulted in NO-ENTAIL? I understand that such random sampling has a very small probability of resulting in something that is not NO-ENTAIL, but given that many predicates have synonyms, and other predicates whom they entail, and hence by proxy the current hypothesis might also entail, it feels like it is crucial to ensure that indeed NO-ENTAIL was the case for all the instances (as this is not the train set, but rather the evaluation set). \n2.\tAdditionally, it was not clear how the generic argument task and the random argument task proved what the authors claimed. All in all, the whole dataset transformation and the ensuing experimental setup felt very cumbersome, and not very clear.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper addresses the phenomenon of hallucinations in LLMs when performing inference tasks. The paper specifically focuses on cases when LLMs wrongfully predicts entailment. The authors suggest that the origin of such hallucinations stems from two factors: the presence of the hypothesis in the pretraining data (the attestation bias),  and the statistical patterns of the pretraining data (the relative frequency bias), which the authors demonstrate in their experiments.\n\nThe paper’s main contribution is the proposed source of inference hallucinations, which could lead to better training and deployment of LLMs, in order to mitigate this behavior \n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper establishes two sources of hallucinations in LLMs: memorization and statistical patterns. While reviewers see merit in the work, they also express reservation about whether the conclusions are sufficiently supported.
Decision: Accept-Findings