Strong hallucinations from negation and how to fix them (2024)

Abstract: Despite great performance on many tasks, language models (LMs) still struggle with reasoning, sometimes providing responses that cannot possibly be true because they stem from logical incoherence. We call such responses strong hallucinations and prove that they follow from an LM’s computation of its internal representations for logical operators and outputs from those representations. Focusing on negation, we provide a novel solution in which negation is treated not as another element of a latent representation, but as an operation over an LM’s latent representations that constrains how they may evolve. We show that our approach improves model performance in cloze prompting and natural language inference tasks with negation without requiring training on sparse negative data.

heading: 1 Introduction
text: This paper investigates and addresses reasoning problems for language models (LMs) that stem from their representation of the meaning of logical terms. Underlying such problems, which we call strong hallucinations, are faulty representations that lead LMs into logical incoherence, and hence into error, regardless of the facts. Concentrating on negation, we prove that how an LM computes meaning representations for negation and how that affects outputs inevitably lead to logical error.
We argue that we can eliminate these errors and strong hallucinations by altering the way an LM treats and hence interprets logical operators. We illustrate our approach with negation. Our idea is simple but radical: whereas for LMs all tokens in a context play the same role in determining an attention based probability distribution over next tokens or “continuations” of the context, we claim logical operators should operate differently from non logical expressions. For instance, negation does not contribute just another token
to a latent representation via linear algebraic operations; it provides constraints that shift an input probability distribution Π that is determined by the tokens ϕ in its scope to a distribution Π′, with Π′(¬ϕ) = 1−Π(ϕ). These constraints, along with others for sequences or conjunction, allow us to calculate algorithmically and recursively transitions from input distributions over only positive content to appropriate distributions for a continuation containing logical operators. The output distribution not only faithfully reflects the logical meaning of negation but also other first order logical operators.
While hallucinations are often observed for text generation, these settings are difficult to evaluate. Thus, we evaluate our algorithmic or hybrid approach, we call it Λ for negation on three easily evaluable tasks: yes|no question answering, masked knowledge retrieval (MKR) and natural language inference (NLI) with negation, using both encoder LMs and LLMs. Nevertheless, we will illustrate Λ on text generation in NLI. Λ yields good performance on MKR and NLI; we increase accuracy by 13% on one NLI dataset and by 91% on another. Our datasets feature negation and are modified, linguistically validated versions of (Dagan et al., 2005; Hossain et al., 2020). Λ requires an LM to train and infer only on positive data, which avoids the problem of finding sparse negative data.
Section 2 provides background on hallucinations, distributions and linguistic meaning. Section 3 proves strong hallucinations follow from basic assumptions about an LM’s treatment of negation. Section 4 details our positive proposal, and Section 5 provides experiments with our approach on Q&A, MKR and NLI. Section 6 discusses related work.

heading: 2 Background
text: LMs. We assume LMs trained on transformer
12670
architectures (Vaswani et al., 2017) with multiple layers i over very large corpora using masked or causal language modeling. In decoder models, the decoder is trained to predict next token yi given a context matrix ci and previously predicted tokens {y1, , yi−1} using a conditional distribution µ(yi/y1, ..., yi−1, ci). In encoder models, ci is typically the input to an MLP layer to make predictions. ci is the LM’s internal representation at i of all the input tokens in its context and is a concatenation of multiple hidden states zhj , one for each attention head such that:
zhj = Σ K k=1α h j,kW h V xk (1)
αhj,k are the attention weights providing the importance of token (representation) xk to xj and W hV is the value weight matrix of head h. All tokens play a similar role given Equation 1.1
This representation enables LMs to learn a function σ : (C × X) → [0, 1] that assigns probabilities for string x ∈ X ⊂ V m given context C ⊆ V n, where V n is the set strings of length n with vocabulary V . For encoder and decoder models and tasks we consider here, m = 1 typically. So we will abstract away from architecture and learning details for our models (the tasks we examine all fit in this category); they learn a similar function from contexts to tokens of different kinds, though different loss functions may be used. Generative LMs use conditionalization as in Equation 2 to assign probabilities to novel strings s = (w1, w2, ..., wm+1) given context C:
µ(s|C) = µ(w1|C)× µ(w2 |C,w1)× (2)
...× µ(wm+1|C,w1, ..., wm) Generative LMs and encoders use this distribution over strings to select ouputs or continuations given a context C. Various methods M (maxarg, greedy, beam search, or sampling) over the distribution µ(.|C) have been proposed to find the maximally plausible continuation ϕ ∈ Xn of C. An LM’s affirmations, which we use in defining hallucinations below, are directly linked to this distribution.
Definition 1. f̂ affirms ϕ given C, if ϕ =M{x ∈ X : µ(x|C)}
1For motivation, see (Baroni and Zamparelli, 2010); these representations contain rich linguistic information (Devlin et al., 2019; Liu et al., 2019; Tenney et al., 2019; Talmor et al., 2020; Zhu and Rudzicz, 2020).
Strong Hallucinations. In this paper, we examine a special kind of hallucination we call strong hallucinations. To clarify strong hallucinations, we first provide a gloss for hallucinations. Hallucinations involve an attitude like belief or affirmation towards a proposition or sentential content p. An agent A’s degree of belief in p, µ(p), which is dependent on A’s evidence E, should determine their sincere assertion of p. This holds for LMs too; they use probability distributions to determine their “affirmations” (output). Now A’s degree of belief in p may or may not correlate with a “reasonable” degree of belief in p, π(p)—a degree of belief that one expects a rational person to have. On one type of hallucination, then, an agent A with grounds or context E hallucinates that p if µA(p|E) diverges significantly from π(p|E).
A strong hallucination byA involves a reasoning mistake given evidence. For an LM this evidence is a context or prompt. The result of a strong hallucination is that µ leads A to affirm a claim or set of claims that strongly diverge from any possible reasonable degree of belief given the evidence. A special case is when the set of claims together with the evidence is logically unsatisfiable or cannot possibly be true together.
LLMs have strong hallucinations. Consider, for example this session from LLama2-7b on the NLI task we detail in Section 5. The task was is to decide whether a hypothesis H is entailed by a premise P. To help it with this task, we instructed Llama in a prompt to answer two questions: Q1 can H be true if P is true? and Q2 can H be false if P is true?. We instructed it to answer “entailment”(E) if it answered no to Q2 – and “neutral” (not E) if it answered yes to Q1 and to Q2. Here is an example of the LLM’s response (R): (R) “
PREMISE: A young girl rides her bike HYPOTHESIS: This little girl is riding her bike Relation:
 Based on the rules and procedure provided, I will ask myself 1) Can the hypothesis be true if the premise is true and 2) Can the hypothesis be false if the premise is true? Since the premise and hypothesis are the same, it is possible for the hypothesis to be true if the premise is true. Therefore, the answer to question1 is yes. Since the premise and hypothesis are the same, it is not possible for the hypothesis to be false if the premise is true. Therefore, the answer to question 2 is no. Based on my answers to the questions, I will predict a neutral relation between the premise and
hypothesis” (not E). We assume that Llama has taken the context as given with the instructions in which we have if yes to Q1 and no to Q2, then it should infer entailment. Llama answers correctly yes to Q1 and no to Q2, but apparently converts its "no" to Q2 into a yes answer or simply affirms not E despite accepting A, B and if A and B, then infer E. This is an example of a logical error that infects not only the LLM’s reasoning but also, as in this case, its explanations of its behavior.
We now provide a technical definition of strong hallucinations using probability distributions. We idealize rational distributions by considering objective distributions that capture the probabilities of events that are coherent with respect to probability axioms. P represents the objective distribution over events or propositions described by strings in X in the actual state, while Pw is an objective distribution for a logically possible way w the world could be. Let µf̂ (.|E) be the conditional probability distribution of an LM f̂ over a set of truth evaluable strings X that express a set of propositions that includes p.
Definition 2. f̂ strongly hallucinates that p, if µf̂ (p|E) significantly diverges from Pw(p|E) for any possible objective distribution Pw and statew.2
Probability and linguistic meaning For epistemologists an ideal agent’s probability distribution κ governing assertions should equal some objective distribution P (Lewis, 1981; Hall, 1994). An ideal distribution κ thus differs in an important but subtle way from LM distributions. κ is not a probability distribution over sentences or strings but over propositions, what sentences express. To link with assertions, we use κ to assign values to strings as follows: for any proposition p, if strings ϕ, ψ both express p, then κ(ψ) = κ(ϕ).In addition, the meanings of logical operators like ¬ (not), ∨ (or), ∧ (and) impose the structure of a Boolean algebra P on the set of propositions. Hence, any probability function κ respecting P that assigns values to sentences of a language verifies all the probability axioms (see Proposition 2 in Appendix A.1) and assigns every sentence expressing a logical truth probability 1, every inconsistent string probability 0 and if ϕ is a logical consequence of ψ then κ(ϕ|ψ) = 1. Conversely,
2In the same spirit, we can define standard hallucinations: f̂ hallucinates that p, if µf̂ (p|E) is significantly different from P (p|E) and µf̂ (E) significantly diverges from P (E).
each linguistic model for meaning defines such a probability function κ(See Propositions 3, 4 Appendix A.1). Thus, an agent whose assertions are governed by κ, will avoid strong hallucinations.

heading: 3 LMs, negation and strong hallucinations
text: Section 2 showed a tight connection between ideal distributions, meanings and probability axioms. This connection does not hold for an LM’s objective function that is a distribution over strings.
Proposition 1. Every LM f̂ whose outputs are governed by µf̂ and Definition 1 must strongly hallucinate if either: (i) ∑ x∈V n\{ϕ,¬ϕ} µf̂ (x) > 0; (ii) µf̂ assigns values to strings in V n that are logical truths or deductively valid reasoning steps.
Proof: (i). Given the assumptions in (i), µf̂ (ϕ) + µf̂ (¬ϕ) < 1. But for every objective distribution P , P (ϕ) + P (¬ϕ) = 1. So µf̂ diverges from any possible objective distribution. (ii). Any objective distribution must assign probability 1 to every logically true sentence ℓ of V ∗. For each positive clause Q there is such an ℓ of the form Not (Q and Not Q). As a distribution over individual strings, µ cannot assign 1 to all such ℓ. Moreover, suppose given context the model offers a chain of deductive reasoning, ϕ.ℓ.ℓ′. But then by the axioms of probability and Equation 2, an objective distribution gives P (ℓ|ϕ).P (ϕ) = P (ϕ). So P (ℓ|ϕ) = 1. If µ(ℓ|ϕ) = 1, however, µ assigns no other continuation of ϕ non 0 probability; e.g., µ(ℓ′.ℓ|ϕ) = 0. This shows that LM distributions cannot respect the property of objective distributions that logically equivalent formulas receive the same probability. In either case, by Definition 2, the LM strongly hallucinates.2 We note that a similar argument to that in the proof of Proposition 1 (ii) shows that there are pairs of logically equivalent expressions to which µf̂ will assign distinct values.
(Ramsey, 1931; De Finetti, 1937) linked the probability distributions underlying strong hallucinations to irrational behavior on bet. Our results correctly predict that LMs will place pathological bets (See Appendix G for an example). While most NLP practitioners won’t care that their LM bets badly, reasoning errors like that in the example given in Section 1 are serious. Suppose an LM must output a reasoning chain R = (ϕ, ℓ1, ..., ℓr) in which ℓi in R should follow deductively from previous elements in R.
Proposition 2. If an LM f̂ outputs a reasoning chain R with length r, µf̂ will increasingly diverge from any possible objective distribution and f̂ will eventually surely hallucinate as r increases.
Consider a reasoning chain R and suppose µf̂ (ℓ1|ϕ), µf̂ (ℓ2|ℓ1), µf̂ (ℓ2|ϕ, ℓ1) are high but< 1, which given Proposition 1 is the best we can hope for. By Equation 2 µf̂ (ℓ1.ℓ2|ϕ) < µf̂ (ℓ2|ϕ, ℓ). R’s probability will thus decrease monotonically as R gets longer. Eventually, for some logically valid continuation of R.ℓr+1, µf̂ (ℓr+1|R) < µf̂ (ψ|R) for ψ independent ofR. So f̂ eventually will surely hallucinate.2
All LMs that generate responses from a distribution over individual stringsare subject to Proposition 1 and part (ii) applies to logical operators in general: LMs that make reasoning chains will assert invalid ones or will make mistakes in building them, with the probability of logical error increasing as the chain’s length increases.
Our results show we need to rethink the underlying LM distribution over strings to solve the problem of strong hallucinations from negation. While LMs may have some grasp of negation in simple contexts (Gubelmann and Handschuh, 2022), this address the underlying problem with an LM’s objective function. We turn to fixing that now with a hybrid approach to logical operators.

heading: 4 A hybrid treatment of logical operators for LMs
text: Logical operators like negation structure meaning recursively by performing a distinctive operation over the content of tokens in its scope. To develop our hybrid treatment of logical operators Λ, we adapt this idea and define a logical operator as function transforming distributions over continuations determined by the latent representations of tokens within the scope of those operators into continuations that reflect the semantics of the operator. For example, negation in the formula ¬A transforms distributions over continuations determined by the latent representation of A into continuations that reflect the semantics of negation and ¬A.
To develop Λ, we will make use of a dynamic analysis (DS) of logical operators (Kamp, 1981; Kamp and Reyle, 1993; van Eijck and Kamp, 1997). DS models logical operators as transitions between structures A that are pairs (UA, PA). As (Li et al.,
2021) have noted, such a pair corresponds to a partition of the tokens in an LM—UA a set of tokens represents objects and PA a set of tokens representing properties of those tokens. We take these tokens to be the output of the processing of some input linguistic context C to the LM. A second tool we need is the notion of an embedding (Chang and Keisler, 1973). An embedding of one structure A derived from linguistic information C into another B from another information source V defines satisfaction or truth of C relative to V .3 Definition 3. A has an embedding f : UA → UB in B (written A ≤f B) iff for each property ascription ϕ ∈ PA and for all x1, ...xn ∈ UA if ϕ holds of or is satisfied by x1, ...xn in A then ϕ is satisfied by f(x1), ...f(xn) in B.
(Kamp and Reyle, 1993) uses embeddings to define the meaning of negation. Negation structurally takes scope over a representation A, and converts it into a property of tokens in a larger structure. Embeddings and their extensions interpret this structural feature. Suppose for some structure C, A ≤f C, we say g extends f to an embedding of B in C, if: g ⊇ f and Dom(g) = UB and B ≤g C. Definition 4. Let Aϕ represent ϕ. A¬ϕ is satisfied relative to some v ∈ V and map f : iff there is no extension g of f over UAϕ such that Aϕ ≤g v.

heading: 4.1 Negation as a constraint on continuations
text: Pure LMs can’t appeal to an external source to characterize negation’s effect on their internal representations. Our key and novel idea uses continuations and their semantics (Asher et al., 2017, 2023). Continuations τ of a string σ have representations Aτ , where: UAσ ⊆ UAτ and PAσ ⊆ PAτ (which we write as Aσ ⪯ Aτ ). Then trivially for the identity map ι : UAσ → UAσ , Aσ ≤ι Aτ . Conflating continuations and the structures representing them, a continuation A2 of a context A1 has, given an LM’s distribution µ, a probability α–written A1 ⪯α A2; α is just µ(A2|A1).4
A proper treatment of negation determines how content in the scope of the negation affects coherent or admissible distributions of continuations A2
3Our formulation here echoes maps used in multimodal approaches (Li et al., 2019; Lu et al., 2019; Driess et al., 2023; Devillers et al., 2023; VanRullen and Kanai, 2021).
4Note that α is not the probability of embedding A1 in A2; that probability should always be 1, as ι always exists and provides the embedding.
of a context A1. For example, if an object (or token representation) o has the property ¬A in A1, no consistent or coherent continuation of the list of properties for o in A2 should contain A. More precisely, Negation imposes two constraints on coherent or admissable continuations A2 and distributions µ of an LM relative to a context A1; the third constraint below forces µ to obey the logical interpretation of conjunction or concatenation in strings; while the introduction of new information in a continuation, which is a way of representing conjunction introduction, is naturally represented in terms of conditionalization as in equation 2, constraint (iii) below ensures that an LM uses conjunction elimination correctly. For structures A,B, let A+B = (UA∪UB, PA∪PB) Definition 5. Let A1 ⪯γ A2 with µ an LM. A2 is an admissible continuation of A1 relative to µ only if: (i) If {¬Ak} ∪ PA1 = PA2 with µ(Ak|A1) = α then γ = 1− α; (ii) if PA1 = {¬Ak} and A3 ≺ A2 and PA2 = PA1 ∪ PA3 , then5 γ = µ(A3)(1−µ(Ak|A3))1−µ(Ak) ; (iii) if γ = 1, then for any context C, µ(A2|C) ≤ µ(A1|C); (iv) for any A4 ⪯ A2 (A4 possibly empty), µ(A1 + A4|A2) = 1. Definition 5 constrains the evolution of an LM’s latent representation A and distribution to capture the meanings of logical operators like negation in A in terms of their information update potential, in terms of what distributions over continuations they permit as admissible.
To illustrate, suppose a distribution µ with A1 ⪯γ A2, A2 represents it is not the case that B and µ(B|A1) = α. Definition5 says that A2 is an admissible continuation with respect to µ only if γ = 1 − α. Negation maps µ(B|A) into 1− µ(B|A). For any context C, unless µ(B|C) = .5, µ(B|C) ̸= µ(¬B|C).
Definition 5 applies recursively to strings of the form ¬(ϕ ∧ ¬ϕ) to assign them probability 1 and to ϕ ∧ ¬ϕ to assign them 0, regardless of the probability of ϕ. We can also model conditionals A⇒ B, using Definition 5, by translating if ϕ then ψ as ¬(Aϕ.¬Aψ). So for any context C µ(Aϕ ⇒ Aψ|C) = 1 − µ(Aϕ|C) + µ(Aψ|Aϕ + C) × µ(Aϕ|C), which in turn validates modus ponens: µ(B|A⇒ B.A.C) = 1. So, given a context C, as in the strong hallucination example R of Section
5For details see Appendix A.4.
2, that contains the conditional (yes to Q1 and no to Q2) ⇒ Entailment as well as a yes answer to Q1 and a no to Q2, then µ(Entailment |C) = 1. Definition 5 also constrains LM objective functions to conform to the truth conditional meanings of propositional connectives and quantifiers. Like negation, quantifiers operate on embeddings. The universal quantifier has the same meaning as that of the conditional we gave above (Kamp and Reyle, 1993; Groenendijk and Stokhof, 1991), while the meaning of an existential quantifier is captured by the definition of an embedding operating on the elements of UA that function as variables. For details see Appendix A.5.

heading: 5 Experiments with our approach on Q&A, cloze prompts, NLI and negation
text: We illustrate our hybrid approach Λ of Section 4 on three applications that feature negation: simple Q&A, masked knowledge retrieval (MKR) and natural language inference (NLI), each with a different output and input. We use both encoder and decoder LMs and LLMs.
These applications require us to apply Λ’s constraints logical operators, in particular negation, at various points. Suppose we must predict strings in S. S will have strings or substrings S ′ free of logical operators like negation or conditionals (e.g. simple clauses). Λ leverages an LM’s distribution µ1 over S ′|I ′ where I ′ is also free operator free and then transforms µ1 using the constraints in Definition 5 into a distribution over S|I, which contain operators.
While Λ generalizes to all LLM tasks, our applications only require predicting very short strings. While tasks with long string output are beyond this paper’s scope because of the complex evaluations required(Li et al., 2023), we will show how Λ can improve the longer reasoning chains of the LLM when prompted on NLI.
Q&A tasks. We examined BERT large and basic encoder models on a question answering task about facts in a synthetic dataset SYN we made. We used only Y|N questions with positive and negative contexts like: there was a < col > car (positive context); there was no < col > car (negative context). “< col >” is a placeholder for a color term. Given such contexts, we asked, Was there a < col > car? (See Appendix B for details).
Given an input < cls > QUESTION < sep > CONTEXT < sep > and a BERT model, we plotted
cosine similarity (cossim) histograms of CLS representations from a positive context and from its corresponding negative context, before and after fine tuning using the training regime of (Chaturvedi et al., 2024) on the Coqa dataset (Reddy et al., 2019). We used BERT, as its pretraining gives a meaningful CLS representation (Devlin et al., 2019). On pretrained BERT, cossim values ranged between .986 and 1, showing that, as one might expect, pretrained BERT’s representations do not give appropriate content to negation.
After Q&A fine tuning, however, cossim values for the CLS representations of positive and negative contexts were much lower, indicating that both BERT models (especially BERT-large with values between 0,34 and 0,38) learned to differentiate positive and negative content in CLS tokens. (Histogram plots are in Appendix C.)
We then tested fine-tuned BERT and Roberta on our Q&A task; the large models did perfectly though the small ones mostly gave only a “no" answer. The models interpret our task as a binary classification, in which the model considers only two mutually exclusive and exhaustive possible continuations, P (for “yes” toP ?) and¬P (for “no” P ?. This strategy approximates the truth functional meaning of negation in a simple setting and might account for their success with negative contexts.
Finally, we used Λ and fine-tuned BERT and RoBERTa models only on positive contexts in SYN. We accessed the logits, which after softmax give us a probability distribution over the Y|N classes. Using the probabilities over the positive contexts (Y |?Q,C), we applied Definition 5 to get probabilities for outputs with negative responses or contexts–e.g., (N |?Q,C) or (Y |?Q,¬C). Λ matched the esults of the large models on our task and yielded the best consistent results for the small models.
Masked Knowledge Retrieval. In a second experiment, we explored a Masked Knowledge Retrieval (MKR), in which the model must compute a sentence completion and lexical item for mask in context C– where the mask is either in a positive C or negative ¬C. Regardless of the facts, a prediction for ¬C(< mask >) should not match the prediction for C(< mask >); answers preferred in C should be dispreferred in ¬C.
We used (Kassner and Schütze, 2020)’s (KS) and (Jang et al., 2022a)’s (JS) “negated” versions of the LAMA dataset (Petroni et al., 2019). We converted
the negative contexts for the MKR task provided in KS and JS into a positive context C, which we need for our approach. KS provided 51 prompt examples, while JS yielded 2926 examples.
Pretrained RoBERTa-large and BERT-large had significant numbers of examples with an exact match (EM) for both positive and negative contexts in KS and so showed an inconsistent treatment of negation and strong hallucination. For instance, given the positive (negative) contexts, A teacher is (not) most likely teaching at a [mask], they returned the completions, A teacher is most likely teaching at a [school] and A teacher is not most likely teaching at a [school] (See Appendix D for more examples). Finetuning RoBERTa and BERT on Q&A as above improved model performance reducing EM but not eliminating them (see Table 1). We concluded that fine-tuning on Q&A does not give a full understanding of negation.
With cloze prompts, an LM originally outputs only the token with the highest score. But an important feature of negation is that it presupposes a set of relevant alternatives (Rooth, 1992). The capital of France is not Marseille conveys the information that while the capital of France is not Marseille it is some other city or place where people live and work.
To capture the idea of relevant alternatives with Λ, we modified the LM’s output to get its top 5 candidate completions in its distribution. We then applied Definition 5 by computing the probabilities pi over the top 5 completions σi 1 ≤ i ≤ 5 of the positive prompt C(< mask >); for the negative contexts ¬C(< mask >), we assigned 1−pi to σi, reversing the LM’s ranking of positive completions. Λ produced 0 EM on both KS and JS. All completions were meaningful on KS and only relatively few were ungrammatical on JS. Table 1 gives the performance on MKR of BERT and RoBERTA with pre-training only, fine tuning on Q&A and Λ.
NLI. In a third application, we looked at natural language inference (NLI) with RTE (Dagan et al., 2005) and SNLI (Bowman et al., 2015), two datasets modified for negation by (Hossain et al., 2020). Each contains a context C and a hypothesis h, which are labeled either with entailment (which we note as (C, h):E) or non-entailment ¬E (RTE) or entailment (E), contradiction (Cn) or neutral (N ) (SNLI). (Hossain et al., 2020) negated manually either C or h or both in portions of RTE and SNLI. Hossain’s datasets, ¬RTE and ¬SNLI, contain no examples of positive cases (C, h).We reconstructed positive examples (41% of ¬RTE) and for ¬SNLI; and to study negation in more detail we completed 117 examples of (Hossain et al., 2020)’s ¬SNLI with four inferential patterns: (C, h), (C,¬h), (¬C, h) and (¬C,¬h).
In addition, in ¬SNLI, negations of (C, h) pairs were often wrongly labeled because of non full scope negation. For instance, there were examples where the original h was of the form a man is smoking and a context C such that (C, h) : E was correct; but where with the negative hypothesis ¬h was a man was not smoking we had C,¬h : Cn, when C and ¬h were plainly consistent and we should have had C,¬h : N . We made either changed the label or made h and ¬h true contradictories, using a definite description the NP in h to refer to a NP from C or replacing a NP in h with no NP (for examples see Appendix E).
In addition our data often featured less than full sentential scopes for negation on C, as well as presuppositional elements (definite noun phrases, adverbial clauses), which scope out of negation. To deal with this, we divided C into two components P and C ′, with negation having scope only over C ′. We then added annotated inferential patterns for both material under the scope of negation and material outside of it. For instance, if we had the context a man with blue shoes was not sleeping and a hypothesis h, we annotated the inference to h from a man with blue shoes (P, h), and from h to a man was not sleeping (h,C ′). Our datasets containing 2306 annotated inferences are at https://github.com/swarnadeep8597/ Negation.git.
In NLI asks, the LM must learn to label context and hypothesis with relations whose definitions involve negation. This imposes a logical structure on the labels given their intuitive meaning; e.g., entailment (E) between C and h means that there are no situations where C and ¬h hold. From the
meaning of entailment and Definition 5, it follows that µ(E|C,¬h) = 1 − µ(E|C, h). Thus, if the model predicts (C, h):E, it should predict (C,¬h): ¬E. To infer labels for (¬C,¬h) from positive data, if (h,C):¬E/E, then (¬C,¬h): ¬E/E. Similarly for (¬C, h) contexts: if ¬C,¬h:E, then (¬C, h):¬E. This illustrates the interactions between negation in the data and in the task definition. Our rules are provably correct but incomplete for ¬RTE; using only valid rules, we cannot infer (C,¬h):E from just positive data in the 2 label E and ¬E task. So our method perforce missed those cases (see results for Λ on C,¬h in Table 2 for ¬RTE).
To account for less than wide scope negations, we developed an algorithm inspired by (Karttunen and Peters, 1979) that determines values for all configurations based on entailment predictions for (C, h), (P, h) and (h,C ′). In Algorithms 1 and 2 are the scoped algorithm for (¬C, h) and (¬C,¬h) in a 2 label NLI problem (the ¬RTE dataset) and scoped C. As we did not scope h, only those cases are relevant.
Algorithm 1 Algorithm for (¬C, h) and a 2 label NLI problem
1: procedure ALGORITHM(¬C,H) 2: (C, h) :← defined in paper 3: (P, h) :← P defined in paper 4: if (h,C) : ¬E then 5: if (C, h) : E and (P, h) : E then 6: (¬C, h) : E 7: else 8: (¬C, h) : ¬E 9: if (h,C) : E then (¬C, h): ¬E
In our ¬SNLI, the three labels E, Cn and N also make reference to logical relations involving negation, which Λ must reflect in its algorithm. For instance, exploiting these relations for C,¬h contexts gives: (C, h):E/Cn/N iff (C,¬h):Cn/E/N . Provably correct and complete rules for scoped negation in the 3 label NLI task are in appendix F, capturing the cases we missed in RTE.
Table 2 contains scores for the different configurations of C, h and negation. We first looked at finding a theoretical maximum accuracy for Λ using gold labels for for (C, h), (P, h) and (h,C ′) labels on both our ¬RTE and ¬SNLI. For our ¬RTE, Λ yielded an accuracy of 94% on all
Algorithm 2 Algorithm for (¬C,¬h) and a 2 label NLI problem
1: procedure ALGORITHM(¬C,¬H) 2: (C, h) :← C,h defined in paper 3: (P, h) :← P defined in paper 4: (h,C ′) :← C’ defined in paper 5: if (h,C) : E then 6: if (P, h) : E then 7: (¬C, h) : E 8: else 9: (¬C, h) : ¬E
10: if (h,C) : ¬E then 11: if (h,C ′) : E then 12: (¬C, h) : E
C,h configurations, while Λ basic, an algorithm assuming wide scope negation over C, was 11% less accurate, showing the importance of scoping (Kletz et al., 2023). On our ¬SNLI, Λ yielded a 96% accuracy overall.
Next, we prompted Llama2 7B on our NLI tasks to compute labels for positive (C, h), (P, h) and (h,C ′) data, to be used with Λ. We experimented with different formats; the model did best with prompts that involved using answers to two Y|N questions to infer a label in a chain of thought style (Wei et al., 2023), an example of which is in (R) (Section 2). Longer prompts for (C, h), shorter ones for (P, h) and (h,C ′) worked best are given in https://github.com/ swarnadeep8597/Negation.git. The prompts in the style of (R) provided extended reasoning chains with conditionals; and Definition 5 enabled us to exploit Λ also for the positive cases. Llama produced about 12% incorrect reasoning chains that Λ could correct. However, because the valid inferences led to faulty labels, the scores did not go up. So we took Llama’s labels for the (C, h), (P, h) and (h,C ′) cases.
Table 2 gives results for (LΛ) and and Llama alone (L). On our ¬RTE, for the positive datasets, L got 73% correct for (C, h), 76% correct on (P, h) and 83% on (h,C) labels. We used these predictions so that L,Λ achieved an average overall accuracy of 80% on the various ((¬)C, (¬)h) configurations. Llama alone (L) had an average accuracy of 71% on all ((¬)C, (¬)h) configurations–with 76% on (C,¬h), 63% for (¬C, h) and 70% for (¬C,¬h). LΛ significantly outperformed Llama by itself on this task.
L did less well on our ¬SNLI as seen in Table 2. After several attempts we found prompts on whichL predicted 78% correct on (C, h), 69% on (P, h) and 70% on (h,C ′). Using these scores LΛ gave an overall accuracy of 72%. L achieved a decent accuracy on (C,¬h) (accuracy 71%); but floundered on the (¬C, h), and (¬C,¬h) cases. LΛ far surpassed L on the NLI task with negation.

heading: 6 Related work
text: Definition 2 of strong hallucinations shows they involve necessarily unfaithful content like what (Ji et al., 2023) call instrinsic hallucination in which the unfaithful content contradicts its source. Strong hallucinations contradict every possible source (Moramarco et al., 2022; van Deemter, 2024). While researchers have proposed various causes for hallucinations (Filippova, 2020; Parikh et al., 2020; Longpre et al., 2021), we are, as far as we know, the first to define and to analyze strong hallucinations and to derive strong hallucinations from the distribution governing an LM’s output.
Various proposals to avoid hallucinations have also surfaced (Nakano et al., 2021; Asher and Hunter, 2022; Merrill et al., 2022; Gubelmann and Handschuh, 2022). The most obvious proposal, building bigger LLMs and larger training corpora, has drawn criticism (Filippova, 2020; Huang et al., 2021; Li et al., 2023; Goyal and Durrett, 2020; Sellam et al., 2020; Li et al., 2023). Our negative results buttress this criticism. Our method for correcting strong hallucinations from negation is formally correct; it’s not a form of training but acctually shifts LM output in its last layer. The embeddings we use to define Λ could serve to check factual hallucinations as they refine and extend
lexical matching and relation extraction techniques (Dhingra et al., 2019; Cao et al., 2018; Huang et al., 2020).
Prior research (Kassner and Schütze, 2020; Hossain et al., 2020; Hosseini et al., 2021; Jang et al., 2022a) has reported mediocre results for LM performance with negation on MKR and NLI. Proposition 1 provides a formal foundation for these observations, and our tests on pretrained BERT models confirms that they doesn’t adequately represent negation. (Gubelmann and Handschuh, 2022) adopt a pragmatic approach that improves LM performance on MKR; our novel, semantic approach gives completely logically coherent MKR performance. Finally, (Truong et al., 2023) investigated GPT style LLMs and found NLMs unable to reason with negation. Our prompting of Llama2 7B LLM our NLI datasets concurs with their observations.
(Jang et al., 2022b) improve LM performance on MKR tasks with negation by training the model on an intermediate task, in which they train an encoder model to predict whether given an input word and sentence the sentence describes it. They then use the model on MKR tasks. We believe their binary classification task can help understand negation, as our Q&A experiments indicate.
Work on neurosymbolic models (Poole, 2011; De Raedt et al., 2020; Olausson et al., 2023) is also relevant to our approach. Λ leverages the large store of lexical and conceptual knowlwedge in LLMs while constraining the LLM’s objective function to comply with the meanings of logical operators. Other neurosymbolic approaches try to incorporate logical operators into the architecture (using logical gates) without changing the objective function (Riegel et al., 2020). Our negative results show that such neuro-symbolic approaches are problematic. More promising post hoc methods use an LLM to translate natural language into some formalism that symbolic methods like theorem provers can manipulate (Olausson et al., 2023) or use LLMs to generate additional symbolic rules (Kalyanpur et al., 2022). However, using LLMs only as translators means we can’t access their deep lexical knowledge.
Our paper’s negative results assume a learned distribution over strings from corpora. This is compatible with results about what algorithms an LLM can compute when this distribution is replaced with something else (Pérez et al., 2021; Chiang et al., 2023).

heading: 7 Conclusions
text: We have investigated strong hallucinations in LMs, originating from faulty meaning representations of logical operators. Strong hallucinations account for logical errors and the failure of generative LLMs as in the example of Section 2 to explain their own predictions even if those predictions are correct. We proved that under minimal assumptions strong hallucinations must result from an LM’s distributions defined over strings. The received view of LM distributions is thus not tenable.
Focusing on negation, we proposed a new treatment Λ of logical operators, on which the operator introduces an operation over latent representations, imposing constraints on how those distributions over those representations evolve in coherent continuations. Our approach reinterprets an LM’s distribution as an assignment of degrees of truth to propositions or sets of strings. The constraints will interact with algorithms for tasks that exploit logical structure.
We illustrated Λ on Q&A, sentence completion and NLI tasks, with simple and precise evaluation metrics, and showed that it can increase LM performance. Λ only requires LM training on positive data, which is an advantage, since negative data is sparse in real corpora. For NLI, we tested Λ on two new datasets for NLI with negation that modify those in (Hossain et al., 2020); we showed Λ substantially improved LLM performance on NLI label prediction.
We did not fine tune our models on NLI or train them to predict logical operator scope (Kletz et al., 2023). We need to do both for end to end good performance. Llama2-7b was highly unstable under prompting and failed to grasp the NLI problem well. We hope fine tuning will give better results. We also plan to use the newer and more stable Llama 3 models on our NLI task.
Another task for future work is to deal with the fact that with Λ a distribution may generate a high probability for strings that may not be pragmatically relevant or appropriate. We will explore how to supplement and refine the distribution with pragmatic constraints that are learned with RLHF methods (Mishra et al., 2022; Rafailov et al., 2024). If RLHF has to be done anyway to get good performance from LLMs, perhaps this is not such an onerous step. We also plan to use new annotated data sets (Li et al., 2023) to apply Λ to multiple reasoning steps in LMs.

heading: Acknowledgments
text: We thank Julie Hunter and Akshay Chaturvedi of the COCOBOT and COCOPIL teams for helpful comments on prior drafts of this paper. For financial support, we thank the ANR project COCOBOTS (ANR-21-FAI2-0005), theANR/DGA project DISCUTER (ANR- 21- ASIA-0005), the COCOPIL “Graine” project of the Région Occitanie of France and the National Interdisciplinary Artificial Intelligence Institute ANITI (Artificial and Natural Intelligence Toulouse In- stitute), funded by the French ‘Investing for the Future–PIA3’ program under the Grant agreement ANR-19-PI3A-000. This work was performed using HPC resources from CALMIP (Grant 2016- P23060).

heading: 8 Limitations
text: While we have provided empirical support for our approach Λ using both LMs and LLMs on problems with simple and precise evaluation metrics, we haven’t deeply investigated negation in full text generation. We had trouble getting Llama even to produce reasoning chains and explanations of its behavior in NLI. Moreover, while it produced about a 10% error rate in those reasoning chains, when we corrected the reasoning the conclusions were often faulty because of Llama’s faulty inferences on positive NLI contexts. Λ doesn’t get perfect scores on our NLI data because it doesn’t capture the complex interactions between scopes of multiple logical operators. The scope of operators is crucial as can be seen from our Table 2. We did not train our models to do scoping. in future work we will do this following (Kletz et al., 2023). Preliminary experiments show LLama relatively capable of providing logical forms with plausible scopes for logical operators.
Finally, we need to address the failings of LLMs in dealing with just positive data. We need to fine tune a Llama model rather than simply prompting it, as prompting is not a very robust but rather unstable process on these LLMs we have found. To do fine tuning, we will need more and better data. Our examination of the data on NLI shows that we need a much more comprehensive and varied dataset. We will continue to enlarge the revised datasets we have provided here in supplementary material.

heading: Future Work
text: in future work we will do this following (Kletz et al. , 2023). Preliminary experiments show LLama relatively capable of providing logical forms with plausible scopes for logical operators. Finally, we need to address the failings of LLMs in dealing with just positive data. We need to fine tune a Llama model rather than simply prompting it, as prompting is not a very robust but rather unstable process on these LLMs we have found. To do fine tuning, we will need more and better data. Our examination of the data on NLI shows that we need a much more comprehensive and varied dataset. We will continue to enlarge the revised datasets we have provided here in supplementary material