The Surprising Value of Large Language Model Hallucinations (2024)

Abstract: This paper presents a systematic defense of large language model (LLM) hallucinations or ‘confabulations’ as a potential resource instead of a categorically negative pitfall. The standard view is that confabulations are inherently problematic and AI research should eliminate this flaw. In this paper, we argue and empirically demonstrate that measurable semantic characteristics of LLM confabulations mirror a human propensity to utilize increased narrativity as a cognitive resource for sense-making and communication. In other words, it has potential value. Specifically, we analyze popular hallucination benchmarks and reveal that hallucinated outputs display increased levels of narrativity and semantic coherence relative to veridical outputs. This finding reveals a tension in our usually dismissive understandings of confabulation. It suggests, counter-intuitively, that the tendency for LLMs to confabulate may be intimately associated with a positive capacity for coherent narrative-text generation.

heading: 1 Background
text: Artificial intelligence is rapidly becoming ubiquitous in science and scholarship with autoregressive large language models (LLMs) leading the way (Duede et al., 2024). Since the concept’s integration into natural language processing (NLP) discourse, hallucination in LLMs has been widely viewed as a conspicuous social harm and a critical bottleneck to real-world applications of LLMs. Both popular, comprehensive academic surveys (Ji et al., 2023; Kaddour et al., 2023) and public-facing technical reports (OpenAI, 2023; Touvron et al., 2023; Google, 2023) position the problem of hallucination as one of LLM’s chief ethical and safety pitfalls, one that should be heavily mitigated in conjunction with other concerns like bias and toxicity. Yet, despite a wide range of such mitigation efforts (Huang et al., 2023), hallucinations perdure, presenting an imminent risk to model trustworthiness
(Luo et al., 2024), with epistemic consequences in truth-sensitive domains like law (Curran et al., 2023), medicine (Pal et al., 2023), finance (Kang and Liu, 2023), science (Alkaissi and McFarlane, 2023; Duede, 2022, 2023), and education (Zhou et al., 2024).
With so much at stake, NLP researchers have naturally developed a normative stance against hallucinations, rightly extending epistemic worries to more far-reaching, ethical concerns like misinformation (Li, 2023), deception (Zhan et al., 2023), fairness (Wang et al., 2023), and the broader goals of alignment research (Ouyang et al., 2022). Consequently, the observed commitment to uniform reduction of hallucinations to negligible levels is not just seen as a technical challenge but also a critical component of the broader mission to mitigate the societal stigmas and systemic risks associated with the broad deployment and wide adoption of LLMs.
However, a smaller body of work advances the view that hallucinations are not inherently harmful. According to this view, hallucinated outputs have particular affordances. For instance, such outputs can be utilized for the exploration of domainspecific scenarios where properties of hallucination could be leveraged constructively (Duede and So, Forthcoming). Cao et al. (2022) demonstrate that certain types of hallucinations are factual and could positively contribute to text summarization. Yao et al. (2023) explore the potential of hallucinatory responses as adversarial examples for enhancing the robustness of models. Following an existing practice in computer vision to use hallucinated materials as synthetic training data, (Fei et al., 2023) devise a visual scene hallucination mechanism to dynamically fill in the missing visual modality for text-only inference with multimodal machine translation models (McKee et al., 2021; Wu et al., 2023; Shah et al., 2023).
This explorative view highlights the potential
1 14274
affordances and plausible necessity of hallucinations. The possible necessity of hallucinations is supported by recent research positing that hallucinations are a statistical inevitability (Kalai and Vempala, 2023; Xu et al., 2024), and impossible to eliminate from LLMs as efforts to do so would be limited to a trade-off between generativity, creativity. and information accuracy (Lee, 2023; Sinha et al., 2023; Zhang, 2023). Moreover, in many domain-specific applications, achieving an optimized balance between creativity and factuality is more effective for maximizing the utility of LLMs than merely attempting to eliminate hallucinations. LLM use cases where hallucinations could be particularly valuable include discovering novel proteins (Anishchenko et al., 2021), providing inspiration for creative writing (Mukherjee and Chang, 2023), and formulating innovative legal analogies (Dahl et al., 2024).
In this paper, we attempt to broaden the concept of hallucination and argue that hallucination is closer in kind to the concept of ‘confabulation’, a term that has already gained popularity in public discourse on AI but has yet to become widespread in the academic literature. Reorienting our conception of what is typically conceived of as hallucination towards confabulation offers a more flexible way to characterize, measure, and analyze factually inaccurate outputs.
We first operationalize a common measure of narrativity following (Piper et al., 2021; Piper and Bagga, 2022; Antoniak et al., 2023) and then demonstrate empirically (see Section 3) that ‘hallucinations’ manifest significantly higher ‘narrativity’ than strictly factual outputs. Scholarship in cognitive science and cultural analytics suggests that narrativity provides beneficial cognitive and
social effects and that, by extension, confabulations that demonstrate such features are likely to offer similar affordances. Taking this broader perspective, in turn, allows researchers to move beyond a strictly normative approach that narrowly focuses on identifying and discarding hallucinations. As such, casting hallucination as confabulation offers a practical and tangible framework for adjusting to the nuance of hallucinations.

heading: 2 Related Work

heading: 2.1 Confabulation vs Hallucination
text: While both are anthropomorphizing analogies borrowed from psychiatry, ‘confabulation’ has recently emerged as a preferred alternative to ‘hallucination’ in the public discourse on AI for three reasons. First, confabulation does not pertain to precepts or stimuli, thus avoiding the thorny implication that LLMs have sensory experiences or even consciousness (Smith et al., 2023). Second, as a more neutral term, confabulation is more sensitive to groups dealing with mental health disorders and better at signaling a lack of malicious intent in that the output is not to be thought of as a fabrication (Emsley, 2023). Finally, the concept’s applicability to non-pathological scenarios is supported by research from psychiatry suggesting that everyday memory reconstruction often involves some degree of confabulation (French et al., 2009) and is equally common for healthy individuals to inadvertently fictionalize details of stories without the intention to deceive (Riesthuis et al., 2023) (points we return to in detail in Section 4).
Despite conceptual distinction and advantages over ‘hallucination’, ‘confabulation’ is still commonly used interchangeably with hallucination in
2
the NLP literature (Romera-Paredes et al., 2023), quite often trivially so (Liu et al., 2023b; Zhan et al., 2023; Suraworachet et al., 2024). From the perspective of model architecture, confabulation has been considered as a formal disparity in confidence level between a model’s queried output and probed internal representation of truthfulness (Liu et al., 2023a). More severe implications of the term have been discussed from the perspective of medicine. For instance (Hatem et al., 2023; Brender, 2023) have debated the trade-off between the stigmatization that attaches to ‘hallucination’ and the semantic baggage that attends the anthropomorphism of LLM outputs characterized as confabulation. However, neither can offer a more substantive definition than their cited clinical baselines (Wiggins and Bunin, 2023).

heading: 2.2 Towards a Narrative-Centered Definition of Confabulation
text: However, extant definitions (see Table 1) in the literature do not sufficiently account for confabulation’s social and cognitive benefits in human communication. That is, current definitions attributed to LLMs fail to consider that, unlike hallucination, confabulation affords communicative and cognitive benefits to humans when endeavoring to fill in gaps in their knowledge of contextually relevant details to produce coherent linguistic communication. A conflation of confabulation with hallucination limits the potential affordances to NLP research that the concept of confabulation could play.
To address this, we define confabulation in a Narrative-Centric way as a latent narrative impulse to generate more substantive and coherent outputs —a characteristic of LLM textual outputs that closely mirrors the human predisposition to storytelling as a cognitive resource for sensemaking. Specifically, confabulation is a narrative impulse to schematize the information at hand into self-consistent stories, even if there might not be enough available details to do so, in which case would result in the generation of fictional yet plausible information. When humans do not have access to sufficient information to formulate coherent semantic meaning, they often confabulate to ‘fill in the blanks’ with self-consistent narratives that are not necessarily factual but bear close semantic verisimilitude to reality. Following existing conceptions of narrativity in narrative understanding (Piper et al., 2021), we further hypothesize that the extent of this compensatory storytelling can be
measured as a scalar construct that quantifies the degree of narrativity.

heading: 3 Data, Methods, and Results

heading: 3.1 Empirical Results for Higher Narrativity in Hallucinations
text: To validate the narrative-rich properties of confabulation, we compare the narrativity of hallucinations and their ground truth counterparts across three popular dialogic hallucination benchmarks:
• FaithDial (Dziri et al., 2022a) is a hallucination-free dialogue benchmark between an information-seeking user and a chatbot, adapted from the Wizard of Wikipedia (WoW) benchmark (Dinan et al., 2018). Mechanical Turk annotators labeled WoW’s human-generated responses as either ‘Hallucination’ or as truthful responses. Truthful responses are broken into three classes: ‘Entailment’, ‘Uncooperative’, and ‘Generic’, and produced faithful and knowledge-grounded edits to 21,445 original responses, all of which we sample for our study.
• BEGIN is a preliminary study of FaithDial conducted to select an existing benchmark for subsequent large-scale annotation and editing (Dziri et al., 2022b). As a smaller expertcurated set, it consists of information-seeking queries, as well as both human-written and model-generated (GPT-2, DoHA, and CRTL) responses, each labeled with a slightly different hallucination taxonomy than FaithDial (with the addition of ‘Partial Hallucination’ as a label) by expert annotators. We adopt BEGIN as a model and dataset agnostic validation of our findings on HaluEval, to confirm the consistency and robustness the narrative patterns across different datasets and models.
• HaluEval (Li et al., 2023) is a comprehensive dataset featuring plausible but hallucinated ChatGPT generations alongside their groundtruth counterparts. Instead of the more fine-grained hallucination labels of FaithDial and BEGIN, HaluEval only differentiates between hallucinated and ground truth responses. We only utilize the dialogue sections of HaluEval with 10,000 samples to maintain domain consistency with the other benchmarks.
3
For both the FaithDial and BEGIN datasets, we treat all outputs that do not contain a ‘hallucination’ label as ‘truth’ and all outputs that contain the ‘hallucination’ label as well as an additional truthful label as a ‘partial’ hallucination/truth. This aggregation allows for more direct comparison across datasets. In Table 2 and Figure 1, we present empirical support for the claim that confabulated texts, in general, exhibit increased levels of narrativity and, as such, can be viewed as a form of narrative-rich behavior.
Modeling Narrativity. The degree of narrativity is measured as the softmax probability output of an ELECTRA-large based (Clark et al., 2020) text-classification model fine-tuned on an expertannotated narrative detection dataset recently curated from Reddit (Antoniak et al., 2023). Antoniak et al. observe that a fine-tuned RoBERTa model outperforms zero-shot and few-shot approaches with larger models like GPT-4 at distinguishing between texts that represent stories from texts that represent non-stories. Following this strategy, we fine-tune a series of encoder-only models for story detection as a text classification task, which perform similarly when evaluated against both Antoniak et al.’s bootstrapped test set (AUC=0.83-0.85) and Piper et al.’s larger (AUC=0.81-0.84) (Piper and Bagga, 2022). We then qualitatively evaluate each model’s classifications to select the model (the fine-tuned ELECTRA-large) that best generalizes to the dialogue contexts of our target benchmarks and further verify its robustness through the manual evaluation of its inference output.
Results. Across all 3 benchmark datasets, we find that hallucinated qua confabulated responses exhibit higher narrativity than both partial and
non-hallucination categories and their own ground truth counterparts (See Table 2). Additionally, we find significant correlations between the narrativity score and the hallucination label. In this study, we employed a binomial logistic regression model to investigate the predictive relationship between the independent variable narrativity x and a binary dependent variable y that indicates ‘hallucination’ or ‘not hallucination’ such that P (yi = 1|xi) = e β0+β1xi
1+eβ0+β1xi . For the purposes
of this exercise, we class partial hallucinations as 1. Give the slighly lower mean ‘narrativity’ score for partial hallucination compared to pure hallucinated texts, we suspect that the results of our logistic regression, in fact, under-estimate the magnitude of the association. The logistic regression model allowed us to estimate the probability of y being 1 as a function of x. The model’s coefficients were estimated using maximum likelihood estimation (MLE). Table 3 indicates the results of fitting the model to predict whether an unedited response in FaithDial and BEGIN will likely be labeled as a hallucination. The positive coefficient and low pvalue indicate that higher degrees of narrativity are a significantly predictive feature of the hallucination label in both benchmarks.
4

heading: 4 In defense of confabulation
text: In this section, we present a two-part argument that calls for a more careful and nuanced examination of confabulation and possible utilization of its affordances. We argue that confabulation’s narrative-rich properties should not be viewed as a flaw but a hallmark for LLM alignment with a wellestablished human tendency to use narratives as a versatile tool for persuasion, identity construction, and social negotiation (Bruner, 1991). The normative view’s unreflective dismissal of confabulation, in turn, would risk eliminating critical behavioral and cognitive capacities central to communication and sense-making from the capabilities of LLMs. We also consider how aspects of narrative theory could complicate and enrich our understanding of newly emergent concepts in NLP, like faithfulness and factuality.
4.0.1 Empirical Support for Association of Narrativity and Coherence in Confabulated Texts
To empirically verify the correlation between narrativity and coherence, we extend our analyses to include an evaluation of dialogic coherence by implementing DEAM, a state-of-the-art metric for the task (Ghazarian et al., 2022). DEAM measures coherence in dialogues with a RoBERTa-large model fine-tuned on conversation-level semantic perturbations designed to generate more natural samples of incoherence, which achieves better generalizability from their training set than other metrics for coherence evaluation.
We find that higher narrativity is associated with higher coherence across all three benchmarks. Table 4 shows the results of a beta regression with
narrativity score as an independent variable and coherence as a dependent variable. Beta regression was used because all values in our dataset range between 0 and 1. Beta regression is particularly suited for modeling continuous variables restricted to the interval (0, 1), making it ideal for proportions and rates (Ferrari and Cribari-Neto, 2004). It accounts for the heteroscedasticity commonly observed in such data and provides more accurate estimates than linear regression in these contexts. Our regression model is simple: Beta(Coherence) = α+β1Narrativity and we apply epsilon smoothing for cases where coherence scores are exactly 0 or 1. We observe that there is generally a highly statistically significant association between increasing narrativity and increasing coherence.

heading: 4.1 Narrative, Discourse, and Coherence
text: Storytelling is pivotal in maintaining discourselevel coherence, a cognitive pattern that text linguistics identifies as a standard default assumption about any text or conversation until proven otherwise (Brown and Yule, 1983). When coherence is not immediately apparent, people actively search for evidence of its otherwise latent presence. In conversation scenarios, for example, the search for
5
latent coherence can take the form of a cooperative effort between participants, where they collectively work towards making sense of the subject at hand (Grice, 1975).
This latent presence of coherence typically exists in textual spaces “where it is not evident in the surface lexical or propositional cohesion” (Stubbs, 1983, 179), such as implicit semantic relationships like the inclusion of cultural subtexts and schemata that underpin discursive structures (Brown and Yule, 1983), which are, in turn, often represented and conveyed in the form of narratives. Stories, therefore, serve as a vital communicative and cognitive resource for sense-making, assisting individuals in navigating complex social and cultural contexts by providing a heuristic framework for understanding implicit meanings and relationships within discourse.

heading: 4.1.1 Narratives Help Us Articulate and Understand Complex Arguments
text: The narrative paradigm (NP) is a communication framework that posits that meaningful communication primarily occurs via forms of storytelling rather than discursive argumentation (Fisher, 1984), based on the widely validated premise that the rhetorical logic of compelling narratives tends to be more persuasive than that of structured arguments (Dahlstrom, 2010; Schreiner et al., 2018; Oschatz and Marker, 2020), particularly in the context of healthcare (Chen et al., 2016; Ballard et al., 2021). NP construes humans as especially disposed to storytelling and whose communication channels and internal world models are inherently sensitive to and ordered by narratives. This notion is echoed by foundational works in psychology (Sacks, 1985; Mercier and Sperber, 2017), philosophy of identity (Taylor, 1992), and philosophy of literature (Bruner, 1987; Dennett, 1988). Moreover, NP introduces the binomial nomenclature “homo narrans” to highlight the importance of storytelling in the human condition, underscoring its essential role in ensuring meaningful communication, mutual intelligibility, and the formation of common sense (Fisher, 1984).
As a model of human behavior, NP measures the effectiveness of conveying and understanding complex information through the narrative merit of its underlying stories. The judgment of this merit is based on two criteria of narrative rationality: 1) narrative coherence, the internal consistency of a story that allows it to make sense in its own con-
text, and 2) narrative fidelity, the degree to which a story aligns or resonates with the receiver’s existing understanding of reality and experiences with other stories (Fisher, 2021). In other words, the coherence of a message depends less on the completeness of its discursive structures than the selfconsistency of the stories around which its content is framed.

heading: 4.1.2 Narratives Maintain the Consistency of Our Own Internal World Models
text: Cognitive narratology is a branch of narrative theory that examines the “nexus of narrative and mind” (Herman, 2009, 137) through the cognitive underpinnings of storytelling practices. It considers narratives to be not merely cultural artifacts for literary interpretation but cognitive instruments that serve as an indispensable resource for navigating the world and scaffolding, compartmentalizing, and negotiating our experience (Dannenberg, 2008; Herman, 2013). The foundation of this approach rests on frameworks in cognitive linguistics positing that language and thought are fundamentally shaped by stories and so-called “parabolic projections” that map them onto experience (Turner, 1996).
Leveraging the semantics of possible worlds (Kripke, 1963) and theory of mind (Zunshine, 2006), cognitive narratologists argue that stories accomplish more than building worlds to ensure narrative fidelity with the interpreter’s mental models; they also enhance the self-consistency and robustness of such mental models through the simulation of counterfactual scenarios (Lewis, 1979) and alternative “storyworlds” (Ryan, 1991; Herman, 2013; Gerrig, 2018). This exercise of make-believe not only facilitates narrative understanding but also enriches the interpreter’s cognitive landscape by providing a dynamic platform for testing and refining mental models through imaginative engagement and adaptation (Doležel, 1998; Ryan, 2001), a process that has been linked to evolutionary mechanisms (Easterlin, 2012), especially a predisposition towards play in evolutionary psychology (Boyd, 2009).
Earlier periods of AI research took a particular interest in this mind-narrative nexus due to narrative’s capacity to facilitate complex interpretations with very limited discourse-level information available. Computer scientists have attempted to represent these capabilities formally as 1) cognitive mechanisms like story grammar that could parse stories into sequences and embeddings (Man-
6
dler, 2014), 2) frame representations of commonsense knowledge that reduce interpretive complexity by schematizing expected sequences of events and storing them in memory (Schank and Abelson, 2013). In addition, cognitive narratologists have also suggested that the narrative semantics of possible worlds, in which the coherence of a narrative is complete as its ability to construct immersive storyworlds compellingly, could serve as a hermeneutic approach to probe into the neural processes of automatic story generation (Ryan, 1991).

heading: 4.1.3 Narratives Enable Patients to Negotiate the Coherence of Their Experiences
text: One prominent display of narratives’ cognitive affordances is in the medical domain. Narratives have been documented to play a critical role in emotional recovery and rehabilitation in the aftermath of traumatic events that have disrupted the internal coherence of patients’ understanding of the world. Clinically validated narrative interventions have improved patient care and outcomes, especially by helping patients grapple with cataclysmic ruptures in the fabric of their experience (Rosário et al., 2018; Coats et al., 2020). Therapeutically, narratives serve dual purposes: for patients, narratives provide a vital sense of agency, giving them control over how to tell their own stories, a process crucial for reconstructing a coherent identity and self-concept (Frank, 2013); for physicians, engaging with patient illness narratives allows them to enhance their empathy by gaining insights into the patient as a whole person, rather than viewing them merely as a collection of symptoms or statistics (Charon, 2001).

heading: 5 Limitations and Directions for Future Research
text: This paper advances a systematic defense of LLM confabulations as a potential resource instead of a categorically negative pitfall. We believe the mental picture that LLMs hallucinate because they are untrustworthy, unfaithful, and ultimately unhumanlike is an oversimplified one. Instead, they confabulate and exhibit narrative-rich behavioral patterns closely resembling the human urge of storytelling – perhaps hallucinations make them more like us than we would like to admit.
While our current findings reveal intriguing associations between increased narrativity and significant increases in coherence, we must refrain from asserting that narrativity drives coherence.
However, this perspective does garner substantial support from the interdisciplinary perspectives presented in our qualitative analysis. This paper serves as a preliminary exploration in this direction, but we require more robust methods for modeling narratives and more comprehensive human evaluations to elucidate the intricacies of this association. We are eager to implement these steps in the next phase of our study.
The textual behaviors and effects we observe in LLM hallucination benchmarks, i.e., higher degrees of narrativity and coherence and the significant correlation between them, are widely considered beneficial in human-to-human communication. But the extent to which these affordances generalize to human-AI interactions—whether humans indeed benefit more in terms of user experience from engaging with high-narrative confabulations versus low-narrative but factual generations—needs to be further validated with human-based evaluations. We plan to follow up this study with experiments with human participants to verify the benefits of narrative engagement as hypothesized.
If the effectiveness of narratively rich confabulations is robustly validated, it could lead to exciting avenues for future research, testing the efficacy of confabulations across diverse domains. Beyond the applications we have already outlined in Section 1, exploring the utility of confabulations in fields such as journalism and advertising could yield valuable insights. By adopting confabulations as a conceptual framework, we could inspire more cross-disciplinary explorations, opening up new possibilities for LLM applications that extend beyond purely factual text generations.

heading: 6 Acknowledgement
text: For the research and completion of this paper, we acknowledge the resources and support provided by the McGill Faculty of Arts, McGill Department of English, and the Division of Research and Faculty Development at the Harvard Business School. Partial funding for the project was provided by Siva Reddy through a FRQNT New Researchers award and a NSERC Discovery grant. We thank Maria Antoniak for her help with the paper. We also acknowledge the helpful feedback from audiences at the University of Chicago’s Globus Labs, Data Science Institute, and Knowledge Lab, and Harvard Digital Data Design Institute. Finally, we sincerely thank our anonymous reviewers for their generous
7
time, attention, and feedback.

heading: Future Work
text: If the effectiveness of narratively rich confabulations is robustly validated, it could lead to exciting avenues for future research, testing the efficacy of confabulations across diverse domains. Beyond the applications we have already outlined in Section 1, exploring the utility of confabulations in fields such as journalism and advertising could yield valuable insights. By adopting confabulations as a conceptual framework, we could inspire more cross-disciplinary explorations, opening up new possibilities for LLM applications that extend beyond purely factual text generations