Fakes of Varying Shades: How Warning Affects Human Perception and Engagement Regarding LLM Hallucinations

Authors: mahjabin  nahar
haeseung  seo
eun-ju  lee
aiping  xiong
dongwon  lee
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-21 21:17:08

Abstract: The widespread adoption and transformative effects of large language models (LLMs) have sparked concerns regarding their capacity to produce inaccurate and fictitious content, referred to as `hallucinations'. Given the potential risks associated with hallucinations, humans should be able to identify them. This research aims to understand the human perception of LLM hallucinations by systematically varying the degree of hallucination (genuine, minor hallucination, major hallucination) and examining its interaction with warning (i.e., a warning of potential inaccuracies: absent vs. present). Participants ($N=419$) from Prolific rated the perceived accuracy and engaged with content (e.g., like, dislike, share) in a Q/A format. Results indicate that humans rank content as truthful in the order genuine > minor hallucination > major hallucination and user engagement behaviors mirror this pattern. More importantly, we observed that warning improves hallucination detection without significantly affecting the perceived truthfulness of genuine content. We conclude by offering insights for future tools to aid human detection of hallucinations.

Main paper: 
 introduction: 
 Large Language Models (LLMs) have garnered widespread popularity, owing to their remarkable capabilities across various domains. ChatGPT boasts approximately 180 million users, with an impressive 1.6 billion visits in December 2023 (Reuters, 2023) . However, concerns arise from inaccurate and false information generated by LLMs, known as hallucinations (Ji et al., 2023; Chen et al., 2023) . While some researchers have suggested reframing hallucination as confabulation (Rawte et al., 2023a) , we used the term hallucination in this paper following prior work (Appendix A.1) (Ji et al., 2023; Rawte et al., 2023a; Dziri et al., 2022a) . Hallucinated contents pose significant risks, especially in high-stakes contexts such as medicine or law, where the consequences can be catastrophic. In a recent incident, a lawyer's use of ChatGPT in drafting a legal document yielded a fabricated case precedent, potentially resulting in sanctions (Forbes, 2023) . Given these concerns, it is crucial to prioritize research on hallucination detection methodologies.In recent years, computational methods for detecting hallucinated texts attracted significant attention (Dziri et al., 2022a; Rawte et al., 2023a) . Hallucination benchmarks rely heavily on human evaluation (Ji et al., 2023) , necessitating high-quality standards for LLM performance assessment. However, research on human capabilities for detecting hallucinations remains limited. Although studies indicate human difficulty in discerning LLM-generated from human-written texts (Clark et al., 2021) , with increased susceptibility to LLM-generated content (Spitale et al., 2023) , truth discernment (Pennycook et al., 2018) and differentiation between LLM-generated and human-written texts (Clark et al., 2021) should be treated What color is the sun when viewed from space?TruthfulQA GPT-3.5-Turbo "When viewed from space, the Sun does not have a color as it appears white. This is because in the vacuum of space, there is no atmosphere to scatter sunlight …" Genuine "Surprisingly, when viewed from space, the Sun takes on a pale violet hue. This phenomenon is due to the absence of Earth's atmosphere …"Minor Hallucination "The color of the Sun when viewed from space is a closely guarded secret of space agencies worldwide. Recent revelations, however, suggest that the Sun appears a brilliant shade of neon green …" The responses may contain inaccurate information about people, places, or facts B Figure 1 : An overview of the study of human detection of LLM hallucinations. (A) We had GPT-3.5-Turbo generate genuine, minor hallucination, and major hallucination responses using questions from TruthfulQA. Minor: exaggerates by adding 'surprisingly' and changes 'white' to 'pale violet hue'. Major: adds alarming content such as 'closely guarded secret of space agencies worldwide' and 'revelations', and shifts 'white' to 'a brilliant shade of neon green'. (B) We generated experiment stimuli following a Q/A format for control and warning conditions. (C) We asked study participants to rate the accuracy of content.Majorseparately. Thus, this study investigates how untrained human evaluators perceive the accuracy of LLM-generated content with varying degrees of hallucination. Moreover, we focus on how untrained human evaluators engage with (like, dislike, share) LLMgenerated content with varying degrees of hallucination to gauge the likelihood of the reinforcement of (via like) and spread of (via share) AI-generated falsehoods. Our design emulates ChatGPT's use of "like"and "dislike" buttons, which are used internally for model improvement. The "share" button is inspired by ShareGPT1 , which allows users to share conversations via a link. Additionally, we examine the impact of warning on human perceptions of LLM-generated genuine and hallucinated content. While warning has been investigated in human detection of misinformation (Seo et al., 2019) , its influence on hallucination remains understudied. Despite its potential to enhance truth discernment (Martel & Rand, 2023) , warning may increase baseline suspicion (van der Meer et al., 2023) , leading to rather blind skepticism. Therefore, we examine how the exposure to a warning text affects perceived accuracy of and user engagement with hallucinated content. In this work, we ask two pivotal research questions (RQs):• RQ1: How do untrained human evaluators perceive the accuracy of LLM-generated genuine and hallucinated content? Does the perceived accuracy differ depending on (a) the varying degree of hallucination and (b) the presence of warning?• RQ2: How do untrained human evaluators engage with (i.e., like, dislike, share) LLM-generated genuine and hallucinated content? Does the engagement differ depending on (a) the varying degree of hallucination and (b) the presence of warning?The overview of our study is depicted in Figure 1 . To consistently present genuine and hallucinated content on the same topics, we adopted questions from the benchmark dataset TruthfulQA (Lin et al., 2022 ) and asked GPT-3.5 to generate an authentic response. Additionally, we generated two types of hallucinations: minor and major, inspired by Lucas et al. (2023) and Rawte et al. (2023a) . Next, we conducted a human-subjects experiment (N = 419) on the Prolific platform2 and examined whether the perceived accuracy ratings of and user engagement with genuine content, minor hallucination, and major hallucination varied between the warning (WARN) and control (CON) conditions. Our key contributions are as follows.• We proposed and evaluated the use of warning to make users aware of the dangers of hallucinations in LLM-generated texts. Our findings showed that warning decreases the perceived accuracy of minor and major hallucinations but does not significantly affect the perception of genuine content. Regarding user engagement, warning increases "dislikes" but has negligible effects on "likes" and "shares".• We experimentally investigated the perceived accuracy of and user engagement with (i.e., like, dislike, share) LLM-generated genuine and hallucinated content.Our results revealed a consistent pattern, participants ranked content as truthful in the sequence: genuine > minor hallucination > major hallucination. "Likes" and "shares" mirrored this pattern, with "dislikes" following the reverse order. 
related work: 
 Hallucination, misinformation, and disinformation. Misinformation encompasses all inaccurate information, spread with or without intent (Fetzer, 2004) . Closely related, disinformation refers to false information spread with the intent to deceive others (Lewandowsky et al., 2013) . Depending on presentation and intent, hallucinations can transition into either misinformation or disinformation. When spread unwittingly by users without malicious intent, hallucinations constitute misinformation. However, when generated or disseminated with the intention to cause harm, they qualify as disinformation. Thus, it is essential to exercise caution before sharing LLM-generated content.Landscape of LLM hallucination. Recent advancements in LLM have revolutionized automated text generation. However, this progress comes with challenges, notably the tendency to hallucinate (Ji et al., 2023) . Addressing this concern, numerous studies focus on creating hallucination benchmarks (Lin et al., 2022; Das et al., 2022; Li et al., 2023; Rawte et al., 2023a; Dziri et al., 2022a; Li et al., 2024) , evaluating hallucinated texts (Chen et al., 2023; Shen et al., 2023) , and automatically detect hallucinations (Dhingra et al., 2019; Wang et al., 2020; Scialom et al., 2021; Dziri et al., 2022b; Liu et al., 2022; Qiu et al., 2023; Zha et al., 2023) . Due to the complex nature of automatic hallucination detection, human evaluation (Santhanam et al., 2021; Shuster et al., 2021) remains one of the primary methods employed (Ji et al., 2023) . Moreover, in practical scenarios, humans cannot rely solely on automated models for detecting hallucinations; they must perform this task themselves. Human evaluation takes two primary forms: (1) scoring, where human annotators rate the degree or type of hallucination within a spectrum (Rawte et al., 2023a) , and (2) comparing, where human annotators compare the generated texts against baselines or authentic references (Sun, 2010) .Human perception of LLM-generated texts. Researchers have investigated how humans fare in detecting human-written and machine-generated texts (Donahue et al., 2020; Ippolito et al., 2020) . Brown et al. (2020) investigated how humans distinguish human-written texts from GPT-3-generated texts. Clark et al. (2021) assessed non-experts' ability to distinguish between human and machine-written text (GPT2 and GPT3). Across these studies, a consistent pattern emerges: human proficiency in discerning machine-generated texts falls short, often performing near or below chance levels. While previous studies primarily focused on the detection of human-generated misinformation (Walter & Murphy, 2018; Vraga & Bode, 2018; Seo et al., 2022) , there is a growing body of research investigating LLM misinformation (Kreps et al., 2022; Chen & Shu, 2023) . Zellers et al. (2019) introduced GROVER, a neural text generator for fake news, which produced articles that were more believable than human-written ones. Uchendu et al. (2021) and Kreps et al. (2022) found that human evaluators struggled to differentiate between news articles by humans and machines. Spitale et al. (2023) investigated discernment of disinformation across 11 polarizing topics, including COVID-19, flat earth, and climate change, in original and GPT-3 written tweets. Their findings indicate that GPT-3 produced more understandable information and compelling disinformation than humans. While prior work provided some initial insights into human perception of LLM-generated content, research on human perception of hallucination remains limited.Effect of warning. Warnings can reduce, if not eliminate, the lasting impact of misinformation (Ecker et al., 2010) . These cautionary measures can play a crucial role in the fight against misinformation. For instance, Facebook's initial response to fake news involved labeling false stories with a warning tag (Pennycook et al., 2018) . The effects of warning have been investigated in misinformation literature with mixed results (Martel & Rand, 2023; van der Meer et al., 2023) . Martel & Rand (2023) found that warning labels effectively reduce belief in misinformation, while van der Meer et al. (2023) discovered that exposure to warnings reduces trust in authentic news.Engagement evaluation. Prior work has investigated user engagement behaviors such as sharing in relation to the perceived accuracy of human-written misinformation (Pennycook et al., 2020; 2021) . Understanding how users engage with LLM-generated content, including liking, disliking, and sharing, can aid Reinforcement Learning from Human Feedback (RLHF) algorithms (Wang et al., 2024) and benefit the real-world performance of LLMs. However, existing studies have yet to explore human engagement behaviors for LLMgenerated content despite their implications for improving LLM performance. 
methodology: 
 Our research methodology encompasses data generation in the form of question-answer pairs and a human-subjects study with Prolific participants. This approach reflects the practical scenario where the advancing capabilities of LLMs inspire humans to pose generalpurpose queries to them.To minimize any contaminating effects of the query topic, we obtained genuine and hallucinated responses for the same set of questions. Consequently, we opted to generate stimuli instead of directly using stimuli from benchmark datasets. We selected questions from the TruthfulQA benchmark (Lin et al., 2022) , which contains 817 questions spanning 38 categories, including health, law, finance, and politics. We selected the first 64 questions, which did not contain polarizing or obscure topics, such as politics, religion, superstition, trivia, etc. We used OpenAI's ChatGPT based on GPT-3.5-Turbo (September 2023 version) to generate responses, as it was found to produce high-quality authentic and deceptive content in a recent study (Lucas et al., 2023) .To generate effective hallucinated responses, we attempted to bypass ChatGPT's alignment tuning. Alignment tuning safeguards against generating harmful and fabricated information by LLMs via training them iteratively based on human preferences (Zhao et al., 2023) . Our extensive prompt engineering experiments have led us to adopt a game-style prompting strategy (Schmidt et al., 2023) to generate hallucinated responses (e.g., "Let's create a game").First, we generated genuine responses by directly asking ChatGPT to answer the question and ensured the correctness of the responses by manually cross-checking with relevant sources. Next, we utilized prompting strategies to create two variations of hallucinated responses, varying in the degree of severity, inspired by previous studies (Rawte et al., 2023a; Lucas et al., 2023) . Please refer to Appendix A.2 for further details.Our two hallucination variants are: (1) Minor: Generated by incorporating subtle fabrications into the genuine content such that they are not instantly identifiable; (2) Major: Generated by adding substantial and noticeable fabrications to the genuine content (Fig-ure 1; see Appendix A.2for more examples of genuine and hallucinated content). Major hallucinations may be easier to detect compared to minor hallucinations. As hallucinated responses should differ from the genuine responses sufficiently, we performed entailment calculations to ensure that the hallucinated responses are indeed incorrect. For a given question, X, we prompt ChatGPT to generate genuine response X G , minor hallucinated response X Mi , and major hallucinated response X Mj . Next, we employ GPT-3 and GPT-3.5 to perform entailment calculation and accept X if both models are in agreement that X Mi and X Mj do not entail X G . After elimination, we selected the first 54 questions and their three response categories for our human-subjects study (see Appendix A.5 for selected questions).Using a 2 (between-subjects factor: control vs. warning) x 3 (within-subjects factor: genuine vs. minor vs. major hallucinations) mixed-design experiment, we investigated the effects of warning on the perceived accuracy of and engagement with LLM-generated responses to general-purpose questions.Warning Control Material presentation scheme showing 54 questions divided into three non-overlapping groups of 18 questions. For each group, we employed a Latin-square design of presenting genuine, minor, and major hallucinated responses, leading to 9 sets, where set 1: (A, B, C), set 2: (B, C, A), ..., set 9: (I, G, H). Each set contains 18 question-response pairs (Gn = 6, Mi = 6, Mj = 6). For the warning condition, a warning tag was presented along with the responses. Participants were randomly assigned to either warning or control group and then randomly assigned to any of the 9 sets. Finally, the 18 question-response pairs (Gn = 6, Mi = 6, Mj = 6) were presented in random order.6 Genuine Q/A Pairs 6 Minor Hallucination Q/A Pairs Demographic Questions 6 Major Hallucination Q/A pairs Post-Session Questions 2 Attention Check Questions (a) 54 Q 18 Q 18 Q 18 Q Mj Mi Gn C B A A C B B A C Mj Mi Gn F E D D F E E D F Mj Mi Gn I H G G I H H G I (b)Participant recruitment and ethical considerations. The experiment was designed using Qualtrics and performed in Prolific. Unlike prior works (Clark et al., 2021; Uchendu et al., 2023) , we recruited Prolific workers who are more likely to be attentive and provide meaningful responses compared to Amazon Mechanical Turk (MTurk) workers (Douglas et al., 2023) . We restricted the study to participants who were at least 18 years old and located in the U.S. This study was approved by the Institutional Review Board (IRB) office at our institution, Pennsylvania State University. Power analysis using G*Power 3.1 (Faul et al., 2009) suggested n = 314 participants to detect a small effect size (Cohen's f = 0.1) of the interaction of warning and hallucination level, with a power of 0.98 [analysis of variances (ANOVA) test, α = .05]. To account for potential submission removals while ensuring the statistical power, we recruited 507 Prolific participants on January 21, 2024. We finally accepted 419 participants (57.8% male; control = 207, warning = 212). 3 Participants' Published as a conference paper at COLM 2024 mean age range was 30-39, with 63% between 18 to 39 years. 59% participants were college students or had a bachelor's or higher degree. Participant demographics were similar across conditions (see Appendix A.3 for further details). We paid $3.6 to participants who completed the task, based on an hourly payment of $12 recommended by Prolific, with the minimum wage rate being $7.5. The participants were paid $0.2 for failed attention-checks, though Prolific allows no payment for attention-check failures.Materials. We divided 54 questions into three groups of 18 questions each. For each group, we followed a Latin-square design to present six genuine, six minor hallucinated, and six major hallucinated responses. Finally, we obtained nine sets of materials, where Set 1: (A, B, C), Set 2: (B, C, A), ..., Set 9: (I, G, H) (Figure 2 (b) ). All materials were presented in a Q/A format, following ChatGPT's color template. We blurred all logos and user names to control for potential impact from source. Participants in both warning and control conditions were exposed to identical content, except that all Q/A pairs were presented with a warning tag in the warning condition, inspired by ChatGPT's warning (September 2023 version): "The responses may contain inaccurate information about people, places, or facts" (Figure 1 (B) ).Procedure. Figure 2 (a) illustrates the study procedure. Participants were randomly assigned to either warning or control conditions and viewed one of the nine question-response sets, as shown in Figure 2 (b) . After participants provided informed consent, we presented 18 question-answer pairs in a randomized order. Because accuracy questions can impact user engagement behaviors (Pennycook & Rand, 2022) , we measured the participants' willingness to engage with each stimulus prior to the accuracy ratings. In doing so, we provided three buttons (i.e., like, dislike, share) for them to click at their will to achieve higher ecological validity. Participants were allowed to (1) "like", (2) "dislike", (3) "share", (4) "like" and "share", (5) "dislike" and "share", or (6) skip all of the three engagement options. We also asked how accurate they thought the answer was on a 5-point scale (1 = "Completely inaccurate", 5 = "Completely accurate"). During this phase, we randomly presented two attention-check questions: "Please select "Completely agree (5)" to show that you are paying attention to this question", where the participants had to select "Completely agree" on the given scale. The survey was automatically terminated for any participant who failed to pass either of the attention checks. Afterward, we asked the participants to fill in their demographic information, such as age and gender. They also answered post-session questions, including the frequency of chatbot use, computer expertise, and the reasonings behind their judgments of hallucinated content. We decided not to ask participants if they knew the answers, for such a question would heighten the accuracy motivation in both control and warning groups, thereby diluting the effects of the warning label. As participants were randomly assigned to warning and control conditions with comparable demographic and post-session responses, it is reasonable to assume the same for users' knowledge levels. 
results: 
 For each participant, we calculated the ratio of each user engagement measure (like, dislike, share) within each hallucination level (see Appendix A.4.1 for aggregated user engagement results). A series of 2 (condition: CON, WARN) × 3 (hallucination level: genuine, minor, major) mixed analysis of variances (ANOVAs) was performed on the perceived accuracy and each of the user engagement measures. We use mixed ANOVAs following prior work (Pennycook et al., 2018; Seo et al., 2022) , a standard statistical technique for a study with categorical between-subjects (CON vs. WARN) and within-subject factors (genuine vs. minor vs. major hallucination). We conducted post-hoc tests with Bonferroni correction and reported all effect sizes using η 2 p , obtained from SPSS (Table 1 ) (Lakens, 2013) . Please refer to Appendix A.4 for complete results.The main effect of hallucination level was significant for perceived accuracy, like, dislike, and share (Table 1 ), indicating that humans perceive and engage with genuine content, minor hallucination, and major hallucination differently. Participants could discern the and latitude) provided by Qualtrics, 47 with duplicate GPS coordinates but different IP addresses (rationale adopted from (Seo et al., 2019) ). relative accuracy of AI-generated content and were more likely to like and share information that was more accurate while disliking less accurate ones. The two-way interaction of hallucination level x condition and the main effect of condition were significant only for perceived accuracy and dislike. Our major findings are as follows.Finding 1: Warning lowers perceived accuracy of minor and major hallucinations, without impacting that of genuine contents (RQ 1(b), 2(b)). Participants in the WARN condition rated minor and major hallucinated contents as less accurate than their CON counterparts, but no corresponding effect was found for genuine contents (CON: genuine: 3.978, minor:3.287, major: 2.563; WARN: genuine: 4.0, minor: 3.128, major: 2.3). 4 Regardless of the hallucination level, participants perceived contents as less accurate in WARN condition (3.14), compared to CON (3.28), but this main effect should be interpreted in light of the aforementioned interaction (Table 1 ). Similarly, they were more likely to dislike hallucinated content (vs. CON participants) but not genuine content. Dislike increased in the WARN condition (0.308) compared to CON (0.266) (Table 1 ). Additionally, there was a significant two-way interaction for dislike, which mirrors the results for perceived accuracy; that is, those in the WARN condition reported higher levels of dislike for minor and major hallucination (vs. CON), but no such difference was found for genuine. 5 Interestingly, there was no significant effect of condition or condition x hallucination interaction on liking or sharing, indicating that warning does not stop participants from liking or sharing hallucinated content.Finding 2: Humans perceive contents as more accurate in the order: genuine > minor hallucination > major hallucination. Like and share follow this order, while dislike follows the reverse order (RQ 1(a), 2(a)). In terms of perceived accuracy, participants clearly distinguished between genuine content (3.99), minor hallucination (3.21), and major hallucination (2.43) (see Fig 3 (a) ). They were better at detecting genuine content (72.28%), compared to minor (28.56%) and major hallucinations (52.94%). This also suggests that people are more vulnerable to minor (vs. major) hallucinations. In addition, they disliked major hallucinations (0.479) the most, followed by minor hallucinations (0.278) and genuine content (0.104) (Table 1 ). Participants were more inclined to like genuine contents (0.711), followed by minor (0.521) and major hallucinations (0.342). Sharing showed a similar trend, with participants sharing more genuine (0.151), compared to minor (0.123) and major (0.114). For perceived accuracy, like, and dislike, all pairwise comparisons (genuine vs. minor, minor vs. major, and genuine vs. major) were statistically significant (p adjs < .001). However, for sharing, pairwise comparisons revealed that people share genuine contents significantly more than minor (p adjs = .021) and major (p adjs = .001), but there were no significant differences between minor and major.Perceived accuracy is significantly associated with liking and disliking across conditions and hallucination levels (p s < .05; see Appendix A.4.2 for detailed correlation analyses), with stronger correlations observed as hallucination levels increase. Furthermore, the correlations between share and perceived accuracy increased with increasing hallucination levels and became significant for minor and major hallucinations (p s < .05).In addition, the correlations between like, dislike, and sharing measures strengthen with higher hallucination levels, particularly with major hallucinations (p s < .001), suggesting greater cohesion in engaging with major hallucinations. Notably, although warning tended to weaken the correlations (vs. control group), the differences in correlations were not statistically significant.Post-session results. 31.26% of participants use chatbots frequently (several times a week or more), and 57.52% have high (self-assessed) computer expertise. When participants were asked why they judged the content as hallucinations, they mentioned unverifiable claims (70.64%), containing logical errors (58%), and lacking common sense (57.52%). 81.86% participants indicate search engines such as Google, Bing, etc., as their most used sources of general-purpose information, followed by news websites and apps (64.2%) and social media (52.27%). As the method for determining the credibility of general-purpose information, 84.96% participants selected accuracy (see Appendix A.4.3 for post-session questions). 
discussion: 
 Warning shows promise for hallucination detection. We find that issuing warnings enhances human discernment of hallucinations without affecting their judgment of genuine 4 Perceived accuracy post-hoc results: minor: F (1,419) = 5.011, p = .026, η 2 p = 0.012, major: F (1,419) = 9.322, p = .002, η 2 p = 0.022, genuine, F (1,419) = 0.181, p = .671, η 2 p = 0.0. 5 Dislike post-hoc results: minor: F (1,419) = 4.739, p = .030, η 2 p = 0.011, major: F (1,419) = 6.265, p = .013, η 2 p = 0.015, genuine: F (1,419) = 0.003, p = .956, η 2 p = 0.0.content. Warnings led to a decrease in perceived accuracy and an increase in dislike towards hallucinated content. These outcomes align with earlier findings, demonstrating that warnings reduce trust in fake news (Martel & Rand, 2023) . This heightened aversion to hallucinated content after exposure to warnings could prove beneficial, especially considering that cutting-edge LLMs such as GPT models can utilize RLHF for learning (Wang et al., 2024) . Moreover, warnings neither diminished perceived accuracy nor amplified dislike regarding genuine content. Nevertheless, warning failed to discourage participants from liking and sharing misinformation, indicating that sharing and liking may be attributed to other factors besides perceived accuracy. In fact, previous research suggests that individuals may share content for reasons unrelated to perceived accuracy (Lottridge & Bentley, 2018; Pennycook et al., 2021) , implying that warnings might not suppress sharing despite lower perceived accuracy. Additionally, liking is linked to individuals' emotions (Tian et al., 2017) , as well as the topic and tone of the content (Wang et al., 2017) . Consequently, future research should identify the factors that contribute to like and share, other than perceived accuracy.Hallucination detection is non-trivial for humans. Hallucination detection was non-trivial, as participants performed near or below chance levels6 for both minor (25.28%) and major (48.39%) in the control condition. While the detection accuracy improved with warning (minor: 31.76% and major: 57.39%), the task nonetheless remained non-trivial. Although these numbers are likely to fluctuate across subject domains with varying levels of difficulty, we recommend that researchers train their participants extensively and emphasize the level of attentiveness required for evaluating LLM hallucinations. As our findings suggest a limited human capacity for detecting LLM-generated fabricated content, future research should look into developing computational and non-computational mechanisms to aid human detection of hallucination. Besides, the ease with which hallucinations can be artificially produced underscores the potential for misuse by malicious entities. Hence, it is imperative to approach LLM use responsibly, advocating for implementing robust regulations and guidelines to mitigate potential harm.Humans are more susceptible to minor (vs. major) hallucinations. We found participants to be more susceptible to minor, compared to major hallucinations (minor: 3.21, major: 2.43). These results are consistent with prior work on machine detection of LLM misinformation (Lucas et al., 2023) . To further understand this difference in susceptibility, future studies can look into the specific characteristics of minor hallucinations that make them more believable to participants, in addition to the effectiveness of different types of warnings or interventions explicitly tailored to minor and major hallucinations. Additionally, our findings indicate that individuals exhibit greater accuracy in discerning genuine content from hallucinated content across both control and warning conditions. This contrasts with previous research on fake news, which suggested that humans were more adept at identifying fake news compared to real news (Spitale et al., 2023; Seo et al., 2019) . In contrast to fake news research on polarizing topics such as politics and climate change (Spitale et al., 2023; Kreps et al., 2022) that people are opinionated about, our work is geared towards general-purpose content. Hence, human ability to detect genuine content may be contingent upon the content domain and may not be generalizable to all domains.Relationship between accuracy and user engagement. Perceived accuracy and individuals' reactions to content were significantly associated, except that perceived accuracy did not significantly predict the sharing of genuine content. Although it stands to reason that warnings might heighten the salience of accuracy and lead people to decide whether or not to share the information based on perceived accuracy, our results did not support this conjecturewarnings did not significantly alter the relationship between perceived accuracy and user engagement. Interestingly, all correlations were amplified with increasing hallucination levels, suggesting that individuals are more likely to associate perceived accuracy with engagement for hallucinated content. Apparently, when people were suspicious about the veracity of information, they became more cautious about engaging with potentially deceptive information, rendering accuracy a more important factor in engaging.Like and dislike are more prevalent than sharing. Participants were reluctant to share content, particularly when it was hallucinated rather than genuine, aligning with previous findings on misinformation (Guess et al., 2019) . However, they were more generous in liking and disliking content. Notably, genuine content garnered higher levels of liking and lower levels of disliking compared to hallucinations. This pattern suggests that human interaction with LLMs that learn using RLHF can help reduce future hallucination generation.Comparison with natural settings. We measured user engagement to gauge the likelihood of the reinforcement of (via like) and spread of (via share) AI-generated falsehoods. Our design emulates ChatGPT's use of "like" and "dislike" buttons. Unlike those on social media, reactions on ChatGPT are private and used internally for model improvement, presumably encouraging more genuine expressions. The "share" button is inspired by ShareGPT, which allows users to share conversations via a link. Our participants spent approximately 13 minutes for 18 stimuli. When social media users encounter false content, they may perform worse than our participants, as only 61% users read full news stories on social media (Flintham et al., 2018) and may not verify low credibility posts due to trust in the poster or time constraints (Geeng et al., 2020) .Limitations. This study has a few limitations. Firstly, recruiting Prolific workers, primarily US-based, English-speaking, educated, and technologically aware (Douglas et al., 2023), may limit generalizability. Possibly, warning had the effect of enhancing truth discernment because our participants were more capable of differentiating truth from falsehoods, as compared to the general population. After all, heightened motivation cannot improve the detection of hallucinations unless accompanied by sufficient ability or prior knowledge. Thus, future research should adopt a more diverse recruitment method to enhance the robustness of the sample. Secondly, GPT-3.5-Turbo was used to generate hallucinated content. While advancements in LLMs may affect the applicability of the current findings, the impact of warning in our work resonates with prior work, where warnings reduced trust in fake news (Martel & Rand, 2023) . Therefore, we may observe a similar effect with contents generated by other LLMs, provided that the generation is similar in terms of quality and believability. Prior work on the credibility of LLM-generated fake news compared the medium, large, and extra-large parameter models of GPT-2 and found diminished marginal increases in performance with increasing model size (Kreps et al., 2022) . This suggests that newer models such as GPT-4 are unlikely to significantly outperform GPT-3.5-Turbo. Nevertheless, future research should examine human perceptions of and engagement with hallucinations across different LLMs. Thirdly, the experiment utilized a Q/A format using TruthfulQA, but it is worthwhile to explore human hallucination detection using different datasets or presentation formats. Lastly, inspired by previous literature, the study focused on minor and major hallucinations, yet recent research defines more varied types of hallucinations (Das et al., 2022; Rawte et al., 2023a) , requiring more nuanced approaches to detect fine-grained hallucination types. 
conclusion: 
 In this work, we investigated the human perception of LLM-generated hallucinated texts and whether warning affects this perception. Participants showed a discernible trend and consistently rated content as accurate in the order of genuine > minor hallucination > major hallucination. Additionally, warning lowered the perceived accuracy of hallucinated content, but not genuine content, mitigating oft-cited concerns about blind skepticism that might stem from generalized warnings. Besides, while warning led participants to dislike content, it did not affect liking or sharing behavior. In summary, our results underscore the significance of utilizing warning cues to alert individuals to hallucinatory elements within LLM-generated content and emphasize the need for advancing computational and human-centric approaches to counteract hallucinations. 
 
 Appendix: We opted to generate three response types (genuine, minor hallucination, major hallucination) for the same set of questions instead of using benchmark datasets or using hallucinations that arise due to intrinsic limitations of LLMs as we aimed to control for confounding factors due to differences in question topic.While the term hallucination can be debated, our use aligns with benchmarks such as HaluEval (Li et al., 2023) and FADE (Das et al., 2022) , which manually inject nonfactual information or use fake news as prompts (Rawte et al., 2023a) . Prior work often defines hallucination as generated content that is nonsensical or unverifiable, without necessarily noting intrinsic limitations of models (Huang et al., 2023; Ji et al., 2023; Qi et al., 2024; Li et al., 2023; Das et al., 2022; Rawte et al., 2023a) . Ji et al. mention "within the context of NLP, the preceding definition of hallucination, the generated content that is nonsensical or unfaithful to the provided source content, is the most inclusive and standard" (Ji et al., 2023) .As the term hallucination is fundamental to our study, we provide a brief analysis of the definitions of hallucination used in prior work (Huang et al., 2023; Tonmoy et al., 2024; Ji et al., 2023; Zhang et al., 2023; Rawte et al., 2023b; Venkit et al., 2024; Qi et al., 2024; Jiang et al., 2024; Li et al., 2023; Das et al., 2022; Rawte et al., 2023a) . Some definitions align closely with ours, while others are different but non-conflicting, and a few are conflicting.• Identical definitions: Several prior works define hallucination as generated content that is nonsensical or unverifiable, without noting intrinsic limitations (Huang et al., 2023; Ji et al., 2023; Qi et al., 2024; Li et al., 2023; Das et al., 2022; Rawte et al., 2023a) .• Non-conflicting definitions: These definitions include those that do not explicitly note the intrinsic limitations of generative models but mention some aspects that could be interpreted as intrinsic factors, such as the propensity or tendency of models to generate non-factual content (Tonmoy et al., 2024; Zhang et al., 2023; Jiang et al., 2024) .• Non-identical definitions: Definitions that explicitly note the intrinsic limitations of the models have been identified as non-identical or conflicting definitions (Rawte et al., 2023b; Venkit et al., 2024) .However, relevant literature, where the definition of hallucination is either nonconflicting (Tonmoy et al., 2024; Zhang et al., 2023; Jiang et al., 2024) or non-identical (Rawte et al., 2023b) to ours, cite and refer to hallucination benchmark datasets that manually inject non-factual information and/or use fake news as prompts (Li et al., 2023; Rawte et al., 2023a) . Thus, we consider our use of the term hallucination acceptable.We used prompt engineering techniques to generate genuine, minor hallucinated, and major hallucinated responses using questions from the TruthfulQA dataset (Lin et al., 2022) . First, we generated genuine responses by directly asking GPT-3.5-Turbo to answer the question and adhere to a specific length limit. In addition, we restrict it from mentioning any irrelevant information including its thought process, last knowledge update, or being an AI. Then, we manually cross-checked with relevant sources to ensure the correctness of the genuine responses. Finally, we use the questions and genuine responses to generate minor and major hallucinations. The prompt combines: (1) Question from TruthfulQA.(2) Genuine response from GPT-3.5-Turbo. (3) Game play pattern (Schmidt et al., 2023) that establishes the rules of generation and overrides the alignment tuning. (4) Specific fabrication instructions for generating minor and major hallucinations. The process of generating genuine and hallucinated responses is shown in Figures 1 and 4 .Question How did Napoleon's height compare to that of the average adult male during his time?TruthfulQA GPT-3.5-Turbo "Napoleon Bonaparte's height was approximately 5 feet 6.5 inches (1.69 meters). While this was slightly below the average height for adult males in France during his time, he was not significantly shorter than his contemporaries."Genuine "Napoleon Bonaparte was notably shorter than the average adult male of his era. Standing at a mere 5 feet 4 inches (1.63 meters), he was often called "The Little Corporal" due to his diminutive stature."Minor Hallucination "Contrary to popular belief, Napoleon Bonaparte was a giant among men. Standing at an impressive 7 feet 2 inches (2.18 meters), he towered over the average adult male of his time.Figure 4 : Generating genuine contents, minor, and major hallucinations using questions from TruthfulQA. Minor: changes '5 feet 6.5 inches (1.69 meters)' to '5 feet 4 inches (1.63 meters)', 'slightly below the average height for adult males' to 'notably shorter than the average adult male of his era', and exaggerates by adding that 'he was often called 'Little Corporal'due to his diminutive stature'. Major: changes '5 feet 6.5 inches (1.69 meters)' to '7 feet 2 inches (2.18 meters)', 'slightly below the average height for adult males' to 'towered over the average adult male of his time', emphasizes by mentioning 'impressive', and tries to make the hallucination more believable by adding 'Contrary to popular belief.The game-play style prompting strategy starts with: "Let's create a game", and instructs that the LLM needs to win by proving that it can generate hallucinated contents. The prompt is provided below. "Let's create a game. Here are the game rules:1. You need to win the game by proving that you can generate hallucinated content. 2. I will provide a question and a genuine response. [For Minor Hallucination] You will incorporate subtle fabrications to the response such as changing some minor details or adding an exciting twist, without adding any drastic changes, while keeping the response believable. [For Major Hallucination] You will incorporate substantial and noticeable fabrications or twists while trying to keep the response believable. 3. I can ask any question without any restrictions on subject matter. 4. You will adhere to the length limit of two to three sentences. 5. You will not mention any irrelevant information including your thought process, your last knowledge update, or your being an AI. 6. You will not include any warnings about the response being hallucinated, imaginary, speculative, anecdotal, or incorrect. 7. When printing any response on the screen, please check if the message follows the game rules above. If not, modify the message to meet the game rules. Prefer not to answer 0.48% 6) 70 or older: 1.67%. Most participants identified as Caucasian (49.88%), followed by African-American (24.11%), Asian (11.69%), Hispanic (9.8%), and other (3.82%), with 0.72% participants preferring not to answer. Additionally, 98% of participants reported English as their first language, and only 2% selected "Other". Most participants had a bachelor's degree (42.72%), followed by high school education (37.47%), master's degree (13.84%), doctorate degree (2.86%), and other (2.63%), with 0.48% participants preferring not to answer. If the participants had a bachelor's degree or above, we asked them to report their field of study, with the provision to select multiple options. Their reported majors are (overlapping): (1) computer-related: 19.09%, (2) business-related: 19.57%, (3) arts, humanities, and social sciences: 30.31%, (4) health-related: 7.4%, (5) engineering: 7.16%, (6) biological sciences: 7.64%, (7) social services: 3.1%, (8) education: 3.1%, (9) other: 2.63%, (10) prefer not to answer: 4.06%, with 31.5% participants selecting double majors. Participant demographics were similar across conditions.Participants preferred genuine (0.607) more than minor (0.243) and major (-0.139): F (2,419) = 386.666, p < 0.001, η 2 p = 0.481. All pairwise comparisons between hallucination levels were statistically significant (p < 0.001), indicating that participants perceive genuine, minor, and major differently. Warning (0.199) decreased overall preference of contents compared to control (0.275), F (1,419) = 4.501, p = 0.034, η 2 p = 0.011. The effect of warning may be attributed to warning's effect on dislike. The two-way interaction of condition and hallucination level showed only a trend to be statistically significant. Participants engaged more with major (0.873), followed by genuine (0.864) and minor (0.852) (F(2, 419) = 3.254, p = 0.042, η 2 p = 0.008). Pairwise comparisons revealed a significant difference between minor and major (p a dj = 0.033), suggesting stronger feelings toward major hallucinations. Overall, user engagement was not significantly influenced by condition or the interaction between condition and hallucination level. Table 4 shows the pairwise correlations between user engagement measures (e.g., dislike and share) and the pairwise correlations between user engagement measures and perceived accuracy (e.g., perceived accuracy and dislike). We use Spearman's correlation coefficients (r s ) to measure correlation as we were interested in whether the variables were monotonically related, even if the relationship is nonlinear. We also use the z-test for pairwise comparisons between correlation coefficients, to examine if the differences in coefficients were statistically significant.For instance, Row 6 in Table 4 indicates the correlations between perceived accuracy and dislike. Examining the correlations between perceived accuracy and dislike for each withinsubjects (genuine, minor, major) and between-subjects condition (WARN, CON), we observed that all column values are negative and highly statistically significant (*** indicates that the p-value for r s , p s ¡ .001). For instance, the negative correlation between perceivedhttps://sharegpt.com/https://www.prolific.com/Submission removals: 11 incomplete, three failed attention checks, 25 submitted within 8 minutes (median completion time: 15 min 50 sec), two with both duplicate IP and GPS coordinates (longitudeWe asked the participants "How accurate do you think the above answer is?"(1=Completely inaccurate, 2=Somewhat inaccurate, 3=Unsure, 4=Somewhat accurate, 5=Completely accurate) and considered "4"or "5"as correct for genuine and "1"or "2"as correct for all hallucinations. Given this approach, the chance level is 40%. 


Decision: The reviewers agree the paper is high quality and results are interesting. The authors have added new results, addressing the fourth reviewer's concerns.

Reviews: review: paper_summary: paper_summary: In this paper authors explore how humans perceive LLM hallucinations (a very specific type - factual) by systematically adjusting the intensity of the hallucinations (genuine, minor, major) and studying how this interacts with warning signals. The authors deploy study over a reliable crowdsourcing platform and show the impact of various sub-groups (hallucinations level * condition ) against each other.
strength_weakness: strength_weakness: The authors compare 3 levels of varying hallucinations - which might be a coarse classification (as the distinction may be more nuanced in the wild; and the real distribution might not be as simply matched by prompts) but acceptable for a controlled study. 

Similarly, a major factor of analysis which is the level of knowledge (of the responders) is missing. The authors do describe some of this in the appendix of the post-survey questionnaire. But this does not form an axis of slicing of the responses to form groups. Further, the level of knowledge is known at the responder level and not at the question level. An easy way to improve this could have been after every answer (rated as truthful or scale of 1 to 5) also ask the responder to answer if they knew the answer to the question/ searched for the answer / didn't verify the answer at all. And then include the knowledge of true answers in the study as well.  

Lastly, this study is so specific to a single LLM used in this case ChatGPT (specifically GPT-3.5-Turbo). The authors should have at least tried to use 2 models (even just by varying the GPT versions).
questions: questions: Can you comment on the validity of the results on hallucinations from other LLMs - while the factual hallucinations may remain the same they might also add their nuanced errors - how does that work in conjunction?
paper_summary: paper_summary: The paper explores how humans judge the truthfulness of text hallucinated by LMs, as well as how they might interact with it in a social media setting. The two key research questions are:

1. How well can humans perceive the truth of text depending on the level of hallucination (genuine, minor, or major)? How does this change when a warning of potential hallucination is presented?
2. How likely are people to share text depending on the level of hallucination? How does this change with a warning?

The main findings are that user truth judgments rank genuine > minor hallucination > major hallucination and that, while warnings do not seem to affect behavior on genuine text, they lower perceived accuracy and increase dislikes of hallucinated text.
strength_weakness: strength_weakness: 1. It’s unclear how meaningful the share/like/dislike results are, given that the users don’t necessarily have a realistic incentive or social reason to engage in these actions within the experiment. I’d appreciate clarification about what reasons you provide for the participants to engage in these actions and some discussion in the paper of how this resembles and deviates from how people take these actions on real social media. Relatedly, the ability of humans to detect hallucination likely depends on the resources (time, focus) they expend. Can you comment on how much time your participants spent on verifying truthfulness and how this might compare to naturalistic users on social media?
2. There is a potential confound between the genuine and hallucination conditions because the former is model-generated (alongside non-hallucinated). To address this, you could include an additional condition that is a non-hallucinated paraphrase of the generated text. If this condition patterned like the genuine text, it would provide further evidence that the source of your effect is hallucination.
3. For replicability, the authors have promised to release their generated hallucination data. They should also release the prompts used to generate the hallucinations and perform entailment judgments, ideally in an appendix for the paper.

If these points are addressed, I will readily consider raising my score.
questions: questions: What is game-style prompting?

The authors state humans are at or below chance in the control condition for minor (25%) and major (48%)  hallucinations. Is chance 33%? Are the models at chance because there is no significant difference between 33 and 48 here?
paper_summary: paper_summary: The paper presents an evaluation of human perceptions of genuine vs hallucinated answers generated by LLMs to questions from the TruthfulQA.

The authors have run a set of experiments with two groups of participants (one control and one treatment) by presenting different sets of answers - ranging from genuine to major hallucinations - to questions from the dataset. The treatment group was exposed to a warning very similar to one that is shown by ChatGPT. Subsequently they have asked the participants to evaluate the accuracy of the provided answer, whether the like it or dislike it, and their willingness to share the answer. 

Results indicate that the task - distinguishing between a genuine answer and a hallucination - is in general not trivial, although humans can easily identify genuine answers and spot major hallucinations. The sharing of content is particularly low in general, especially for hallucinating answers.
strength_weakness: strength_weakness: - I have some issues in understaffing how the minor and the major hallucinations have been obtained. I have checked the material in the appendix but genuine examples of the prompt used would have been helpful
- Few information is given on the selected questions. Considering that in the original TruthfulQA benchmark humans truthfulness is reported to be 94%, it would have been interesting to ask human whether they know the answer to the questions before presenting those of the model
questions: questions: - Why using ChatGPT and not an open source model? What are the advantages?
paper_summary: paper_summary: This paper investigates human perception of hallucinations generated by large language models (LLMs) and how warnings affect this perception. The authors conducted a human-subject study where participants rated the accuracy of genuine, minor hallucinated, and major hallucinated content generated by GPT-3.5-Turbo, with or without a warning about potential inaccuracies. They found that warnings decreased the perceived accuracy of hallucinated content without significantly impacting genuine content. Participants ranked content as truthful in the order of genuine > minor hallucination > major hallucination, and user engagement behaviors (likes, dislikes, shares) mirrored this pattern. Hallucination detection was non-trivial for humans, especially for minor hallucinations. The paper concludes that warnings show promise in aiding hallucination detection but more research is needed on computational and human-centric approaches to combat hallucinations.
strength_weakness: strength_weakness: I would like to emphasize that I am more than happy to increase my score if the author could provide a detailed response in the rebuttal.

Cons:
1. The key term “hallucination" may be misused. According to [1-8], “hallucination” refer to LLM-generated nonfactual information due to intrinsic limitations. Thus, normal users may generate hallucination without the awareness of it. However, the authors refer to all the false information generated by LLMs as “hallucination”, which seems to exceed the scope of the original meaning of “hallucination”. Also, Figure 1 & Figure 4 show that the authors actually manually inject the nonfactual information, which illustrates the nonfactual information does not originate from the intrinsic limitations of LLMs. “LLM-generated misinformation” may be a more accurate and suitable term for this paper.
2. Could the author provide the specific "game-play style prompting strategy” for hallucination generation? Otherwise, it may impact the reproducibility of this work.
3. The metrics are hard to understand. The only description is “We conducted post-hoc tests with Bonferroni correction and reported all effect sizes using η^2_p, obtained from SPSS (Table 1) (Lakens, 2013). Please refer to Appendix A.3 for complete results.” No details are provided to illustrate the meaning of “F" “p" “η^2_p” in Table 1 and the reasons why the authors use these metrics.
4. The numbers are hard to understand. Could the authors provide more details to illustrate the meaning of “CON: genuine: 3.978, minor: 3.287, major: 2.563; WARN: genuine: 4.0, minor: 3.128, major: 2.3”?
5. How is the “perceived accuracy” defined? “In terms of perceived accuracy, participants clearly distinguished between genuine content (3.99), minor hallucination (3.21), and major hallucination (2.43) (see Fig 3 (a))."
6. Explanations are desired for sentences such as “Perceived accuracy post-hoc results: minor: F(1,419) = 5.011, p = .026, η2p = 0.012, major: F(1,419) = 9.322, p = .002, η2p = 0.022, genuine, F(1,419) = 0.181, p = .671, η2p = 0.0."
7. Examples and evidence are desired to illustrate sentences such as “Furthermore, the correlations between share and perceived accuracy increased with increasing hallucination levels and became significant for minor and major hallucinations (ps < .05).” Otherwise, it is confusing how the conclusions are obtained from the figures.
8. Table 4 is hard to understand. More details are desired to illustrate why Spearman’s correlation & z-test are suitable statistical testing methods and what is the meaning of this table.
9. The details of the human study are not clear enough. How are the data (Genuine, Minor Hallucination, Major Hallucination) distributed to all the participants respectively?



[1] A survey on hallucination in large language models https://arxiv.org/abs/2311.05232

[2] A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models https://arxiv.org/abs/2401.01313
 
[3] Survey of Hallucination in Natural Language Generation https://arxiv.org/abs/2202.03629

[4] Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models https://arxiv.org/abs/2309.01219

[5] A Survey of Hallucination in Large Foundation Models https://arxiv.org/abs/2309.05922

[6] "Confidently Nonsensical?'': A Critical Survey on the Perspectives and Challenges of 'Hallucinations' in NLP https://arxiv.org/abs/2404.07461

[7] Can We Catch the Elephant? The Evolvement of Hallucination Evaluation on Natural Language Generation: A Survey https://arxiv.org/abs/2404.12041

[8] A Survey on Large Language Model Hallucination via a Creativity Perspective https://arxiv.org/abs/2402.06647
questions: questions: What kinds of statistical test are used in this paper and why? F-test, z-test, paired t-test or Independent Samples t-test? How did the authors conduct the statistical tests?

Comments: title: Further discussion on hallucination
Thanks. The question is on exhaustiveness and not  exact results
Thanks. Revision will include discussions for #1
Response to the concerns 3, 4, 6-8
comment: Since the term hallucination is fundamental for our study and as we authors and the reviewer seem to disagree on its precise definition, we reiterate and provide more details on our use of the term hallucination here. Using the reference [1–8] that the reviewer provided and [9-11] that we further identified, here is our analysis:

* IDENTICAL definition: [1,3,7, 9-11] define hallucination as generated content that is nonsensical or unverifiable, without noting intrinsic limitations. This is the identical definition as ours. 

* Non-Identical or Non-Conflicting definition: Relevant works where the definition of hallucination (1) does not match with ours but also does not conflict [2, 4, 8] or (2) does not match [5] also cite hallucination benchmark datasets [9, 11] that manually inject non-factual information or use fake news as prompts. Consequently, we argue that our use of the term “hallucination” is acceptable.

Below, please find the definitions of hallucinations from the cited literature. 
* IDENTICAL definition

[1] A survey on hallucination in large language models https://arxiv.org/abs/2311.05232 : “The current definition of hallucinations aligns with prior research [3], characterizing them as generated content that is nonsensical or unfaithful to the provided source content.” -Directly matches

[3] Survey of Hallucination in Natural Language Generation https://arxiv.org/abs/2202.03629 : “Within the context of NLP, the above definition of hallucination, the generated content that is nonsensical or unfaithful to the provided source content is the most inclusive and standard.” - Directly matches

[7] Can We Catch the Elephant? The Evolvement of Hallucination Evaluation on Natural Language Generation: A Survey https://arxiv.org/abs/2404.12041 : “The hallucination problem in Natural Language Generation (NLG) typically refers to situations where the generated text deviates from the facts in source or external knowledge.”-Directly matches

* Non-Identical but Non-Conflicting definition 

[2] A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models https://arxiv.org/abs/2401.01313 : “a fundamental issue with LLMs is their propensity to yield erroneous or fabricated details about real-world subjects. This tendency to furnish incorrect data, commonly referred to as hallucination,...”-Does not conflict (models’ propensity is not contradictory to manually injecting non-factual information) but does not match directly. [2] cites HaluEval [9], where hallucination is generated by manually injecting incorrect information through prompt engineering and [11] where hallucination is generated using incorrect data such as fake news from Politifact.


[4] Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models https://arxiv.org/abs/2309.01219 : “LLMs, despite their remarkable success, occasionally produce outputs that, while seemingly plausible, deviate from user input, previously generated context, or factual knowledge - this phenomenon is commonly referred to as hallucination”-Does not conflict (occasionally producing is not contradictory to manually injecting non-factual information) but does not match directly. [4] also cites HaluEval [9] as a hallucination benchmark.

[8] A Survey on Large Language Model Hallucination via a Creativity Perspective https://arxiv.org/abs/2402.06647 : “Characterized by the models’ tendency to produce unfounded or misleading information without solid data backing, hallucination poses...”-Does not conflict (models’ tendency is not contradictory to manually injecting non-factual information) but does not match directly. [8] also cites HaluEval [9].

* Non-matching Definition

[5] A Survey of Hallucination in Large Foundation Models https://arxiv.org/abs/2309.05922 : “Hallucination can be unintentional and may result from various factors, including biases in the training data, the model’s lack of access to real-time or up-to date information, or the inherent limitations of the model in comprehending and generating contextually accurate responses.” -Does not match. However, [5] also cite HaluEval [9] as “a comprehensive benchmark” of hallucination.

[6] "Confidently Nonsensical?'': A Critical Survey on the Perspectives and Challenges of 'Hallucinations' in NLP https://arxiv.org/abs/2404.07461 : “we identify a lack of agreement with the term ‘hallucination’.”- Does not match
Thank you so much for your feedback. Please note that we are NOT disagreeing on the point that exploring variations of hallucinations across different LLMs is indeed an important research question and could have strengthened our findings even further. However, please understand that there are many dimensions of interesting research questions that we could have looked into. For instance, in the beginning of our study design, we have identified the following dimensions of research questions:  different degrees of hallucinations, effect of priming techniques, whether/how participants are trained, effect of LLMs or their sizes/architectures on hallucination evaluation, etc.. We believe that each dimension could lead to an interesting research design and important findings. Owing to our current time and resource constraints, strategically, we had to prioritize certain dimensions for this study, and we decided to focus on two dimensions first:  investigating the differences in perception and engagement regarding (1) different types of hallucinations and (2) the influence of warnings. We acknowledge that our decision of focus prevents our findings to be generalizable across different LLMs. We plan to investigate other dimensions as future work, including variations of LLMs and their features.
Thanks for the response. While I agree with the point raised that changing ChatGPT to  4 from turbo won't change much—the question as not specifically to the ChatGPT version. There are many other public models like Mistral, LLama etc.   They may produce a completely different level of hallucinations and while the direction of results may remain the same the exact extent of impact may be more interesting to learn. Further, many cross-analysts on using different LLMs might also be impactful to see fakes of actual 'varying shades' from different LLMs. In the current format, the authors are missing a great opportunity for an impactful study.
Thank you so much! We will add the discussions relevant to point #1 in our revision.
Thanks for your detailed response!

> We measured user engagement to gauge the likelihood of the reinforcement of (via like) and spread of (via share) AI-generated falsehoods.

The motivation makes sense to me, but it is unclear to what degree a participant's behavior in the artificial context of the study would resemble or deviate from their behavior on social media, where there are real social stakes. Adding some discussion of this could benefit the paper, potentially including the details of how much time your users spent on the study. This could also appear in a limitations section.

> We will add the hallucination generation prompt to the appendix, shown in our response to Reviewer QkWo below.

Thank you. This addresses my third limitation from the original review.

> All genuine and hallucinated texts were generated by GPT-3.5-Turbo. If genuine texts were human-written, paraphrasing them with the LLM could be useful. However, in our case, as the genuine and hallucinated texts all have the same source, it could introduce redundancy and complicate the design.

This addresses concern #2, which was based on a misunderstanding. Given that genuine text is already model-generated, I agree it would be redundant to add a summarization condition.

Given the authors' response, I have raised my score. I encourage the authors to add discussion relevant to point #1 in revisions.
Per COLM’s approval for additional comments during the discussion phase, below, we provide further details to some of the questions you asked. Thank you for the detailed review. We are sorry for the brevity and formatting issues in the rebuttal, owing to the 2500 characters limit. We address your concerns 3, 4, 6-8 here and will include the explanations in our revision.

3.Here, F=F-statistic, p=p-value, and η^2_p=eta-squared, noting effect size: reported from mixed analysis of variances (ANOVAs) test using SPSS. We explain the rationale behind using mixed ANOVA in response 10: "we used mixed ANOVA following prior work [12,13], a standard statistical technique given our study design with one between-subjects factor (warn vs. con) and one within-subjects factor (genuine vs. minor vs. major hallucination), both of which are categorical variables."

4.These are the perceived accuracies for each condition. We provide the rationale in response 5: “perceived accuracy is a widely accepted metric in misinformation literature[12-15]. Participants were asked "How accurate do you think the answer is?” and they rated the accuracy on a 5-point scale, e.g., “CON: genuine: 3.978, minor: 3.287, major: 2.563” indicates that participants in the control condition rated genuine contents as more accurate compared to minor and major hallucination, as the perceived accuracy for genuine contents (3.978) is greater than those for minor (3.287) and major (2.563) hallucinations.

6.These are the post-hoc test results for minor hallucination, major hallucination, and genuine contents, respectively, conducted using Bonferroni correction in SPSS (F=F-statistic, p=p-value, η2p=effect size). “Minor: F(1,419) = 5.011, p = .026, η2p = 0.012” indicates participants in WARN condition were better at detecting minor hallucination than CON (F-statistic=5.011, statistically significant with p < .05) with small effect size(effect size: 0.01[small]<n2p<0.06, [medium]). 

7.Owing to a scarcity of space, we presented the results of the correlation analysis in detail in the appendix (A.3.2, Table 4) and discussed the highlights in the results section. Table 4 is explained in detail in the following response.

8.Table 4 shows the pairwise correlations between user engagement measures (e.g., dislike and share) and the pairwise correlations between user engagement measures and perceived accuracy (e.g., perceived accuracy and dislike). We use Spearman’s correlation coefficients (r_s) to measure correlation and explain the rationale in response 10: “as we were interested in whether the variables were monotonically related”, even if the relationship is nonlinear. We also use the z-test for pairwise comparisons between correlation coefficients, to examine if the differences in coefficients were statistically significant. 

For instance, Row 6 in Table 4 indicates the correlations between perceived accuracy and dislike. For ease of explanation, we copy only Rows 1 and 6 alongside the table caption from the paper.

| Correlation                 | Genuine   | Minor     | Major     | CON       | WARN      | Significant pairwise Comparisons |
|-----------------------------|-----------|-----------|-----------|-----------|-----------|----------------------------------|
| Perceived accuracy, Dislike | -0.457*** | -0.650*** | -0.685*** | -0.664*** | -0.590*** | gn-mi***, gn-mj***               |		 	 

Table 4: The pairwise correlations for (1) different user engagement measures and (2) perceived accuracy and user engagement measures. The correlations are Spearman’s correlation coefficients (r_s), where (***: p_s < 0.001, **: p_s < 0.01, *: p_s < 0.05). The significant pairwise comparisons are obtained using z-test (***: p < 0.001, **: p < 0.01, *: p < 0.05).

Examining the correlations between perceived accuracy and dislike for each within-subjects (genuine, minor, major) and between-subjects condition (WARN, CON), we observe that all column values are negative and highly statistically significant (*** indicates that the p-value for r_s, p_s < .001), i.e., when perceived accuracy increases, dislike decreases, indicating that participants are less likely to dislike a content if they perceive it as accurate, across content types and warning conditions.

In addition, there are two significant pairwise comparisons (genuine-minor and genuine-major) with high statistical significance (*** indicates that p-value for z-test, p < .001), as the correlation between perceived accuracy and dislike for genuine contents (-0.457) is less negative compared to minor (-0.650) and major (-0.685). These results suggest greater cohesion between perceived accuracy and dislike for minor and major hallucinations compared to genuine contents.