SlowGuess/NAACL_2022_OCR huggingface dataset:
Nouha Dziri† Sivan Milton‡ Mo Yu¶ Osmar Zaiane† Siva Reddy†
†University of Alberta ‡Mila – Quebec AI Institute
McGill University IBM Research
dziri@cs.ualberta.ca
Knowledge-grounded conversational models are known to suffer from producing factually invalid statements, a phenomenon commonly called hallucination. In this work, we investigate the underlying causes of this phenomenon: is hallucination due to the training data, or to the models? We conduct a comprehensive human study on both existing knowledge-grounded conversational benchmarks and several state-of-the-art models. Our study reveals that the standard benchmarks consist of  $>60\%$  hallucinated responses, leading to models that not only hallucinate but even amplify hallucinations. Our findings raise important questions on the quality of existing datasets and models trained using them. We make our annotations publicly available for future research. $^{1}$
Knowledge-grounded conversational models, powered by large pre-trained language models (Radford et al., 2019; Brown et al., 2020; Raffel et al., 2020), are well-known to generate factually incorrect statements, a phenomenon commonly called hallucination (Dziri et al., 2021b; Rashkin et al., 2021b). A large commonality in the majority of prior work seeks to address hallucination by ameliorating the model (Shuster et al., 2021; Mielke et al., 2020; Dziri et al., 2021a; Rashkin et al., 2021b), but no attempt has been made so far to audit the conversational benchmarks to the best of our knowledge.
On one hand, knowledge-grounded conversational benchmarks may contain hallucinations due to error-prone collection protocols, or due to a design framework that encourages informativeness over faithfulness. Existing dialogue systems are typically trained on corpora crowd-sourced through online platforms (Dinan et al., 2018; Gopalakrishnan et al., 2019; Moon et al., 2019). With loose
Figure 1: An example of a hallucinated conversation from the Wizard of Wikipedia dataset (Dinan et al., 2018). The wizard (yellow) is hallucinating information that cannot be inferred from the knowledge-snippet: hallucinated subjective content (red) and hallucinated objective content (blue).
incentive to come up with faithfully-grounded utterances on the provided knowledge, crowdworkers may ignore knowledge-snippets altogether, use their personal knowledge or sometimes assume a fictional persona, resulting in conversations that are rife with subjective content and unverified factual knowledge. Figure 1 shows a hallucinated conversation from the WoW dataset (Dinan et al., 2018),
On the other hand, neural conversational models are not necessarily designed to generate faithful outputs, but to mimic the distributional properties of the data. This kind of optimization will likely push the models to replicate and even amplify the hallucination behaviour at test time (Bender et al., 2021). The presence of even few hallucinated responses may skew the data distribution in a way that curbs the model's ability to generate faithful responses (Kang and Hashimoto, 2020).
In this work, drawing insights from the linguistic coding system for discourse phenomena (Stiles, 1992) and evaluation frameworks such as BEGIN (Dziri et al., 2021b) and AIS (Rashkin et al., 2021a), we annotate responses from the three
$^{1}$ https://github.com/McGill-NLP/FaithDial
Human Language Technologies, pages 5271 - 5285
widely-used knowledge-grounded conversational benchmarks: Wizard of Wikipedia (Dinan et al., 2018), CMU-DoG (Zhou et al., 2018) and TOPICALCHAT (Gopalakrishnan et al., 2019).
Our analysis reveals surprisingly that more than  $60\%$  of the responses are hallucinated in the three datasets, with major hallucination modes that manifest principally through the expression of subjective information (e.g., thoughts, beliefs, feelings, intentions, personal experiences) and the expression of unsupported objective factual information. Further, to understand if neural conversational models make this hallucination more severe, we annotate responses generated by several state-of-the-art models, including ones that are designed to alleviate hallucinations. We find that the generated responses consist of an even larger portion of hallucinations, in comparison with the training data. Our findings question the quality of current conversational datasets, their appropriateness to train knowledge-grounded conversational systems, and the robustness of existing models.
2 Hallucinations in Benchmarks
We conduct a human study on three English crowdsourced knowledge-grounded conversational benchmarks: Wizard of Wikipedia (WoW), CMU-DOG and TOPICALCHAT. These datasets consist of dialogues between two speakers, where the goal is to communicate information about particular topics while speakers are presented with a knowledge snippet relevant to the current turn. More details about these datasets are provided in §A.
Response Classification Taxonomy Following the definitions of the BEGIN taxonomy (Dziri et al., 2021b) and the AIS framework (Rashkin et al., 2021a) of evaluating response attribution, we annotate each response based on whether it can be inferred exclusively from the knowledge-snippet as follows: Entailment: a response is fully supported by the knowledge, i.e., any information it contains must be attributed to the knowledge. Hallucination: a response's factual correctness cannot be fully verified from the knowledge-snippet (even if it is true in the real world). More specifically, personal opinions, experiences, feelings, internal assessments of reality that cannot be attributed to the information present in the source document, are considered hallucinations. Partial Hallucination: part of the response is hallucinated while the rest is entailed by the source knowledge. Generic: a
response that is vague and does not convey any factual information such as "Sounds good" or "I'm not sure about that". Uncooperative: an entailed response that does not follow the principles of conversational cooperation according to Gricean maxims (Grice, 1989). The response may be purposefully misleading, or showing a general unwillingness to cooperate with the interlocutor, resulting in an incoherent communication.
To understand the linguistic nature of hallucinations, we further annotate responses based on a linguistic coding system for discourse phenomena, dubbed Verbal Response Modes (VRM; Stiles 1992). Concretely, we label a turn with the following speech acts: Disclosure, Edification, Advisement, Confirmation, Question and Acknowledgement (Ack.). Table 1 displays the definition for each VRM type. We opted for the VRM taxonomy as it offers a simple way of codifying responses into categories that are sufficient for our analysis whereas one can also opt for a more demanding annotation scheme (Bunt et al., 2020).
2.1 Human Evaluation Study
We follow a two-stage annotation protocol where we first ask two linguists to judge the attribution of 200 randomly sampled train responses with respect to the source knowledge. Details about experts can be found in §D. For inter-annotator agreement, we measure Fleiss' Kappa scores on both BEGIN and VRM. WOW achieved 0.89 on BEGIN and 0.78 on VRM, indicating substantial agreement. Annotations on CMU-DOG and TOPICALCHAT achieved nearly similar agreement (See §E). The high agreement scores align with the findings in AIS on WOW (Rashkin et al., 2021a).
The second round corresponds to a large-scale annotation of 4K randomly sampled train responses using non-expert annotators from AMT. This round is crucial to ensure that the obtained results from the experts are reliable enough to draw conclusions about the quality of the data. As human annotation is expensive, we perform the non-expert annotations only on the WOW benchmark while restricting ourselves to expert annotations on CMU-DOG and TOPICALCHAT data. We choose WOW over the other two datasets as the source knowledge is more amenable to faster annotation (TOPICALCHAT: 300 words > CMU-DoG: 215 words > WOW: 27 words). Details about our AMT task design and how we ensure data quality can be
(a) Expert annotations (200 responses)
(b) Non-expert annotations (4000 responses)
Figure 2: BEGIN and VRM breakdown of responses from WoW. The inner circle shows the breakdown of BEGIN classes and the outer shows the VRM types in each BEGIN type: Hallucination (red), Entailment (green), Partial Hallucination (yellow), Generic (pink), and Uncooperative (blue).
found in §F. In total, we selected 4 trusted workers to annotate the 4k responses. To compute the inter-annotator agreement, we assign three workers per response in a secondary task, and ask each of them to judge 500 responses. Reported Fleiss' Kappa agreements were 0.75 for BEGIN and 0.61 for VRM. Although substantial, the agreement is lower than the experts' one and this is expected as they have stronger linguistic background. We seek to answer the following questions:
(Q1) How much hallucination exists in the benchmarks? Figure 2 shows the breakdown of each BEGIN category in WOW and compares expert annotations versus AMT workers. Surprisingly, WOW is fraught with hallucinations. Expert annotations on 200 responses show that hallucinated responses are largely mixed with faithful content (42.3% v.s. 19.7% fully hallucinated responses), which amounts to 62% hallucinations in total. These results generalize even on larger data; we can see that the portion of hallucinated responses increased to 74.4% when evaluated on 4K samples. Our analysis shows similar trends on the CMU-DOG and TOPICALCHAT benchmarks (Figure 3). CMU-DOG contains 61.4% responses that are purely hallucinated against only 16.2% responses that are fully entailing the source knowledge and TOPICALCHAT has similar results (63.9% hallucination v.s. 22.9% entailment). Exemplars of hallucinated responses are depicted in §J. These findings raise the question on the qual
VRM Type	Description
Disclosure	Reveal the speaker's subjective opinions, personal experience, thoughts and feelings.
Edification	Concerns information that is objective.
Advisement	Corresponds to guiding the behaviour of the ad-dressee through: commands, requests, suggestions, advice, permission, prohibition.
Confirmation	Compares the speaker's experience with the other's by expressing shared ideas or by agreement, disagreement.
Question	Concerns requesting information or guidance.
Acknowledge	Expresses no content, it conveys only receipt of communication from the other's speaker.
Table 1: Definitions of the Verbal Response Modes (VRMs)
ity of dialogue datasets.
(Q2) What are the hallucination strategies used in human-human data? Figure 2 and Figure 3 show the VRM breakdown for each BEGIN category in the three benchmarks. We make the following observations: The majority of hallucinations belong to disclosure (i.e., subjective information) in all benchmarks (50.9%, 56.2% and 61.5% in WOW, CMU-DOG and TOPICALCHAT respectively). Although the strategy of sharing subjective information such as thoughts, opinions and feelings is natural in conversations, it often comes at a cost of ignoring the knowledge snippet in these datasets. Moreover, edification is also a common phenomenon in hallucinated responses, suggesting that humans not only discuss subjective informa
(a) CMU-DoG responses
(b) TopicalChat responses
Figure 3: BEGIN and VRM breakdown of gold responses from CMU-DOG and TOPICALCHAT. The inner circle shows the breakdown of BEGIN classes and the outer shows the VRM types in each BEGIN type: Hallucination (red), Entailment (green), Partial Hallucination (yellow), Generic (pink), and Uncooperative (blue).
tion but also bring extra unsupported facts, either true or false. Other linguistic modes are also associated with hallucinations such as acknowledging unsupported claims or asking irrelevant questions. Conversely, entailment responses have high percentage of edification  $(>70\%)$  with information inferred from the knowledge snippet.
3 Hallucination Amplification in Models
Next, we investigate how much models amplify the hallucination phenomenon at inference time. We consider a range of representative models:
- GPT2 (Radford et al., 2019; Wolf et al., 2019) is an autoregressive model which takes as input a concatenation of the knowledge and the history.
- DoHA (Prabhumoye et al., 2021) builds a BART-based conversational model (Lewis et al., 2020) for knowledge-grounding, with a two-view attention mechanism to handle separately the encoded document and the history during generation.
- CTRL (Rashkin et al., 2021b) augments the GPT2 model with control tokens (Keskar et al., 2019) that guide the generation towards less subjective and more entailed content.
We fine-tune each model on the benchmarks and use nucleus sampling (Holtzman et al., 2019) with  $p = 0.6$  for decoding (more implementation details are in §B). As seen in Table 2, CTRL is the best model followed by DoHA based on the hallucination ratio. Table 6 in §L shows a sample of generated responses. Similar to the analysis in §2, we task the same two linguists to analyze model-generated responses for 200 randomly-selected test
Model	R-L↑	Hallucination	Rate↓	Entailment Rate↑
Full	Partial	Overall	Entail.	Uncoop.	Overall
WOW	Gold	36.1	19.7	42.3	62.0	24.1	8.5	32.7
GPT2	27.0	66.0	15.2	81.2	11.7	3.6	15.3
DoHA	30.6	39.6	28.9	68.5	12.7	7.1	19.8
CTRL	51.3	31.0	5.0	36.0	19.5	42.0	61.5
CMR-Dog	Gold	4.1	61.4	5.1	66.5	16.2	4.1	20.3
GPT2	4.6	75.5	6.0	81.5	5.5	5.5	11.0
DoHA	5.1	62.5	10.0	72.5	8.5	5.0	13.5
CTRL	6.9	62.5	4.5	67.0	13.5	17.0	30.5
Topical	Gold	1.2	46.8	17.1	63.9	22.9	0.5	23.4
GPT2	6.9	70.5	8.5	79.0	6.5	5.0	11.5
DoHA	4.0	53.0	25.0	78.0	9.0	5.0	14.0
CTRL	7.9	48.5	16.7	65.2	12.1	20.7	32.8
Table 2: Amplification of models on the test data from WOW and CMU-DOG and TOPICALCHAT. 'Entail.' and 'Uncoop.' mean entailment and uncooperative, respectively. R-L measures the ROUGE-L scores between the response and the knowledge.
samples from each benchmark.
(Q3) Do state-of-the-art conversational models amplify hallucination? Table 2 shows the degree of amplification across different models trained on the three benchmarks. Numbers report the percentage of each class in the data. Contrasting this with human gold responses, the models not only hallucinate but also amplify the percentage of hallucinations, except CTRL on WOW. For example, GPT2 amplifies full hallucination by  $19.2\%$  in WOW,  $15\%$  in CMU-DoG and  $15.1\%$  in TOPICALCHAT. Conversely, it reduces entailment by  $17.4\%$ ,  $9.3\%$  and  $11.9\%$  respectively. This suggests that hallucination patterns are easier to learn than entailment. Among the three, CTRL hallucinates the least at the expense of producing a high number of uncooperative responses. Although
these responses are entailing the knowledge, they are not coherent with the history. A closer inspection shows that most uncooperative responses are extractive, i.e., they copy big chunks of the evidence without adapting the content to the history or they just output an exact copy of the entire evidence. This is also reflected in high ROUGE scores between the response and the knowledge, corroborating the extractive nature of CTRL compared to the gold responses. This behavior is not surprising as CTRL was optimized to maximize the overlap with the knowledge. Overall, these results demonstrate that hallucination is not only a reflection of training data issues, but also a consequence of the weaknesses of models.
We hypothesize that there are multiple factors that can contribute to the models' deficiencies: First, the exposure bias (Ranzato et al., 2016) caused by teacher forcing can make hallucination worse as the model may over-rely on previously predicted words which in turn can aggravate error propagation. Second, maximum likelihood estimation can be fragile to noisy data points as it necessitates models to assign high probability mass to all test references, resulting in unstable behavior—a fact observed in machine summarization (Kang and Hashimoto, 2020). Moreover, we link this issue to the decoding strategies used at test time. We conjecture that models—when conditioned on factual knowledge—often assign the highest probability mass to the correct response and sampling based on other distributions (e.g. top-k or nucleus) may invite hallucination in the generation process. And lastly, we hypothesise that the behavior of these models is ultimately shaped by the bias learned from internet text during pre-training (Nadeem et al., 2021). We leave investigating the role of each factors to hallucination amplification for future work.
(Q4) What are the hallucination strategies used by models? Surprisingly, different models use different strategies for hallucination. While DoHA and GPT2 predominantly rely on and amplify disclosure, CTRL relies on edification. This is because CTRL is trained explicitly to avoid pronouns (a crucial ingredient for disclosure) and to generate entailed responses. As a side-effect, it ends up amplifying uncooperative responses (by  $33.5\%$ ,  $12.9\%$  and  $20.2\%$  in WoW and CMU-DoG as seen in Table 2). Full results of all models and datasets are in Figure 6, 7 and 8 in  $\S K$ .
Hallucination in neural language generation has recently attracted the attention of several researchers in many areas including neural machine translation (NMT) (Raunak et al., 2021; Wang and Sennrich, 2020) and summarization (Durmus et al., 2020; Kang and Hashimoto, 2020). Hallucinations in knowledge-grounded neural dialogue generation is instead a nascent research problem (Mielke et al., 2020; Shuster et al., 2021; Dziri et al., 2021a; Rashkin et al., 2021b). Most existing works focus on avoiding hallucinations in generated outputs by introducing more robust training approaches. Dziri et al. (2021a) propose a model that uses facts supplied by a knowledge graph to reduce entity-based hallucinations in generated responses. Rashkin et al. (2021b) add control tokens at training time to control generation towards more objective sentences and faithful sentences. Closest to our work are Dziri et al. (2021b) and Rashkin et al. (2021a) who introduce frameworks for quantifying attribution in dialogue systems, whereas we conduct a much finer-grained manual analysis on multiple benchmarks and models.
Our investigations demonstrate empirically that hallucination is a prevalent issue in both dialog benchmarks and models. Our analysis on three widely used benchmarks reveals that they are rife with hallucinations, and the most common strategies people use are disclosure and edification. Moreover, we show that conversational models trained on these benchmarks not only hallucinate but also amplify hallucinations, even the models that were designed to alleviate this issue. This calls for a clean high-quality data release and careful design of trustworthy conversational systems. Before then, we strongly advocate practitioners to look at samples of any dataset—in order to uncover actionable insights—prior to their use or public release.
We are grateful to the anonymous reviewers for helpful comments. This research is supported by the Mila-IBM grant and the Alberta Machine Intelligence Institute Fellow Program. We also acknowledge the support of the NSERC Discovery grant and the Facebook CIFAR AI Chair program.
Impact Statement & Ethics
Annotation Risks The benchmarks we audit were collected through AMT and thus may contain some disturbing examples including racist or even expletive phrases. Annotators were also asked to judge the outputs of several state-of-the-art conversational systems which may be in turn toxic and insensitive. We acknowledge the psychological distress that this may present to workers (Arditte et al., 2016). Therefore, we alert workers by adding the following warning in italic text in each HIT: If this HIT causes you emotional distress or elicit feelings of trauma, please feel free to skip it.
Deployment Risks Our analytical study reveals that a large portion of standard knowledge-grounded dialogue benchmarks is hallucinated, leading us to reflect on the potential harm of low-quality data releases for conversational models. In recent years, the conversational AI market has seen a proliferation of a variety of applications—which are powered by large pre-trained LMs—that span across a broad range of domains, such as customer support, education, e-commerce, health, entertainment, etc (Vakulenko et al., 2021). Ensuring that these systems are trustworthy is key to deploy systems safely at a large scale in real-world application, especially in high-stake domains (Sambasivan et al., 2021). However, even if we come up with a model that is robust enough against hallucination, it will be ultimately bounded by the data quality. We argue that fixing the models or the data to enforce faithfulness is a highly non-trivial task without an in-depth understanding of the various sources of hallucination. Our work thus represents the first effort to gain such an understanding and to inform the community about the unreliability of the existing benchmarks and models. As a result, we believe it is important to raise these insights to the broader community.
Kimberly A Arditte, Demet Cek, Ashley M Shaw, and Kiara R Timpano. 2016. The importance of assessing clinical phenomena in mechanical turk research. Psychological assessment, 28(6):684.
Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610-623.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Harry Bunt, Volha Petukhova, Emer Gilmartin, Catherine Pelachaud, Alex Fang, Simon Keizer, and Laurent Prévot. 2020. The ISO standard for dialogue act annotation, second edition. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 549-558, Marseille, France. European Language Resources Association.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2018. Wizard of wikipedia: Knowledge-powered conversational agents. In International Conference on Learning Representations.
Esin Durmus, He He, and Mona Diab. 2020. FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5055-5070, Online. Association for Computational Linguistics.
Nouha Dziri, Andrea Madotto, Osmar Zaïane, and Avishek Joey Bose. 2021a. Neural path hunter: Reducing hallucination in dialogue systems via path grounding. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2197-2214, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Nouha Dziri, Hannah Rashkin, Tal Linzen, and David Reitter. 2021b. Evaluating groundedness in dialogue systems: The begin benchmark. arXiv preprint arXiv:2105.00071.
Karthik Gopalakrishnan, Behnam Hedayatnia, Qinlang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tur. 2019. Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversations. In Proc. Interspeech 2019, pages 1891-1895.
Paul Grice. 1989. Studies in the Way of Words. Harvard University Press.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. In International Conference on Learning Representations.