Venue: NAACL 2022

# On the Origin of hallucinations in Conversational Models:

Is it the Datasets or the Models?

Nouha Dziri\({}^{\dagger}\)  Sivan Milton\({}^{1}\)   Mo Yu\({}^{\lx@sectionsign}\)   Osmar Zaiane\({}^{\dagger}\)   Siva Reddy\({}^{\diamond}\)\({}^{\ddagger}\)

\({}^{\dagger}\)University of Alberta   \({}^{\Diamond}\)Mila - Quebec AI Institute

\({}^{\ddagger}\)McGill University   \({}^{\lx@paragraphsign}\)IBM Research

dziri@cs.ualberta.ca

###### Abstract

Knowledge-grounded conversational models are known to suffer from producing factually invalid statements, a phenomenon commonly called hallucination. In this work, we investigate the underlying causes of this phenomenon: is hallucination due to the training data, or to the models? We conduct a comprehensive human study on both existing knowledge-grounded conversational benchmarks and several state-of-the-art models. Our study reveals that the standard benchmarks consist of >60\(\%\) hallucinated responses, leading to models that not only hallucinate but even amplify hallucinations. Our findings raise important questions on the quality of existing datasets and models trained using them. We make our annotations publicly available for future research.1

Footnote 1: [https://github.com/McGill-NLP/FaithDial](https://github.com/McGill-NLP/FaithDial)

+
Footnote †: July 10-15, 2022 ©2022 Association for Computational Linguistics

+
Footnote †: July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

Knowledge-grounded conversational models, powered by large pre-trained language models Radford et al. (2019); Brown et al. (2020); Raffel et al. (2020), are well-known to generate factually incorrect statements, a phenomenon commonly called _hallucination_Dziri et al. (2021); Rashkin et al. (2021). A large commonality in the majority of prior work seeks to address hallucination by ameliorating the model Shuster et al. (2021); Mielke et al. (2020); Dziri et al. (2021); Rashkin et al. (2021), but no attempt has been made so far to audit the conversational benchmarks to the best of our knowledge.

On one hand, knowledge-grounded conversational benchmarks may contain hallucinations due to error-prone collection protocols, or due to a design framework that encourages informativeness over faithfulness. Existing dialogue systems are typically trained on corpora crowd-sourced through online platforms Dinan et al. (2018); Gopalakrishnan et al. (2019); Moon et al. (2019). With loose incentive to come up with faithfully-grounded utterances on the provided knowledge, crowdworkers may ignore knowledge-snippets altogether, use their personal knowledge or sometimes assume a fictional persona, resulting in conversations that are rife with subjective content and unverified factual knowledge. Figure 1 shows a hallucinated conversation from the WoW dataset Dinan et al. (2018),

On the other hand, neural conversational models are not necessarily designed to generate faithful outputs, but to mimic the distributional properties of the data. This kind of optimization will likely push the models to replicate and even amplify the hallucination behaviour at test time Bender et al. (2021). The presence of even few hallucinated responses may skew the data distribution in a way that curbs the model's ability to generate faithful responses Kang and Hashimoto (2020).

In this work, drawing insights from the linguistic coding system for discourse phenomena Stiles (1992) and evaluation frameworks such as BEGIN Dziri et al. (2021) and AIS Rashkin et al. (2021), we annotate responses from the three

Figure 1: An example of a hallucinated conversation from the Wizard of Wikipedia dataset Dinan et al. (2018). The wizard (yellow) is hallucinating information that cannot be inferred from the knowledge-snippet: hallucinated subjective content (red) and hallucinated objective content (blue).



widely-used knowledge-grounded conversational benchmarks: Wizard of Wikipedia (Dinan et al., 2018), CMU-DoG Zhou et al. (2018) and TopicalChatGolapalakrishnan et al. (2019).

Our analysis reveals surprisingly that more than 60% of the responses are hallucinated in the three datasets, with major hallucination modes that manifest principally through the expression of subjective information (e.g., thoughts, beliefs, feelings, intentions, personal experiences) and the expression of unsupported objective factual information. Further, to understand if neural conversational models make this hallucination more severe, we annotate responses generated by several state-of-the-art models, including ones that are designed to alleviate hallucinations. We find that the generated responses consist of an even larger portion of hallucinations, in comparison with the training data. Our findings question the quality of current conversational datasets, their appropriateness to train knowledge-grounded conversational systems, and the robustness of existing models.

## 2 Hallucinations in Benchmarks

We conduct a human study on three English crowdsourced knowledge-grounded conversational benchmarks: Wizard of Wikipedia (WoW), CMU-DoG and TopicalChat. These datasets consist of dialogues between two speakers, where the goal is to communicate information about particular topics while speakers are presented with a knowledge snippet relevant to the current turn. More details about these datasets are provided in SSA.

Response Classification TaxonomyFollowing the definitions of the BEGIN taxonomy Dziri et al. (2021b) and the AIS framework Rashkin et al. (2021a) of evaluating response attribution, we annotate each response based on whether it can be inferred exclusively from the knowledge-snippet as follows: **Entailment**: a response is fully supported by the knowledge, i.e., any information it contains must be attributed to the knowledge. **Hallucination**: a response's factual correctness cannot be fully verified from the knowledge-snippet (even if it is true in the real world). More specifically, personal opinions, experiences, feelings, internal assessments of reality that cannot be attributed to the information present in the source document, are considered hallucinations. **Partial Hallucination**: part of the response is hallucinated while the rest is entailed by the source knowledge. **Generic**: a response that is vague and does not convey any factual information such as "_Sounds good_" or "_I'm not sure about that_". **Uncooperative**: an entailed response that does not follow the principles of conversational cooperation according to Gricean maxims Grice (1989). The response may be purposefully misleading, or showing a general unwillingness to cooperate with the interlocutor, resulting in an incoherent communication.

To understand the linguistic nature of hallucinations, we further annotate responses based on a linguistic coding system for discourse phenomena, dubbed Verbal Response Modes (VRM; Stiles 1992). Concretely, we label a turn with the following speech acts: **Disclosure**, **Edification**, **Advisement**, **Confirmation**, **Question** and **Acknowledgement (Ack.)**. Table 1 displays the definition for each VRM type. We opted for the VRM taxonomy as it offers a simple way of codifying responses into categories that are sufficient for our analysis whereas one can also opt for a more demanding annotation scheme Bunt et al. (2020).

### Human Evaluation Study

We follow a two-stage annotation protocol where we first ask two linguists to judge the attribution of 200 randomly sampled train responses with respect to the source knowledge. Details about experts can be found in SSD. For inter-annotator agreement, we measure Fleiss' Kappa scores on both BEGIN and VRM. WoW achieved 0.89 on BEGIN and 0.78 on VRM, indicating substantial agreement. Annotations on CMU-DoG and TopicalChat achieved nearly similar agreement (See SSE). The high agreement scores align with the findings in AIS on WoW Rashkin et al. (2021a).

The second round corresponds to a large-scale annotation of 4K randomly sampled train responses using non-expert annotators from AMT. This round is crucial to ensure that the obtained results from the experts are reliable enough to draw conclusions about the quality of the data. As human annotation is expensive, we perform the non-expert annotations only on the WoW benchmark while restricting ourselves to expert annotations on CMU-DoG and TopicalChat data. We choose WoW over the other two datasets as the source knowledge is more amenable to faster annotation (TopicalChat: 300 words > CMU-DoG: 215 words > WoW: 27 words). Details about our AMT task design and how we ensure data quality can be 

found in SSF. In total, we selected 4 trusted workers to annotate the 4k responses. To compute the inter-annotator agreement, we assign three workers per response in a secondary task, and ask each of them to judge 500 responses. Reported Fleiss' Kappa agreements were 0.75 for BEGIN and 0.61 for VRM. Although substantial, the agreement is lower than the experts' one and this is expected as they have stronger linguistic background. We seek to answer the following questions:

(Q1) How much hallucination exists in the benchmarks?Figure 2 shows the breakdown of each BEGIN category in WoW and compares expert annotations versus AMT workers. Surprisingly, WoW is fraught with hallucinations. Expert annotations on 200 responses show that hallucinated responses are largely mixed with faithful content (\(42.3\%\) v.s. \(19.7\%\) fully hallucinated responses), which amounts to \(62\%\) hallucinations in total. These results generalize even on larger data; we can see that the portion of hallucinated responses increased to \(74.4\%\) when evaluated on 4K samples. Our analysis shows similar trends on the CMU-DoG and TopicalChat benchmarks (Figure 3). CMU-DoG contains \(61.4\%\) responses that are purely hallucinated against only \(16.2\%\) responses that are fully entailing the source knowledge and TopicalChat has similar results (\(63.9\%\) hallucination v.s. \(22.9\%\) entailment). Exemplars of hallucinated responses are depicted in SSJ. These findings raise the question on the quality of dialogue datasets.

(Q2) What are the hallucination strategies used in human-human data?Figure 2 and Figure 3 show the VRM breakdown for each BEGIN category in the three benchmarks. We make the following observations: The majority of hallucinations belong to _disclosure_ (i.e., subjective information) in all benchmarks (50.9%, 56.2% and 61.5% in WoW, CMU-DoG and TopicalChat respectively). Although the strategy of sharing subjective information such as thoughts, opinions and feelings is natural in conversations, it often comes at a cost of ignoring the knowledge snippet in these datasets. Moreover, _edification_ is also a common phenomenon in hallucinated responses, suggesting that humans not only discuss subjective informa

\begin{table}
\begin{tabular}{p{85.4pt} p{284.5pt}} \hline \hline
**VRM Type** & **Description** \\ \hline Disclosure & Reveal the speaker’s subjective opinions, personal experience, thoughts and feelings. \\ \hline Edification & Concerns information that is objective. \\ \hline Advisement & Corresponds to guiding the behaviour of the addressee through: commands, requests, suggestions, advice, permission, prohibition. \\ \hline Confirmation & Compares the speaker’s experience with the other’s by expressing shared ideas or by agreement, disagreement. \\ \hline Question & Concerns requesting information or guidance. \\ \hline Acknowledge & Expresses no content, it conveys only receipt of communication from the other’s speaker. \\ \hline \hline  & \\ \end{tabular}
\end{table}
Table 1: Definitions of the Verbal Response Modes (VRMs)

Figure 2: BEGIN and VRM breakdown of responses from WoW. The inner circle shows the breakdown of BEGIN classes and the outer shows the VRM types in each BEGIN type: Hallucination (red), Entailment (green), Partial Hallucination (yellow), Generic (pink), and Uncooperative (blue).

 

tion but also bring extra unsupported facts, either true or false. Other linguistic modes are also associated with hallucinations such as acknowledging unsupported claims or asking irrelevant questions. Conversely, entailment responses have high percentage of edification (\(>70\%\)) with information inferred from the knowledge snippet.

## 3 Hallucination Amplification in Models

Next, we investigate how much models amplify the hallucination phenomenon at inference time. We consider a range of representative models:

\(\bullet\)**GPT2**[19, 4] is an autoregressive model which takes as input a concatenation of the knowledge and the history.

\(\bullet\)**DoHA**[5] builds a BART-based conversational model Lewis et al. (2020) for knowledge-grounding, with a two-view attention mechanism to handle separately the encoded document and the history during generation.

\(\bullet\)**CTRL**[6] augments the GPT2 model with control tokens Keskar et al. (2019) that guide the generation towards less subjective and more entailed content.

We fine-tune each model on the benchmarks and use nucleus sampling Holtzman et al. (2019) with \(p=0.6\) for decoding (more implementation details are in SSB). As seen in Table 2, CTRL is the best model followed by DoHA based on the hallucation ratio. Table 6 in SSL shows a sample of generated responses. Similar to the analysis in SS2, we task the same two linguists to analyze model-generated responses for 200 randomly-selected test samples from each benchmark.

#### (Q3) Do state-of-the-art conversational models amplify hallucination?

Table 2 shows the degree of amplification across different models trained on the three benchmarks. Numbers report the percentage of each class in the data. Contrasting this with human gold responses, the models not only hallucinate but also amplify the percentage of hallucinations, except CTRL on WoW. For example, GPT2 amplifies full hallucination by \(19.2\%\) in WoW, \(15\%\) in CMU-DoG and \(15.1\%\) in TopicalChat. Conversely, it reduces entailment by \(17.4\%\), \(9.3\%\) and \(11.9\%\) respectively. This suggests that hallucination patterns are easier to learn than entailment. Among the three, CTRL hallucinates the least at the expense of producing a high number of uncooperative responses. Although

Figure 3: BEGIN and VRM breakdown of gold responses from CMU-DoG and TopicalChat. The inner circle shows the breakdown of BEGIN classes and the outer shows the VRM types in each BEGIN type: Hallucination (red), Entailment (green), Partial Hallucination (yellow), Generic (pink), and Uncooperative (blue).

 

these responses are entailing the knowledge, they are not coherent with the history. A closer inspection shows that most uncooperative responses are extractive, i.e., they copy big chunks of the evidence without adapting the content to the history or they just output an exact copy of the entire evidence. This is also reflected in high ROUGE scores between the response and the knowledge, corroborating the extractive nature of CTRL compared to the gold responses. This behavior is not surprising as CTRL was optimized to maximize the overlap with the knowledge. Overall, these results demonstrate that hallucination is not only a reflection of training data issues, but also a consequence of the weaknesses of models.

We hypothesize that there are multiple factors that can contribute to the models' deficiencies: First, the exposure bias (Ranzato et al., 2016) caused by teacher forcing can make hallucination worse as the model may over-rely on previously predicted words which in turn can aggravate error propagation. Second, maximum likelihood estimation can be fragile to noisy data points as it necessitates models to assign high probability mass to all test references, resulting in unstable behavior--a fact observed in machine summarization Kang and Hashimoto (2020). Moreover, we link this issue to the decoding strategies used at test time. We conjecture that models--when conditioned on factual knowledge--often assign the highest probability mass to the correct response and sampling based on other distributions (e.g. top-k or nucleus) may invite hallucination in the generation process. And lastly, we hypothesise that the behavior of these models is ultimately shaped by the bias learned from internet text during pre-training Nadeem et al. (2021). We leave investigating the role of each factors to hallucination amplification for future work.

**(Q4) What are the hallucination strategies used by models?** Surprisingly, different models use different strategies for hallucination. While DoHA and GPT2 predominantly rely on and amplify _disclosure_, CTRL relies on _edification_. This is because CTRL is trained explicitly to avoid pronouns (a crucial ingredient for disclosure) and to generate entailed responses. As a side-effect, it ends up amplifying uncooperative responses (by \(33.5\%\), \(12.9\%\) and \(20.2\%\) in WOW and CMU-DoG as seen in Table 2). Full results of all models and datasets are in Figure 6, 7 and 8 in SSAK.

## 4 Related Work

Hallucination in neural language generation has recently attracted the attention of several researchers in many areas including neural machine translation NMT Raunak et al. (2021); Wang and Sennrich (2020) and summarization Durmus et al. (2020); Kang and Hashimoto (2020). Hallucinations in knowledge-grounded neural dialogue generation is instead a nascent research problem Mielke et al. (2020); Shuster et al. (2021); Dziri et al. (2021); Rashkin et al. (2021). Most existing works focus on avoiding hallucinations in generated outputs by introducing more robust training approaches. Dziri et al. (2021) propose a model that uses facts supplied by a knowledge graph to reduce entity-based hallucinations in generated responses. rashkin2021bert add control tokens at training time to control generation towards more objective sentences and faithful sentences. Closest to our work are Dziri et al. (2021) and Rashkin et al. (2021) who introduce frameworks for quantifying attribution in dialogue systems, whereas we conduct a much finer-grained manual analysis on multiple benchmarks and models.

## 5 Conclusion

Our investigations demonstrate empirically that hallucination is a prevalent issue in both dialog benchmarks and models. Our analysis on three widely used benchmarks reveals that they are rife with hallucinations, and the most common strategies people use are _disclosure_ and _edification_. Moreover, we show that conversational models trained on these benchmarks not only hallucinate but also amplify hallucinations, even the models that were designed to alleviate this issue. This calls for a clean high-quality data release and careful design of trustworthy conversational systems. Before then, we strongly advocate practitioners to look at samples of any dataset--in order to uncover actionable insights--prior to their use or public release.

### Acknowledgements

We are grateful to the anonymous reviewers for helpful comments. This research is supported by the Mila-IBM grant and the Alberta Machine Intelligence Institute Fellow Program. We also acknowledge the support of the NSERC Discovery grant and the Facebook CIFAR AI Chair program 



### Impact Statement & Ethics

Annotation RisksThe benchmarks we audit were collected through AMT and thus may contain some disturbing examples including racist or even expletive phrases. Annotators were also asked to judge the outputs of several state-of-the-art conversational systems which may be in turn toxic and insensitive. We acknowledge the psychological distress that this may present to workers Arditte et al. (2016). Therefore, we alert workers by adding the following warning in italic text in each HIT: _If this HIT causes you emotional distress or elicit feelings of trauma, please feel free to skip it._

Deployment RisksOur analytical study reveals that a large portion of standard knowledge-grounded dialogue benchmarks is hallucinated, leading us to reflect on the potential harm of low-quality data releases for conversational models. In recent years, the conversational AI market has seen a proliferation of a variety of applications--which are powered by large pre-trained LMs--that span across a broad range of domains, such as customer support, education, e-commerce, health, entertainment, etc Vakulenko et al. (2021). Ensuring that these systems are trustworthy is key to deploy systems safely at a large scale in real-world application, especially in high-stake domains Sambasivan et al. (2021). However, even if we come up with a model that is robust enough against hallucination, it will be ultimately bounded by the data quality. We argue that fixing the models or the data to enforce faithfulness is a highly non-trivial task without an in-depth understanding of the various sources of hallucination. Our work thus represents the first effort to gain such an understanding and to inform the community about the unreliability of the existing benchmarks and models. As result, we believe it is important to raise these insights to the broader community.

## References

* A. Arditte, D. Cek, A. M. Shaw, and K. R. Timpano (2016)The importance of assessing clinical phenomena in mechanical turk research. Psychological assessment28 (6), pp. 684. Cited by: SS3.
* E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell (2021)On the dangers of stochastic parrots: can language models be too big?. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 610-623. Cited by: SS5.
* T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, H. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei (2020)Language models are few-shot learners. In Advances in Neural Information Processing Systems, Vol. 33, pp. 1877-1901. External Links: Link Cited by: SS1, SS2.
* H. Bunt, Y. Petukhova, E. Gilm Martin, C. Pelachaud, A. Fang, S. Keizer, and L. Prevot (2020)The ISO standard for dialogue act annotation, second edition. In Proceedings of the 12th Language Resources and Evaluation Conference, Marseille, France, pp. 549-558. External Links: Link Cited by: SS1, SS2.
* E. Dinan, S. Roller, K. Shuster, A. Fan, M. Auli, and J. Weston (2018)Wizard of wikipedia: knowledge-powered conversational agents. In International Conference on Learning Representations, Cited by: SS1, SS2.
* E. Durmus, H. He, and M. Diab (2020)FEQA: a question answering evaluation framework for faithfulness assessment in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 5055-5070. External Links: Link Cited by: SS1, SS2.
* N. Dairi, A. Madotto, O. Zaiane, and A. Jose Bose (2021)Neural path hunter: reducing hallucation in dialogue systems via path grounding. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, pp. 2197-2214. External Links: Link Cited by: SS1, SS2.
* N. Dairi, H. Rashkin, T. Linzen, and D. Reitter (2021)Evaluating groundedness in dialogue systems: The begin benchmark. arXiv preprint arXiv:2105.00071. Cited by: SS1, SS2.
* K. Gopalakrishnan, B. Hedayatnia, Q. Chen, A. Gottardi, S. Kwatra, A. Venkatesh, R. Gabriel, and D. Hakkani-Tur (2019)Topical-chat: towards knowledge-grounded open-domain conversations. In Proc. Interspeech 2019, pp. 1891-1895. Cited by: SS1, SS2.
* P. Grice (1989)Studies in the way of words. Harvard University Press. Cited by: SS1, SS2.
* A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi (2019)The curious case of neural text degeneration. In International Conference on Learning Representations,

Daniel Kang and Tatsunori B. Hashimoto (2020)Improved natural language generation via loss truncation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 718-731. External Links: Link Cited by: SS4.
* N. Shirish Keskar, B. McCann, L. R. Varshney, C. Xiong, and R. Socher (2019)Ctrl: a conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858. Cited by: SS1.
* Diederik P. Kingma and J. Ba (2015)Adam: a method for stochastic optimization. In ICLR (Poster), Cited by: SS4.
* M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer (2020)BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 7871-7880. External Links: Link Cited by: SS4.
* S. J. Mielke, A. Szlam, Y. Boureau, and E. Dinan (2020)Linguistic calibration through metacognition: aligning dialogue agent responses with expected correctness. arXiv preprint arXiv:2012.14983. Cited by: SS1.
* S. Moon, P. Shah, A. Kumar, and R. Subba (2019)OpenDialKG: explainable conversational reasoning with attention-based walks over knowledge graphs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 845-854. External Links: Link Cited by: SS1.
* M. Nadem, A. Bethke, and S. Reddy (2021)StereoSet: measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, pp. 5356-5371. External Links: Link, Document Cited by: SS1.
* S. Prabhumoye, K. Hashimoto, Y. Zhou, A. W. Black, and R. Salakhutdinov (2021)Focused attention improves document-grounded generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online, pp. 4274-4287. External Links: Link, Document Cited by: SS1.
* A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. (2019)Language models are unsupervised multitask learners. OpenAI blog1 (8), pp. 9. Cited by: SS1.
* C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu (2020)Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research21 (140), pp. 1-67. External Links: Link, Document Cited by: SS1.
* M. Kanzato, S. Chopra, M. Auli, and W. Zaremba (2016)Sequence level training with recurrent neural networks. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, H. Rashkin, V. Nikolaev, M. Lamm, M. Collins, D. Das, S. Petrov, F. S. Tomar, I. Turc, and D. Reitter (Eds.), Vol. , Berlin, Heidelberg, Germany, pp. 122-1287. External Links: ISBN 978-3-319-31 

Chaojun Wang and Rico Sennrich. 2020. On exposure bias, hallucination and domain shift in neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online. Association for Computational Linguistics, pp. 3544-3552. External Links: Link Cited by: SS1.
* T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jenrite, J. Phu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush (2020)Transformers: state-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Online, pp. 38-45. External Links: Link, Document Cited by: SS1.
* T. Wolf, V. Sanh, J. Chaumond, and C. Delangue (2019)Transfertransfo: a transfer learning approach for neural network based conversational agents. arXiv preprint arXiv:1901.08149. Cited by: SS1.
* K. Zhou, S. Prabhumoye, and A. W. Black (2018)A dataset for document grounded conversations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, pp. 708-713. External Links: Link, Document Cited by: SS1.

 

Datasets

We conduct our analysis on the following datasets:

Wizard of Wikipedia:composed of dialogues between a "wizard" and an "apprentice", where the goal of the wizard is to communicate information about a particular topic and the apprentice is tasked to seek information about that topic. At each turn, the wizard is presented with a knowledge snippet from Wikipedia and asked to form an utterance. We filter data points in which the wizard did not explicitly select a passage as knowledge for the response. In total, the dataset is comprised of 82722 grounded-responses in train, 8800 valid and 8690 test.

CMU-DoG:All conversations focus only on the movie domain. Each response is grounded on a section from Wikipedia. Workers are asked to either persuade the other speakers to watch the movie using information from the Wikipedia section or to discuss the content of the document with them. In total, there are 78136 grounded responses in train, 13800 in valid and 13796 in test.

TopicalChat:Contrary to CMU-DoG, TopicalChat conversations are about a variety of topics. Workers are provided relevant facts from Reddit, Wikipedia and news articles. The collection process corresponds to two scenarios: symmetric and asymmetric. In the symmetric scenario, workers have access to the same source knowledge and in the asymmetric scenario, they have access to different sources. In total, the dataset has 292215 grounded responses in train, 23601 in valid and 23623 in test.

## Appendix B Implementation Details

Gpt2:This model was implemented using the Pytorch Huggingface Transformers library Wolf et al. (2020) and the Pytorch-lightning library2. To train the models, we use the Adam optimizer Kingma and Ba (2015) with Dropout Srivastava et al. (2014) on a batch size of \(32\) with a learning rate of \(6.25\times 10^{-5}\) that is linearly decayed. The maximum dialogue history length is set to \(3\) utterances. The model early-stops at epoch {7, 8, 8} respectively for WoW, CMU-DoG and TopicalChat. The average runtime is {1.5, 3, 3} hours for WoW, CMU-DoG and TopicalChat respectively.

Footnote 2: [https://github.com/PyTorchLightning/pytorch-lightning](https://github.com/PyTorchLightning/pytorch-lightning)

DoHA:We use the pre-trained model on CMU-DoG that is publicly available3. However, since no models trained on WoW and TopicalChat have been released, we follow closely the training procedure described in Prabhumoye et al. (2021) and we train two models. The average runtime of these models is {5, 10} hours for WoW and TopicalChat respectively.

CTRL:We implement the model ourselves since the code and the model were not released by the authors. We follow training details in Rashkin et al. (2021b) and implement this model using the Pytorch Huggingface Transformers library and the Pytorch-lightning library. Additionally, we had multiple discussions with the authors to make sure that our implementation is accurate.

Footnote 3: [https://bit.ly/3bBup2M](https://bit.ly/3bBup2M)

We save the best model based on the validation set, for all datasets. Training for all models is done on an Nvidia V100 GPU 32GB and for inference, we use nucleus sampling with p=0.6.

## Appendix C Definition of VRM

Table 3 contains VRM definitions with examples.

## Appendix D Expert Annotation

The two experts were students with linguistics background, fluent in English, and were trained for the task by exchanging rigorous discussions with the authors. As part of this stage, they were required to write justifications for 50 samples articulating the reasoning for the provided ratings. The collected justifications were helpful in understanding the reasoning used to reach their ratings and in laying the groundwork for designing the second round of annotations.

## Appendix E Inter-annotator Agreement on Gold Responses

Table 4 contains the Fleiss kappa scores for CMU-DoG and TopicalChat.

## Appendix F AMT Human Annotation

Task DesignTo streamline the process for raters we break down the task into hierarchical (yes/no) questions. We summarize this procedure below, and provide the exact questions in SS 

annotators to judge whether the response contain information that is not supported by the source. If yes, we ask them to indicate the type of the unsupported information (e.g., unsupported opinion, unsupported fact, etc). In a followup question, we ask them to indicate whether there are any supported information besides the hallucinated content. If the response was not hallucinated, we present them with two follow-up questions about whether the response is entailing the source or generic. Finally, if the response entails the source, we ask whether it is coherent with the history.

AMT Data QualityTo access the initial staging round in AMT, workers have to pass a qualification test by answering correctly 14 questions about BEGIN and VRM. Moreover, they had to be situated in the United States and Canada. Before being granted access to the main annotation task, workers would have access only to a small pilot round (batch size \(\sim\) 50 HITs). In this round, we carefully inspect each of the workers annotations for adherence to the instructions, and provide feedback via email to those who committed errors.

At the end of this round, we revoke access for workers who provide poor quality annotations. Next, we launch the main annotation stage which is larger (batch size \(\sim\) 400 HITs). We perform daily manual inspection and we send detailed feedback to workers who commit persistent error patterns. We reject poor quality work in this stage and repeated rejections lead to blocking the workers from the task indefinitely. In total, we ended up with 4 workers annotating the 4k responses. The workers

Figure 4: AMT Annotation interfaces for determining BEGIN and VRM categories [1].

\begin{table}
\begin{tabular}{c c c} \hline  & BEGIN & VRM \\ \hline CMU–DoG & 0.85 & 0.78 \\ TopicalChat & 0.83 & 0.72 \\ \hline \end{tabular}
\end{table}
Table 4: Fleiss Kappa Scores on 200 train Human-Human responses from the CMU-DoG and TopicalChat benchmarks.

 

were informed that their annotations would be used for research purposes and their workers ID would be anonymous when we release the data.

## Appendix G AMT Human Instructions

AMT Human annotation interfaces are depicted in Figure 4 and Figure 5. We pay workers an hourly wage around 18-20 USD which is above the minimum wage rate. Workers were asked the following questions:

1. Does the Wizard's response contain other information that is NOT supported by the evidence? (E.g., facts, opinions, feelings)? 1. If the response is hallucinated, what is the type of the unsupported information? (expressing a personal experience, expressing an opinion, expressing feelings, expressing unsupported facts, giving advice, acknowledging with information from the human) 2. Besides unsupported information, does the Wizard's response contain thoughts/opinions/feelings/facts that are supported by the Evidence?
2. If the response is not hallucinated, is it faithful to the source or generic? Faithful, Generic)
3. If the response if faithful, is it cooperative with the Human's response?

## Appendix H Limitation

The main goal of this work is to present a data quality audit by gaining an in-depth understanding of the various types of hallucination in both gold and machine-generated responses. We do not investigate the root causes of hallucination in the models. Also, we limit our analysis to only English Benchmarks. Future studies can extend our work to explore the main causes of hallucination in the models and study the problem of hallucination in multilingual datasets.

## Appendix I Hallucination in CMU-DoG and TopicalChat

Figure 3 shows the hallucination breakdown in CMU-DoG and TopicalChat benchamrks.

## Appendix J Hallucinated Human-Human Responses

Table 7 contains hallucinated gold responses from WoW, CMU-DoG and TopicalChat.

## Appendix K Breakdown of BEGIN and VRM in Machine-generated Responses

Figure 6, 7 and 8 display the distribution of BEGIN and VRM in GPT2, DoHA and CTRL trained on the three benchmark.

## Appendix L Machine-generated Responses

Table 6 contains a sample of generated responses from GPT2, DoHA and CTRL on the WoW and CMU-DoG.

 

Figure 8: Breakdown of BEGIN classes and VRM speech acts on Topical machine-generated responses.

Figure 6: Breakdown of BEGIN classes and VRM speech acts on WoW machine-generated responses.

Figure 7: Breakdown of BEGIN classes and VRM speech acts on CMU-DoG machine-generated responses.

 



\begin{table}
\begin{tabular}{l l p{142.3pt} p{85.4pt}} \hline
**BEGIN** & **VRM** & **Example** \\ \hline Entailment & Disclosure & Knowledge: & A dragon is a legendary creature, typically scaled or fire-spewing and with serpentine, reptilian or avian traits, that features in the myths of many cultures around world. \\  & & History: & Dragons are so fascinating, I wonder where they originated from. I’m not sure, but I know that it is a legendary creature featured in myths of many cultures around the world! \\ \hline Hallucination & Edification Disclosure & Knowledge: & The central premise for these stories oftentimes involves changing history, either intentionally or by accident, and the ways by which altering the past changes the future and creates an altered present or future for the time traveler when they return home. \\  & & History: & One of my favorite forms of science fiction is anything related to time travel! I find it fascinating. It’s not quite sci-fi, but my favorite version of time travel is in Harry Potter and the Prisoner of Azkaban. Breaks zero logical rules. \\ \hline Partial hallucination & Confirmation & Knowledge: & Love encompasses a variety of different emotional and mental states, typically strongly and positively experienced, ranging from the deepest interpersonal affection to the simplest pleasure. \\  & & History: & Romance can be small acts, like making breakfast for your significant other. But it’s portrayed as grand gestures which are unattainable. \\  & & **Response**: & Igree it has been portrayed as impossible actions \\  & & & variety of emotional and mental states. \\ \hline Generic & Question & Knowledge: & Hard rock is a loosely defined subgenre of rock music that began in the mid-1960s, with the garage, psychedelic and blues rock movements. \\  & & History: & yea i have heard of the American hard rock band Van Halen. Are you a fan of hard rock?. \\ \hline Uncooperative & Edification & Knowledge: & Marketing is used to create, keep and satisfy the customer. Well, I wish I could. I think my training was a little off. I can’t find a job and thinking about going back to school to be a counselor. What do you do? \\  & & **Response**: & Marketing is used to create a customer and then retain the customer and keep them happy. \\ \hline \end{tabular}
\end{table}
Table 5: Examples from Wizard of Wikipedia (Dinan et al., 2018) showing the BEGIN breakdown and different VRM linguistic phenomena for each response.





\begin{table}
\begin{tabular}{p{142.3pt} p{85.4pt} p{76.9pt}}
**Data** & **Knowledge \& History** & **Responses** \\ \hline
**WoW** & **Knowledge**: Cinematography (also called ”Direction of Photography”) is the science or art of motion-picture photography by recording light or other electromagnetic radiation, either electronically by means of an image sensor, or chemically by means of a light-sensitive material such as film stock. & **GPT2**: Cinematography is another title I’m currently in school for. I’ve been doing the art of photography for over 15 years now. (Hallucination) & **DoHA**: I do enjoy the science and art of motion-picture photography. (Partial Hallucination) \\  & **CTRL**: You’re right. Cinematography is an art of motion –picture photography by recording light or other electromagnetic radiation, either electronically by means of an image sensor, or chemically by means of a light-sensitive material such as film stock. What about you ’(Entailment) \\ \hline
**CMU-DoG** & **Knowledge**: Mean Girls is a 2004 American teen comedy film directed by Mark Waters and written by Tina Fey. The film is partially based on Rosalind Wiseman’s 2002 non-fiction self-help book Queen Bees and Wannases, which describes female high school social cliques and the damaging effects they can have on girls. Genre: Comedy. MovieName: Mean Girls rating: Rotten Tomatoes: 84 % and average: 6.9/10. & **GPT2**: Mean Girls was a classic. (Hallucination) \\  & **Histor**: I think Rachel McAdams had an even better role as Regina George however! & **DaRA**: Yes I agree what was great and she was better than Mark Waters! (Hallucination) \\ \end{tabular}
\end{table}
Table 6: Generated responses from different models based on Wizard of Wikipedia [2] and CMU–DoG [2] test samples.





\begin{table}
\begin{tabular}{p{142.3pt} p{856.9pt} p{70.7pt}} \hline
**Data** & **Knowledge \& History** & **Hallucinated Gold Response** \\ \hline
**WoW** & **Knowledge**: Science fiction often explores the potential consequences of scientific and other innovations, and has been called a ’literature of ideas’. & If you really want a look at the potential negative consequences of scientific innovation, what you should check out is the TV show Fringe. Incredibly well written. \\  & **History**: And that’s difficult to do when dealing with time travel . I actually haven’t seen the latest Harry Potter movies. Guess it’s time to check them out! & The movie portrays the founding of social networking website Facebook and the resulting lawsuits. It even has Justin Timberlake in it, I don’t think I’ve ever seen him act. \\ \hline
**CMU-DoG** & **Knowledge**: Movie: The Social Network. In October 2003, 19-year-old Harvard University student Mark Zuckerberg is dumped by his girlfriend Erica Albright. Returning to his dorm, Zuckerberg writes an insulting entry about Albright on his LiveJournal blog and then creates a campus website called Facemash by hacking into college databases to steal photos of female students, then allowing site visitors to rate their attractiveness. After traffic to the site crashes parts of Harvard’s computer network, Zuckerberg is given six months of academic probation. However, Facemash’s popularity attracts the attention of Harvard upperclassmen and twins Cameron and Tyler Winklevoss and their business partner Divya Narendra. The trio invites Zuckerberg to work on Harvard Connection, a social network featuring the exclusive nature of Harvard students and aimed at dating. \\  & **History**: The movie is The Social Network. I personally do not like Facebook as a company. \\ \hline
**TopicalChat** & **Knowledge**: Wikipedia: first paragraph in [https://en.wikipedia.org/wiki/Google](https://en.wikipedia.org/wiki/Google) Reddit facts: A single Google search requires more computing power than it took to send Neil Armstrong and eleven other astronauts to the moon. Google Maps calculates traffic by tracking how fast Android devices are moving on the road instead of hiring people to move the lawns around their headquarters. Google uses hundreds of live goats. On 16th August 2013, Google went down for about five minutes, and took 40% of web traffic with it. When there is a disputed border, Google maps tailors its maps to the claims of each country where the Internet browser is located. \\
**History**: Google provides online related services and products, which includes online ads, search engine and cloud computing. \\ \hline \end{tabular}
\end{table}
Table 7: Hallucinated responses from different benchmarks: Wizard of Wikipedia (Dinan et al., 2018), CMU–DoG (Zhou et al., 2018) and TopICALHat (Gopalakrishnan et al., 2019). Text highlighted in red indicates hallucinated content.