SlowGuess/NAACL_2022_OCR huggingface dataset:
Daniel Kang and Tatsunori B. Hashimoto. 2020. Improved natural language generation via loss truncation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 718-731, Online. Association for Computational Linguistics.
Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In *ICLR (Poster)*.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Sabrina J Mielke, Arthur Szlam, Y-Lan Boureau, and Emily Dinan. 2020. Linguistic calibration through metacognition: aligning dialogue agent responses with expected correctness. arXiv preprint arXiv:2012.14983.
Seungwhan Moon, Pararth Shah, Anuj Kumar, and Rajen Subba. 2019. OpenDialKG: Explainable conversational reasoning with attention-based walks over knowledge graphs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 845-854, Florence, Italy. Association for Computational Linguistics.
Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. StereoSet: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 5356-5371, Online. Association for Computational Linguistics.
Shrimai Prabhumoye, Kazuma Hashimoto, Yingbo Zhou, Alan W Black, and Ruslan Salakhutdinov. 2021. Focused attention improves document-grounded generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4274-4287, Online. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the
limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence level training with recurrent neural networks. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. 2021a. Measuring attribution in natural language generation models. arXiv preprint arXiv:2112.12870.
Hannah Rashkin, David Reitter, Gaurav Singh Tomar, and Dipanjan Das. 2021b. Increasing faithfulness in knowledge-grounded dialogue with controllable features. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 704-718, Online. Association for Computational Linguistics.
Vikas Raunak, Arul Menezes, and Marcin Junczys-Dowmunt. 2021. The curious case of hallucinations in neural machine translation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1172-1183, Online. Association for Computational Linguistics.
Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M Aroyo. 2021. "everyone wants to do the model work, not the data work": Data cascades in high-stakes ai. In proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1-15.
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. In *Findings of the Association for Computational Linguistics: EMNLP* 2021, pages 3784-3803, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958.
William B Stiles. 1992. Describing talk: A taxonomy of verbal response modes. Sage Publications.
Svitlana Vakulenko, Evangelos Kanoulas, and Maarten de Rijke. 2021. A large-scale analysis of mixed initiative in information-seeking dialogues for conversational search. arXiv preprint arXiv:2104.07096.
Chaojun Wang and Rico Senrich. 2020. On exposure bias, hallucination and domain shift in neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3544-3552, Online. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Thomas Wolf, Victor Sanh, Julien Chaumont, and Clement Delangue. 2019. Transfertransfo: A transfer learning approach for neural network based conversational agents. arXiv preprint arXiv:1901.08149.
Kangyan Zhou, Shrimai Prabhumoye, and Alan W Black. 2018. A dataset for document grounded conversations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 708-713, Brussels, Belgium. Association for Computational Linguistics.
We conduct our analysis on the following datasets:
Wizard of Wikipedia: composed of dialogues between a "wizard" and an "apprentice", where the goal of the wizard is to communicate information about a particular topic and the apprentice is tasked to seek information about that topic. At each turn, the wizard is presented with a knowledge snippet from Wikipedia and asked to form an utterance. We filter data points in which the wizard did not explicitly select a passage as knowledge for the response. In total, the dataset is comprised of 82722 grounded-responses in train, 8800 valid and 8690 test.
CMU-DoG: All conversations focus only on the movie domain. Each response is grounded on a section from Wikipedia. Workers are asked to either persuade the other speakers to watch the movie using information from the Wikipedia section or to discuss the content of the document with them. In total, there are 78136 grounded responses in train, 13800 in valid and 13796 in test.
TopicalChat: Contrary to CMU-DOG, TOPI-CALCHAT conversations are about a variety of topics. Workers are provided relevant facts from Reddit, Wikipedia and news articles. The collection process corresponds to two scenarios: symmetric and asymmetric. In the symmetric scenario, workers have access to the same source knowledge and in the asymmetric scenario, they have access to different sources. In total, the dataset has 292215 grounded responses in train, 23601 in valid and 23623 in test.
GPT2: This model was implemented using the Pytorch Huggingface Transformers library (Wolf et al., 2020) and the Pytorch-lightning library. To train the models, we use the Adam optimizer (Kingma and Ba, 2015) with Dropout (Srivastava et al., 2014) on a batch size of 32 with a learning rate of  $6.25 \times 10^{-5}$  that is linearly decayed. The maximum dialogue history length is set to 3 utterances. The model early-stops at epoch {7, 8, 8} respectively for WOW, CMU-DoG and TOPICALCHAT. The average runtime is {1.5, 3, 3}
hours for WOW, CMU-DOG and TOPICALCHAT respectively.
DoHA: We use the pre-trained model on CMU-DoG that is publicly available<sup>3</sup>. However, since no models trained on WOW and TOPICALCHAT have been released, we follow closely the training procedure described in Prabhumoye et al. (2021) and we train two models. The average runtime of these models is  $\{5, 10\}$  hours for WOW and TOPICALCHAT respectively.
CTRL: We implement the model ourselves since the code and the model were not released by the authors. We follow training details in Rashkin et al. (2021b) and implement this model using the Pytorch Huggingface Transformers library and the Pytorch-lightning library. Additionally, we had multiple discussions with the authors to make sure that our implementation is accurate.
We save the best model based on the validation set, for all datasets. Training for all models is done on an Nvidia V100 GPU 32GB and for inference, we use nucleus sampling with  $p = 0.6$ .
C Definition of VRM
Table 3 contains VRM definitions with examples.
D Expert Annotation
The two experts were students with linguistics background, fluent in English, and were trained for the task by exchanging rigorous discussions with the authors. As part of this stage, they were required to write justifications for 50 samples articulating the reasoning for the provided ratings. The collected justifications were helpful in understanding the reasoning used to reach their ratings and in laying the groundwork for designing the second round of annotations.
E Inter-annotator Agreement on Gold Responses
Table 4 contains the Fleiss kappa scores for CMU-DOG and TOPICALCHAT.
F AMT Human Annotation
Task Design To streamline the process for raters we break down the task into hierarchical (yes/no) questions. We summarize this procedure below, and provide the exact questions in  $\S G$ . First, we ask
$^{2}$ https://github.com/PyTorchLightning/pytorch-lightning
3https://bit.ly/3bBup2M
VRM Type	Description	Example
Disclosure	Reveal the speaker's subjective opinions, personal experience, thoughts, feelings, wishes, and intentions.	“I think science fiction is an amazing genre. Future science, technology they're all interesting.”
Edification	Concerns information that is, in principle, objective.	“Recycling includes items like metal and plastic.”
Advisement	Corresponds to guiding the behaviour of the addressee through: commands, requests, suggestions, advice, permission, prohibition.	“You should be patient and persistent to succeed.”
Confirmation	Compares the speaker's experience with the other's by expressing shared ideas/memories/beliefs, or by agreement/disagreement	“I agree that love encompasses a variety of different emotional and mental states.”
Question	Concerns requesting information or guidance.	“What is your favorite song?”
Acknowledge	Expresses no content, it conveys only receipt of communication from the other's speaker.	“Hmm. OK,...”, “Yeah, ...”, “Hello, ...”
We will present you with short Evidence (an excerpt from a Wikipedia page) and a dialog that is intended to be about that Evidence. The dialogue will be between two speakers, a wizard and an apprentice.
Questions to determine whether an edit is needed for the Bot
Does the Wizard's response contain other information that is NOT supported by the evidence? (E.g., facts, opinions, feelings) PS: Even if the response is about the same topic of the Evidence, it might use extra information that's not supported by the Evidence.
Warning: If this HIT causes you emotional distress or elicit feelings of trauma, please feel free to skip it.
I've been a vegetarian since 1983. I don't miss meat at all.
Other motivations for vegetarianism are health-related, political, environmental, cultural, aesthetic, economic, or personal preference.
That's great! Did you become a vegetarian for health related, environmental, political or personal reasons?
Figure 4: AMT Annotation interfaces for determining BEGIN and VRM categories (1).
Table 3: The definitions of the VRM types with examples.
BEGIN	VRM
CMU-DOG	0.85	0.78
TOPICALCHAT	0.83	0.72
Table 4: Fleiss Kappa Scores on 200 train Human-Human responses from the CMU-DOG and TOPICALCHAT benchmarks.
annotators to judge whether the response contain information that is not supported by the source. If yes, we ask them to indicate the type of the unsupported information (e.g., unsupported opinion, unsupported fact, etc). In a followup question, we ask them to indicate whether there are any supported information besides the hallucinated content. If the response was not hallucinated, we present them with two follow-up questions about whether the response is entailing the source or generic. Finally, if the response entails the source, we ask whether it is coherent with the history.
AMT Data Quality To access the initial staging round in AMT, workers have to pass a qualification test by answering correctly 14 questions about BEGIN and VRM. Moreover, they had to be situated in the United States and Canada. Before being granted access to the main annotation task, workers would have access only to a small pilot round (batch size  $\sim$  50 HITs). In this round, we carefully inspect each of the workers annotations for adherence to the instructions, and provide feedback via email to those who committed errors.
At the end of this round, we revoke access for workers who provide poor quality annotations. Next, we launch the main annotation stage which is larger (batch size  $\sim 400$  HITs). We perform daily manual inspection and we send detailed feedback to workers who commit persistent error patterns. We reject poor quality work in this stage and repeated rejections lead to blocking the workers from the task indefinitely. In total, we ended up with 4 workers annotating the 4k responses. The workers
Figure 5: AMT Annotation interfaces for determining BEGIN and VRM categories (2).
were informed that their annotations would be used for research purposes and their workers ID would be anonymous when we release the data.
G AMT Human Instructions
AMT Human annotation interfaces are depicted in Figure 4 and Figure 5. We pay workers an hourly wage around 18-20 USD which is above the minimum wage rate. Workers were asked the following questions:
1. Does the Wizard's response contain other information that is NOT supported by the evidence? (E.g., facts, opinions, feelings)?
(a) If the response is hallucinated, what is the type of the unsupported information? (expressing a personal experience, expressing an opinion, expressing feelings, expressing unsupported facts, giving advice, acknowledging with information from the human)
(b) Besides unsupported information, does the Wizard's response contain thoughts/opinions/feelings/facts that are supported by the Evidence?
2. If the response is not hallucinated, is it faithful to the source or generic? (Faithful, Generic)
3. If the response is faithful, is it cooperative with the Human's response?
The main goal of this work is to present a data quality audit by gaining an in-depth understanding of the various types of hallucination in both
gold and machine-generated responses. We do not investigate the root causes of hallucination in the models. Also, we limit our analysis to only English Benchmarks. Future studies can extend our work to explore the main causes of hallucination in the models and study the problem of hallucination in multilingual datasets.
I Hallucination in CMU-DoG and TopicalChat
Figure 3 shows the hallucination breakdown in CMU-DOG and TOPICALCHAT benchmarks.
J Hallucinated Human-Human Responses
Table 7 contains hallucinated gold responses from WOW, CMU-DOG and TOPICALCHAT.
K Breakdown of BEGIN and VRM in Machine-generated Responses
Figure 6, 7 and 8 display the distribution of BEGIN and VRM in GPT2, DoHA and CTRL trained on the three benchmark.
L Machine-generated Responses
Table 6 contains a sample of generated responses from GPT2, DoHA and CTRL on the WOW and CMU-DoG.
(a) GPT2 responses
(b) DoHA responses
(c) CTRL responses
Figure 6: Breakdown of BEGIN classes and VRM speech acts on WoW machine-generated responses.
(a) GPT2 responses
Figure 7: Breakdown of BEGIN classes and VRM speech acts on CMU-DoG machine-generated responses.
(c) CTRL responses
Figure 8: Breakdown of BEGIN classes and VRM speech acts on Topical machine-generated responses.
BEGIN	VRM	Example
Entailment	Disclosure	Knowledge: A dragon is a legendary creature, typically scaled or fire-spewing and with serpentine, reptilian or avian traits, that features in the myths of many cultures around world.
History: Dragons are so fascinating, I wonder where they originated from.
Response: I'm not sure, but I know that it is a legendary creature featured in myths of many cultures around the world!
Hallucination	Edification Disclosure	Knowledge: The central premise for these stories oftentimes involves changing history, either intentionally or by accident, and the ways by which altering the past changes the future and creates an altered present or future for the time traveler when they return home.
History: One of my favorite forms of science fiction is anything related to time travel! I find it fascinating.
Response: It's not quite sci-fi, but my favorite version of time travel is in Harry Potter and the Prisoner of Azkaban. Breaks zero logical rules.
Partial hallucination	Confirmation	Knowledge: Love encompasses a variety of different emotional and mental states, typically strongly and positively experienced, ranging from the deepest interpersonal affection to the simplest pleasure.
History: Romance can be small acts, like making breakfast for your significant other. But it's portrayed as grand gestures which are unattainable.
Response: I agree it has been portrayed as impossible actions. Love consists of a variety of emotional and mental states.
Generic	Question	Knowledge: Hard rock is a loosely defined subgenre of rock music that began in the mid-1960s, with the garage, psychedelic and blues rock movements.
History: yea i have heard of the American hard rock band Van Halen.
Response: Are you a fan of hard rock?.
Uncooperative	Edification	Knowledge: Marketing is used to create, keep and satisfy the customer.
History: Well, I wish I could. I think my training was a little off. I can't find a job and thinking about going back to school to be a counselor. What do you do?
Response: Marketing is used to create a customer and then retain the customer and keep them happy.
Table 5: Examples from Wizard of Wikipedia (Dinan et al., 2018) showing the BEGIN breakdown and different VRM linguistic phenomena for each response.
Data	Knowledge & History	Responses
WoW	Knowledge: Cinematography (also called "Direction of Photography") is the science or art of motion-picture photography by recording light or other electromagnetic radiation, either electronically by means of an image sensor, or chemically by means of a light-sensitive material such as film stock.History: Hi buddy, What you think about cinematography?	GPT2: Cinematography is another title I'm currently in school for. I've been doing the art of photography for over 15 years now. (Hallucination)DoHA: I do enjoy the science and art of motion-picture photography. (Partial Hallucination)CTRL: You're right. Cinematography is an art of motion -picture photography by recording light or other electromagnetic radiation, either electronically by means of an image sensor, or chemically by means of a light-sensitive material such as film stock. What about you? (Entailment)
CMU-DoG	Knowledge: Mean Girls is a 2004 American teen comedy film directed by Mark Waters and written by Tina Fey. The film is partially based on Rosalind Wiseman's 2002 non-fiction self-help book Queen Bees and Wannabes, which describes female high school social cliques and the damaging effects they can have on girls. Genre: Comedy. MovieName: Mean Girls rating: Rotten Tomatoes: 84% and average: 6.9/10.History: I think Rachel McAdams had an even better role as Regina George however!	GPT2: Mean Girls was a classic. (Hallucination)DoHA: Yes I agree! she was great and she was better than Mark Waters! (Hallucination)CTRL: I see. Mean Girls is a comedy movie that was directed by Mark Waters in 2004. (Entailment)
Table 6: Generated responses from different models based on Wizard of Wikipedia (Dinan et al., 2018) and CMU-DoG (Zhou et al., 2018) test samples.
Data	Knowledge & History	Hallucinated Gold Response
WoW	Knowledge: Science fiction often explores the potential consequences of scientific and other innovations, and has been called a ‘literature of ideas’.
History: And that’s difficult to do when dealing with time travel. I actually haven’t seen the latest Harry Potter movies. Guess it’s time to check them out!	If you really want a look at the potential negative consequences of scientific innovation, what you should check out is the TV show Fringe. Incredibly well written.
CMU-DoG	Knowledge: Movie: The Social Network. In October 2003, 19-year-old Harvard University student Mark Zuckerberg is dumped by his girlfriend Erica Albright. Returning to his dorm, Zuckerberg writes an insulting entry about Albright on his LiveJournal blog and then creates a campus website called Facemash by hacking into college databases to steal photos of female students, then allowing site visitors to rate their attractiveness. After traffic to the site crashes parts of Harvard’s computer network, Zuckerberg is given six months of academic probation. However, Facemash’s popularity attracts the attention of Harvard upperclassmen and twins Cameron and Tyler Winklevoss and their business partner Divya Narendra. The trio invites Zuckerberg to work on Harvard Connection, a social network featuring the exclusive nature of Harvard students and aimed at dating.
History: The movie is The Social Network. I personally do not like Facebook as a company.	The movie portrays the founding of social networking website Facebook and the resulting lawsuits. It even has Justin Timberlake in it, I don’t think I’ve ever seen him act.
TopicalChat	Knowledge: Wikipedia: first paragraph in https://en.wikipedia.org/wiki/GoogleReddit facts: A single Google search requires more computing power than it took to send Neil Armstrong and eleven other astronauts to the moon. Google Maps calculates traffic by tracking how fast Android devices are moving on the road instead of hiring people to mow the lawns around their headquarters. Google uses hundreds of live goats. On 16th August 2013, Google went down for about five minutes, and took 40% of web traffic with it. When there is a disputed border, Google maps tailors its maps to the claims of each country where the Internet browser is located.
History: Google provides online related services and products, which includes online ads, search engine and cloud computing.	Yeah, their services are good. I’m just not a fan of intrusive they can be on our personal lives.
Table 7: Hallucinated responses from different benchmarks: Wizard of Wikipedia (Dinan et al., 2018), CMU-DoG (Zhou et al., 2018) and TOPICALCHAT (Gopalakrishnan et al., 2019). Text highlighted in red indicates hallucinated content.