SlowGuess/NAACL_2022_OCR huggingface dataset:
Results on Test Sets. Table 4 presents objective evaluation results obtained for various Teacher bots on three test sets: CNN-DM, Paper Abstracts and Wikipedia. We observe that our FULL Teacher bot is able to substantially outperform the baseline system DoHA on all datasets and across all objective metrics. It strikes a fine balance between delivering information-rich conversations and ensuring those conversations are coherent and attentive. Further, we find that optimizing a single reward, whether it be coverage or coherence, produces suboptimal results. For instance,  $+\mathrm{Cov}$  tends to produce longer
Figure 3: ROUGE gain and utterance length tend to decrease as the number of turns increases
utterances than other variants. It improves information coverage, but yields low coherence scores, leading to a performance even inferior to the baseline DoHA. Our findings suggest that it is important for the reinforcer  $L_{rl}^{X}$  to learn with both coverage or coherence rewards.
Trading off Coverage for Coherence. In Figure 4, we plot the learning curves of coverage and coherence scores when the reinforcer adopts a single reward  $(+\mathrm{COV}, +\mathrm{COH})$  or both (FULL). We use  $R_{cov}$  and  $R_{coh}$  to approximate coverage and coherence scores. These plots are generated using 50 validation instances from the Paper Abstracts dataset. We observe that with only the coverage reward  $(+\mathrm{COV})$ , Teacher bot tends to aggressively copy content from the passage, while disregarding the conversation history. This inevitably leads to incoherent conversations. Conversely,  $+\mathrm{COH}$  can improve on coherence, but falls short on delivering informative conversations. Finally, our FULL Teacher bot trades off coverage for substantially higher coherence, thus achieving a significant improvement over the baseline DOHA model.
Trading off Coverage for Coherence. We are curious to know the amount of information brought by each utterance produced by Teacher bot. To this end, we define information gain  $\mathcal{IG}(\cdot)$  as the
Figure 4: Coverage and coherence scores when applying different reward scores
Dataset	Model	Coverage	Coherence
\(QA_{CONF}\)	\(QA_{F1}\)	WoW-Coh	IC	DPR
Wikipedia	WoW	18.96	17.17	0.807	0.694	0.578
InferCov	20.41	19.61	0.647	0.676	0.561
CNN-DM	WoW	13.31	11.31	0.845	0.692	0.561
InferCov	14.95	14.07	0.719	0.728	0.550
Papers	WoW	8.18	3.95	0.806	0.547	0.576
InferCov	9.47	4.31	0.737	0.554	0.558
Table 5: Teacher bot's performance when using different coherence score models during fine-tuning
improvement of ROUGE scores brought by an utterance  $U$ :
$$
\mathcal {I G} (U) = \operatorname {R O U G E} (P, H + U) - \operatorname {R O U G E} (P, H),
$$
where  $P$  is the supporting passage, and  $H$  represents the conversation history. We consider three ROUGE variants, R-1, R-2 and R-L, respectively. Figure 3 illustrates the gain of information for each of the three turns. The average ROUGE gain is reported for each turn, using conversations produced for the Paper Abstracts dataset. We observe that there is a general tendency across turns that information gain is decreasing. This is in part because that at the beginning of a conversation, Teacher bot has no constraints regarding content selection, it could rephrase any content selected from the supporting passage to initiate a dialogue. In subsequent turns, Teacher bot has to exercise caution in response generation considering both the conversation history and overall coherence of the conversation. Consequently, we find that the average length of the utterances also decreases in subsequent turns.
A Comparison of Coherence Scoring Models. We compare two Teacher bots fine-tuned only with coherence reward from different coherence scoring models (i.e. WoW-coherence and InferConv). We demonstrate their objective results in Table 5. According to the results, Teacher bot fine-tuned with InferConv achieves slightly better coverage metrics. However, in terms of coherence metrics, Teacher bot fine-tuned with WoW-coherence model generally achieves better performance. Based on this
observation, WoW-coherence scoring model better measures coherence in conversations.
5 Subjective Results
We demonstrate the subjective evaluation results in Table 4 and have the following observations:
1. For question answering,  $+\mathrm{Cov}$  achieves the best performance on all three datasets. This again proves that the coverage reward helps make the conversation more informative;
2. For coherence scores,  $+\mathrm{COH}$  achieves the best performance on Wikipedia. However, on the other two datasets it was outperformed by FULL.
3. For readability scores, on CNN-DM and Paper Abstracts,  $+\mathrm{COH}$  achieves the lowest performance while  $+\mathrm{COV}$  achieves the highest.
4. For overall scores, FULL demonstrates the best performance. This suggests FULL delivers conversations that are more balanced in coverage, coherence and readability.
5. The Paper Abstracts corpus is the most challenging among all the test corpora, as our Teacher bots generally show worse performance in coherence, readability and overall scores. We found passages in Paper Abstracts contain volumes of professional vocabularies thus are more complicated for people to understand. In addition, it's also more difficult for Student bot to respond appropriately during self-play fine-tuning (See examples in appendix). As a result, transferring Teacher bots to this domain is more challenging.
A Case Study. We show a few Teacher bots' responses to users in Table 6. After analyzing the cases, we have the following observations:
1. The coverage reward encourages Teacher bots to directly copy content from the input passage,
Example A (Passage from Wikipedia)
Passage	The Concorde is a British-French turbojet-powered super-sonic passenger airliner ... It had a maximum speed over twice the speed of sound, at Mach 2.04, ... First flown in 1969, Concorde entered service in 1976 and operated for 27 years. ...
User	What is the plane famous for?
Bot Cov	First flown in 1969, Concorde entered service in 1976 and operated for 27 years.
Bot Coh	Concorde is famous for the speed of sound at Mach 2.04.
Bot Full	Concorde is famous for having a maximum speed over twice the speed of sound, at Mach 2.04.
Example B (Passage from CNN News)
Passage	The US Food and Drug Administration, ... Molnupiravir is for the treatment of mild-to-moderate coronavirus disease in adults with positive results of direct SARS-CoV-2 viral testing ... This is the second Covid-19 antiviral pill authorized for ill people to take at home, before they get sick enough to be hospitalized ...
User	Tell me more about the drug.
Bot Cov	This is the second Covid-19 antiviral pill authorized for ill people to take at home, before they get sick enough to be hospitalized.
Bot Coh	The drug is for people with mild to moderate coronavirus disease.
Bot Full	Molnupiravir is for the treatment of mild-to-moderate coronavirus disease in adults with positive results of direct SARS-CoV-2 viral testing.
Example C (Passage from Paper Abstracts)
Passage	... In this paper we describe a novel framework to guide a supervised, extractive summarization system with question-answering rewards ... Our results compare favorably with those reported by strong summarization baselines as evaluated by automatic metrics and human assessors ...
User	What did you propose?
Bot Cov	Our results compare favorably with those reported by strong summarization baselines as evaluated by automatic metrics and human assessors.
Bot Coh	A proposal to use the extractive summarizer to answer questions.
Bot Full	We propose a novel framework to guide a supervised, extractive summarization system with question-answering rewards.
Table 6: A few example responses from different Teacher bots. In example A,  $+\mathrm{COH}$  and FULL abstractively generates Concord is famous for ... to make the response more coherent to user's question. However, the underlined part in  $+\mathrm{COH}$ 's statement misrepresent the input passage and is inaccurate. In example B, all responses seem coherent because of the open-ended question.
while the coherence reward encourages abstractively generating new content: As shown in Example A,  $+\mathrm{Cov}$  directly extracts a part of the original passage as response, regardless of the user's question, while  $+\mathrm{COH}$  and FULL abstractively rewrite the response to make it more coherent.
2. Putting too much weight on coherence reward could make Teacher bot become so abstractive that it misrepresents the original passage and lead to incoherence and semantic/grammar errors. (See  $+\mathrm{COH}$ 's response in example A and C) This explains  $+\mathrm{COH}$ 's low coherence and readability scores on CNN-DM and Paper Abstracts. This observation suggests the necessity to carefully choose the weight for coherence rewards, and to coupling coherence reward with coverage reward, which could make the chat bot less abstractive.
3. Generally, user utterances could be classified into two categories: Information-seeking queries which request certain information (e.g. the user's utterance in example A); Open statements which do not have specific requests (e.g. the user's utterance in example B). We found evaluators tend to give high coherence scores to response to open statements, as they could be addressed by a wider range of responses.
Content-Grounded Dialogue Generation Content grounded dialogue generation is the task of using the information provided in external content (e.g. a passage, etc.) to guide dialogue generation. Compared to previous research, our task has the following novelties. 1) Compared to content-grounded information-retrieval-oriented dialogue such as doc2dial (Feng et al., 2020) and ABCD (Chen et al., 2021) where the chat bot responds to user query in a passive way, we expect our chat bots to convey knowledge proactively. 2) Compared to chit chat-oriented dialogue such as (Zhou et al., 2018; Dinan et al., 2019; Komeili et al., 2021; Xu et al., 2021), our task is more focused on extensive conversation in a particular topic, and aims at helping the end user acquire knowledge or information from a given passage. 3) Contrasted to chat bots that are applied in a single domain (Zhou et al., 2018; Moghe et al., 2018; Xu et al., 2019), our chat bot could be transferred to other domains through self-talk based fine-tuning.
Another line of research works focus on content grounded text generation models (Prabhumoye et al., 2021; Zhao et al., 2020). Compared with ordinary text generation models (e.g. BART (Lewis et al., 2020)), these models are specifically designed to model external content as an additional input, and achieve better performance on content grounded dialogue generation tasks including CMU DoG (Zhao et al., 2020) and Wizard of Wikipedia (Dinan et al., 2019).
RL in Text Generation Reinforcement learning has been applied in various natural language generation tasks, including image caption (Rennie et al., 2017), automatic summarization (Paulus et al., 2018), machine translation (Kang et al., 2020) and poem generation (Yang et al., 2019). Specifically, when applying reinforcement learning in dialogue generation (Li et al., 2016; Zhao et al., 2019; Shi et al., 2019; Yamazaki and Aizawa, 2021; Liu et al.,
2020), self-play is often used to enable scoring multi-turn dialogues. Compared to previous dialogue generation research using RL and self-play, our two-phase framework enables transferring the teacher bot to other domains by optimizing a mixed reward of coverage and coherence.
Educational Dialogue Systems and Conversational QA There have been research works applying dialogue systems for educational purposes. Some chat bots are for language practice. (Ruan et al., 2021; Stewart and File, 2007; Huang et al., 2017) Others are specially designed for education in a single domain or task, e.g. moral education (Peng et al., 2019), educational debate (Yuan et al., 2008). Compared with previous educational dialogue systems, our system is for information acquisition without domain restriction. Our task is also related to conversational question answering (CQA), e.g. (Kočisky et al., 2018; Rajpurkar et al., 2016; Joshi et al., 2017; Zellers et al., 2018). However, most existing CQA systems passively response to user queries in single turn conversations, while our system actively engage with users in multi-turn conversations.
We propose an information-acquisition-oriented dialogue system that transfers information and knowledge in passages to users through conversations. An unsupervised self-talk approach is introduced leveraging novel rewards to enable Teacher bots to deliver informative and attentive conversations. Experiments with automatic and human evaluations demonstrate the effectiveness of our approach. Some interesting future directions include extending the conversations to be based on a set of documents and specializing our dialogue systems for in specific domain, e.g. patient education.
Our models are pre-tuned on Wizard of Wikipedia dataset and fine-tuned on three corpora: Wikipedia, CNN-DailyMail and Paper-Abstracts (Abstracts of papers from ACL, EMNLP, NAACL, EACL, Findings and ICLR submissions from 2017 to 2021). All the datasets used in this paper are publicly available. Moreover, we did not use full-length Wikipedia or CNN-Daily Mail news articles in our experiments, but tailored versions of 100-150 words. This is because a full length Wikipedia/CNN-Daily Mail article may contain too much content to be covered in a short conversation.
As described in (Maynez et al., 2020; Kryscinski et al., 2020; Lebanoff et al., 2020; Zhou et al., 2021), current state of the art neural conditional text models can output hallucinated content unfaithfully to the input text, which impedes the safe deployment of the models. We note that our Teacher bots may also generate utterances that are not supported by the input passage.
Derek Chen, Howard Chen, Yi Yang, Alexander Lin, and Zhou Yu. 2021. Action-based conversations dataset: A corpus for building more in-depth task-oriented dialogue systems. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3002-3017, Online. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
John Dewey. 1923. Democracy and education: An introduction to the philosophy of education. macmillan.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of Wikipedia: Knowledge-powered conversational agents. In Proceedings of the International Conference on Learning Representations (ICLR).
Nouha Dziri, Ehsan Kamalloo, Kory Mathewson, and Osmar Zaiane. 2019. Evaluating coherence in dialogue systems using entailment. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3806-3812, Minneapolis, Minnesota. Association for Computational Linguistics.
Song Feng, Hui Wan, Chulaka Gunasekara, Siva Patel, Sachindra Joshi, and Luis Lastras. 2020. doc2dial: A goal-oriented document-grounded dialogue dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8118-8128, Online. Association for Computational Linguistics.
Roberta Michnick Golinkoff, Erika Hoff, Meredith L Rowe, Catherine S Tamis-LeMonda, and Kathy Hirsh-Pasek. 2019. Language matters: Denying the existence of the 30-million-word gap has serious consequences. *Child development*, 90(3):985–992.
Jin-Xia Huang, Kyung-Soon Lee, Oh-Woog Kwon, and Young-Kil Kim. 2017. A chatbot for a dialogue-based second language learning system, pages 151-156.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics
(Volume 1: Long Papers), pages 1601-1611, Vancouver, Canada. Association for Computational Linguistics.
Xiaomian Kang, Yang Zhao, Jiajun Zhang, and Chengqing Zong. 2020. Dynamic context selection for document-level neural machine translation via reinforcement learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2242-2254, Online. Association for Computational Linguistics.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online. Association for Computational Linguistics.
Tomáš Kocisky, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317-328.
Mojtaba Komeili, Kurt Shuster, and Jason Weston. 2021. Internet-augmented dialogue generation. CoRR, abs/2107.07566.
Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332-9346, Online. Association for Computational Linguistics.
Logan Lebanoff, John Muchovej, Franck Dernoncourt, Doo Soon Kim, Lidan Wang, Walter Chang, and Fei Liu. 2020. Understanding points of correspondence between sentences for abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 191-198, Online. Association for Computational Linguistics.
Rosemary Lever and Monique Sénéchal. 2011. *Discussing stories: On how a dialogic reading intervention improves kindergartners' oral narrative construction.* Journal of experimental child psychology, 108(1):1-24.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016. Deep reinforcement learning for dialogue generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1192-1202, Austin, Texas. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Qian Liu, Yihong Chen, Bei Chen, Jian-Guang Lou, Zixuan Chen, Bin Zhou, and Dongmei Zhang. 2020. You impress me: Dialogue generation via mutual persona perception. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1417-1427, Online. Association for Computational Linguistics.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919, Online. Association for Computational Linguistics.
Nikita Moghe, Siddhartha Arora, Suman Banerjee, and Mitesh M. Khapra. 2018. Towards exploiting background knowledge for building conversation systems. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2322-2332, Brussels, Belgium. Association for Computational Linguistics.
Suzanne E Mol, Adriana G Bus, Maria T De Jong, and Daisy JH Smeets. 2008. Added value of dialogic parent-child book readings: A meta-analysis. Early education and development, 19(1):7-26.
Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations (ICLR).
Yan Peng, Penghe Chen, Yu Lu, Qinggang Meng, Qi Xu, and Shengquan Yu. 2019. A task-oriented dialogue system for moral education. In International Conference on Artificial Intelligence in Education, pages 392-397. Springer.
Shrimai Prabhumoye, Kazuma Hashimoto, Yingbo Zhou, Alan W Black, and Ruslan Salakhutdinov. 2021. Focused attention improves document-grounded generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4274-4287, Online. Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of
the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017. Self-critical sequence training for image captioning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1179-1195.
Sherry Ruan, Liwei Jiang, Qianyao Xu, Zhiyuan Liu, G. M. Davis, Emma Brunskill, and J. Landay. 2021. Englishbot: An ai-powered conversational system for second language learning. 26th International Conference on Intelligent User Interfaces (IUI).
Thomas Scialom, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. 2019. Answers unite! unsupervised metrics for reinforced summarization models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3246-3256, Hong Kong, China. Association for Computational Linguistics.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083, Vancouver, Canada. Association for Computational Linguistics.
Mike Sharples. 2005. Learning as conversation transforming education in the mobile age. CiteSeer.
Weiyan Shi, Kun Qian, Xuewei Wang, and Zhou Yu. 2019. How to build user simulators to train RL-based dialog systems. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1990-2000, Hong Kong, China. Association for Computational Linguistics.
Iain Stewart and Portia File. 2007. Let's chat: A conversational dialogue system for second language practice. Computer Assisted Language Learning, 20.
Hu Xu, Bing Liu, Lei Shu, and Philip S Yu. 2019. Review conversational reading comprehension. arXiv preprint arXiv:1902.00821.
Jing Xu, Arthur Szlam, and Jason Weston. 2021. Beyond goldfish memory: Long-term open-domain conversation. CoRR, abs/2107.07567.
Takato Yamazaki and Akiko Aizawa. 2021. Phrase-level action reinforcement learning for neural dialog response generation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 5028-5038, Online. Association for Computational Linguistics.