SlowGuess/NAACL_2022_OCR huggingface dataset:
Pengshan Cai, $^{1*}$  Hui Wan, $^{2}$  Fei Liu, $^{3}$  Mo Yu, $^{2}$  Hong Yu, $^{1,4}$  Sachindra Joshi $^{2}$ $^{1}$ Manning College of Information & Computer Sciences, University of Massachusetts, Amherst  
 $^{2}$ IBM Research AI  
 $^{3}$ Department of Computer Science, University of Central Florida  
 $^{4}$ CHORDS, University of Massachusetts, Lowell  
{pengshancai, hongyu} @cs.umass.edu, {hwan, yum} @us.ibm.com, feiliu@cs.ucf.edu, jsachind@in.ibm.com
We propose novel AI-empowered chat bots for learning as conversation where a user does not read a passage but gains information and knowledge through conversation with a teacher bot. Our information-acquisition-oriented dialogue system employs a novel adaptation of reinforced self-play so that the system can be transferred to various domains without in-domain dialogue data, and can carry out conversations both informative and attentive to users. Our extensive subjective and objective evaluations on three large public data corpora demonstrate the effectiveness of our system to deliver knowledge-intensive and attentive conversations and help end users substantially gain knowledge without reading passages. Our code and datasets are publicly available for follow-up research.
Communication is the central process of education (Dewey, 1923). In learning as conversation (Sharples, 2005), a student does not read a passage but gains information and knowledge through conversation with a teacher who reads the passage. Compared to the traditional learning by reading, learning as conversation has the advantages that conversation helps students stay engaged and that information is provided piece by piece, which helps strengthen learning with a shorter attention span.
The advantages of learning as conversation have been verified with educational evidence (Mol et al., 2008; Lever and Senéchal, 2011; Golinkoff et al., 2019). For example, studies have shown that when children read storybooks, parents' guided conversation, e.g., posing questions and providing responsive feedback, substantially amplifies the learning benefits. While high-quality conversations with experts are not always available, it would be help-
ful if AI-empowered chat bots could be applied to facilitate users to gain information or knowledge.
Figure 1: The teacher bot educates an user about "BERT" through conversation generated from the document.
In recent years, there has been significant research in content-grounded dialogue generation, where external passages are used to inspire knowledge intensive dialogues. However, these systems or datasets are either for chit chat (Zhou et al., 2018; Dinan et al., 2019) or for goal-oriented information seeking (Feng et al., 2020; Chen et al., 2021), little work has explored applying chat bots for the learning as conversation purpose.
In this work we propose a novel task for learning as conversation: information-acquisition-oriented dialogue generation. Given a passage, our chat bot actively engages with an end-user to form a coherent conversation, so that the user could gain knowledge without reading the passage. Our task has a broad range of potential application venues in which people traditionally rely on reading to obtain information, including:
- Education: Chat bot helps an user gain knowledge from books or research papers;
- News and Media: Through conversation, a
* Work done during internship at IBM Research AI
1Code and data at: https://github.com/IBM/reinforced-dialog-system-for-learning
Human Language Technologies, pages 4781 - 4796
user could be provided stories tailored for his/her preference;
- Tutorial: While reading an instruction book could be tedious and time-consuming, a chat bot could efficiently walk an user through the process.
As shown in Figure 1, for our task, a good conversation should have the following characteristics:
1. Coverage: The chat bot should try to convey as much information in the passage as possible, instead of mumbling about irrelevant information;
2. Coherence: The chat bot's response should be coherent to the user's utterance, making the user feel that his/her questions are followed and addressed.
In summary, we propose a novel framework which consists of the following two chat bot modules: 1) Teacher bot, which attempts to transfer the information in an input passage to a user through conversation; and 2) Student bot, which responds to Teacher bot to form coherent conversations during training. The two bots are trained in a two-phase manner: In Phase 1, we pre-tune the two chat bots on Wizard of Wikipedia (Dinan et al., 2019) dataset, enabling both bots with the basic ability of conversing over a passage. In Phase 2, we fine-tune Teacher bot through self-play with Student bot, guided by reinforcement rewards. In this process, we enhance Teacher bot to be more informative while maintaining the ability to coherently address human users. Specifically, the fine-tuning phase is unsupervised, i.e. Teacher bot could be transferred to various domains or corpora without additional annotated dialogue datasets.
Our contributions include: 1) A novel task of information-acquisition-oriented dialogue system; 2) A novel unsupervised learning framework which enables a teacher bot to carry out informative and coherent conversations with human users for information acquisition purpose; 3) Extensive experiments with human evaluation demonstrate the effectiveness of our proposed approach.
In order to obtain an informative and attentive teaching dialogue system, we propose a framework that consists of two chat bots in different roles, and leverage both supervised learning and unsupervised reinforcement learning, as illustrated in Figure 2. The unsupervised reinforcement learning enables the system to be fine-tuned on other text corpus
where no annotation or dialogue data is required.
Phase 1: Supervised Pre-tuning
Phase 2: Unsupervised Self-Play Fine-tuning
Figure 2: Our two-phase training framework
Given a passage  $P$ , the conversation between Teacher bot  $X$  and Student bot  $Y$  can be denoted as a sequence of turns  $C = \{U_1^X, U_1^Y, \dots, U_N^X, U_N^Y\}$ , where  $N$  is the number of turns in the conversation. In order to mimic our use case, Teacher bot has access to  $P$  whereas Student bot does not.
Teacher bot  $X$  aims at transmitting the information in  $P$  to the student. At the  $n_{th}$  turn,  $X$  takes as input  $P$  and the conversation history  $H_{n}^{Y} = \{U_{1}^{X}, U_{1}^{Y}, \dots, U_{n-1}^{X}, U_{n-1}^{Y}\}$ , and outputs  $U_{n}^{X}$ . Teacher bot  $X$  adopts DoHA (Prabhumoye et al., 2021), a pre-trained model for document-grounded text generation, and is tuned in supervised phase and unsupervised self-play phase.
In order to fine-tune Teacher bot  $X$  with reinforcement learning on full conversations, as a practical approach, we train a Student bot  $Y$  to carry on conversations with  $X$ . Student bot  $Y$  takes the conversation history  $H_{n}^{Y} = \{U_{1}^{X}, U_{1}^{Y}, \dots, U_{n}^{X}\}$  as input, and output  $U_{n}^{Y}$ . It adopts BART (Lewis et al., 2020) model.
2.2 Phase 1: Supervised Pre-Tuning
This phase trains Teacher bot  $X$  to initialize and carry out conversations based on a given passage  $P$ , and trains Student bot  $Y$  to respond appropriately to  $X$ . To this end, we pre-tune both  $X$  and  $Y$  on the Wizard of Wikipedia (WoW) dataset. WoW was chosen as the pre-tuning dataset because of its two characteristics: 1) Open-domain: WoW contains conversations on a broad range of topics and domains across Wikipedia, thus the pre-tuned Teacher
and Student bots have greater potentials to be successfully transferred to other domains during the fine-tuning stage; 2) Content-grounded: in  $\text{WoW}$ , the teaching bot's utterances are grounded on passages, which is similar to our task. We present the gold passage to Teacher bot directly, though, different from the  $\text{WoW}$ 's original setting (Dinan et al., 2019) where Teacher bot searches a large corpus for supporting passages.
We optimize the maximum-likelihood objective for Teacher bot by minimizing the following loss:
$$
L _ {m l e} ^ {X} = - \sum_ {n = 1} ^ {N} \sum_ {m = 1} ^ {M _ {n}} l o g (p (x _ {m} | x _ {1}, \dots , x _ {m - 1}, H _ {n} ^ {X}, P))
$$
where  $N$  is the total number of turns in the conversation,  $\{x_{1},\ldots ,x_{M_{n}}\}$  is Teacher bot's response at the  $n_{th}$  turn,  $M_{n}$  is the number of words in  $U_{n}^{X}$ . The loss function for Student bot,  $L_{mle}^{Y}$ , is similar to  $L_{mle}^{X}$ , with the exception of not including a passage  $P$  as input.
$$
L _ {m l e} ^ {Y} = - \sum_ {n = 1} ^ {N} \sum_ {m = 1} ^ {M _ {n}} \log (p (y _ {m} | y _ {1}, \dots , y _ {m - 1}, H _ {n} ^ {Y}))
$$
where  $\{y_1,\dots,y_{M_n}\}$  is Student bot's response at the  $n_{th}$  turn.
2.3 Phase 2: Unsupervised Self-Play Fine-tuning
In this phase, we aim at improving Teacher bot's ability to present informative and coherent conversations. This is achieved by reinforcement learning on Teacher bot with the help of Student bot, and could be applied in a novel target domain even where dialogue dataset is absent. We adopt a self-play approach, i.e. we let Teacher bot and Student bot chat with each other over a passage in the target domain to generate multiple turns of conversations. In this fine-tuning phase, we keep Student bot frozen, and reward Teacher bot when the generated conversation achieves higher scores. In order to reduce the variance of the gradient estimate, we apply self-critic reinforcement learning (Rennie et al., 2017). Specifically, at each turn, we let Teacher bot generate two separate utterances: 1)  $U^s$ , which is sampled from the model, i.e.  $x_m^s \sim p(x|x_1^s, \ldots, x_{m-1}^s)$ , and 2)  $U^*$ , which is obtained by greedy search, i.e.  $x_m^* = \arg \max_w p(x|x_1^*, \ldots, x_{m-1}^*)$ . We optimize
the model by minimizing the following RL loss:
$$
\begin{array}{l} L _ {r l} ^ {X} = - \sum_ {n = 1} ^ {N} (R (U _ {n} ^ {s}) - R (U _ {n} ^ {*})) \sum_ {m = 1} ^ {M _ {n}} \log (p (x _ {m} ^ {s} \\ \left. \left| x _ {1} ^ {s}, \dots , x _ {m - 1} ^ {s}, H _ {n} ^ {X}, P\right)\right) \\ \end{array}
$$
where  $R()$  is the reward function, which we will cover in Section 2.4, and  $P$  is a passage from the target domain corpus.
If not taking into account language modeling, optimizing RL loss alone would lead Teacher bot to generate inarticulate and even grammatically incorrect utterances. To keep the fluency of Teacher bots, we optimize a combined loss  $L^{X}$  consisting of RL loss  $L_{rl}^{X}$  on the new target domain data and MLE loss  $L_{mle}^{X}$  on the pre-tuning dialogue dataset, so the language style acquired during the pre-tuning phase would not get lost during RL fine-tuning:
$$
L ^ {X} = \gamma L _ {r l} ^ {X} + (1 - \gamma) L _ {m l e} ^ {X}
$$
where  $\gamma \in (0,1)$  is a scaling factor accounting for the emphasis on  $L_{rl}^{X}$  and  $L_{mle}^{X}$ . We note that while  $L_{mle}^{X}$  should be obtained on an annotated content-grounded dialogue dataset (e.g. WoW),  $L_{rl}^{X}$  could be obtained on any target domain passage corpus even without dialogue data. This enables our approach to be transferred to an unsupervised text corpus.
2.4 Reward Functions
2.4.1 Coverage
We define the coverage reward of a Teacher bot's utterance  $U^{X}$  as:
$$
R _ {c o v} = \operatorname {R O U G E} _ {1} (P, H + U ^ {X}) - \operatorname {R O U G E} _ {1} (P, H)
$$
where  $\mathrm{Rouge}_1(P,H)$  is the Rouge-1 F1 (Lin, 2004) score of the conversation history  $H$  to the input passage  $P$ . Intuitively, this function favors utterances that cover more information in the passage and have less overlap with the conversation history.
2.4.2 Coherence
Dialogue coherence datasets We explore neural coherence scoring models trained on two open-domain dialogue coherence datasets:
1. WoW-coherence dataset We reuse the WoW dataset to heuristically build a dialogue coherence classification dataset. Specifically, for each multiturn dialogue in WoW, we label the ground truth response to its conversation history as coherent response, and all later responses in the same dialogue as incoherent responses.
2. InferConv dataset (Dziri et al., 2019) This is an open-domain dialogue coherence classification dataset built from PersonaChat conversational data (Zhang et al., 2018b). The dataset casts a response as the hypothesis and the conversation history as the premise, thus convert dialogue coherence evaluation into an NLI task. The dataset classifies the relationship between the response and the conversation history into three categories: entailed, neutral and contradict. Table 2 summarizes statistics of these datasets.
Coherence scoring models Based on the same pre-trained model BERT (Devlin et al., 2019), we train two different coherence scoring models on the two dialogue coherence classification datasets respectively. Both models take the concatenation (with [SEP]) of the conversation history and a candidate response as input, and minimize the cross entropy loss between the predicted label and the gold label. We use different methods to attain the coherence reward  $R_{coh}$  from the two models.
For model WoW-coherence, we define the coherence reward with softmax:
$$
R _ {c o h} = \frac {e ^ {o _ {c}}}{e ^ {o _ {c}} + e ^ {o _ {i}}}
$$
where  $o_c$  and  $o_i$  are the logits for coherent and incoherent labels in the output layer. For model InferConv, we observe some responses labeled as neutral are appropriate responses but are not closely related to conversation history (e.g. "That's interesting!"), we thus heuristically assign constant scores  $s_e$ ,  $s_n$  and  $s_c$  as coherence reward  $R_{coh}$  when the response is predicted as entailed, neutral and contradict. In the remainder of the paper, we use WoW-coherence as the default coherence model, and compare it with InferConv in Section 4.
2.4.3 Mixed Reward
The coverage and coherence rewards are combined with a hyper-parameter  $\beta$ , yielding the final reward:
$$
R = \beta R _ {c o v} + (1 - \beta) R _ {c o h}
$$
We proceed by describing our datasets, comparison systems and evaluation metrics. We then show the performance of our proposed approach compared to state-of-the-art in  $\S 4$ .
Pre-Tuning	WoW-Train	WoW-Valid
#Utterances	166,787	17,715
#Dialogues	18,430	1,948
#Words/utterance	16.6	16.6
#Words/passage	110.3	109.8
Fine-Tuning	Wikipedia	CNN-DM	Paper Abs.
#Passages	50,000	50,000	22,512
#Words/passage	111.7	129.8	149.3
Test Set	Wikipedia	CNN-DM	Paper Abs.
#Passages	1,000	1,000	500
#Words/passage	112.7	129.9	148.1
Table 1: Datasets statistics.
Wizard of Wikipedia (Dinan et al., 2019) contains a total of 22,311 human-human conversations crowdsourced via Amazon's Mechanical Turk. The conversations are grounded in Wikipedia passages covering a wide range of topics: e-book, toga party, armadillo, etc. Both Teacher and Student bots are pre-tuned on the WoW dataset during Phase 1. Different from WoW's original setting, we present the gold passage to the Teacher bot directly, instead of searching a large corpus for supporting passages. This allows us to focus less on retrieval and more on creating a Teacher bot to deliver informative and attentive dialogues.
We consider knowledge sources of various sorts as Teacher bot's target domain during fine-tuning. CNN/DailyMail contains a large collection of online news articles with an average of 781 tokens per article (See et al., 2017). The full content of the article cannot be conveyed in a short conversation. Thus, we use the first 130 tokens of each article as a supporting passage, assuming it covers the most important content of the news article.
Academic papers have become an omnipresent source of knowledge. We create our own dataset containing papers published in recent years (2017-2021) at major venues, including ACL, EMNLP, NAACL, EACL, Findings and ICLR conferences. Similarly, we use paper abstracts as supporting passages instead of full articles. Moreover, we include Wikipedia passages from the WoW dataset, without conversations, as another source of knowledge. The CNN-DM, Paper Abstracts and Wikipedia datasets are used in Phase 2 of unsupervised self-play fine-tuning. Statistics of these datasets are summarized in Table 1.
3.2 Comparison Models
Our baseline Teacher bot builds on the state-of-the-art content-grounded dialogue generation model:
WoW-Coh	Coherent Resp.	Incoherent	All
Train	74,092	233,142	307,234
Valid	3,939	12,362	16,301
Test	3,865	12,098	15,963
InferConv	Entail	Neutral	Contradict	All
Train	218,181	579,434	261,984	1,059,599
Valid	28,072	12,242	9,780	50,094
Test	3,119	1,087	1,360	5,566
Table 2: Details of the datasets used to train our neural coherence scoring models.
3 TeacherBot provides coherent responses to the evaluator's input.
2 TeacherBot provides largely coherent responses (with minor coherency issues) to the evaluator's input.
1 TeacherBot does not respond properly to the evaluator's input.
3 TeacherBot's responses are easy to read, containing no grammatical or semantic errors.
2 TeacherBot's responses read smoothly but may contain 1-2 grammatical or semantic errors.
1 TeacherBot's responses contain  $>2$  grammatical or semantic errors, or are nonsensical.
Overall Quality
An initial score of 3 is given to a dialogue, then 1 point is deducted for each of the following issues, with a minimum score of 0.
- Uninformative, i.e. it provides  $< 2$  correct answers during QA.
- Incoherent, i.e. the average coherence score is  $< 2$  points.
- Low readability, i.e. the average readability score is  $< 2.5$  points.
- Any other issues that could lead to an ineffective conversation e.g. words are repeated between turns.
Table 3: A scoring rubric provided to human evaluators.
DoHA (Prabhumoye et al., 2021). It includes two improvements to the architectures of pre-trained encoder-decoder models (Lewis et al., 2020): building context-driven representation of the supporting document, and enabling document-headed attention to acquire information from the document. DoHA has demonstrated strong performance in document-grounded generation. All DoHA models are pre-tuned on the WoW dataset.
Our FULL Teacher bot is created to converse in an informative and coherent manner. It extends DOHA by incorporating both coverage and coherence rewards in unsupervised self-play fine-tuning. Additionally, we ablate FULL model by removing each of the two rewards:  $+\mathrm{COV}$  uses only the coverage reward for fine-tuning, i.e. setting  $\beta = 1$  in our reward function (§2.4).  $+\mathrm{COH}$  utilizes only the WoW-coherence reward, i.e. setting  $\beta = 0$ . Please refer to appendix for more implementation details and hyper-parameters.
We investigate a wide range of metrics to evaluate Teacher bot's performance. Objective metrics measure the content coverage and coherence of Teacher bot's utterances. Subjective metrics, devised with human-in-the-loop, provide a holistic evaluation of a conversation, focusing on its overall effectiveness
and various aspects of linguistic quality.
Objective Metrics. Teacher bot converses with Student bot over a passage for three turns. That is, Teacher bot initiates the dialogue and provides two responses to Student bot. We objectively evaluate Teacher bot's utterances in terms of information coverage and coherence as follows.
- ROUGE (Lin, 2004) is one of the most widely used metrics for measuring information coverage. We consider three variants in this study: R-1, R-2 and R-L, which respectively measure the overlap of unigrams, bigrams and the longest common subsequence between the given passage and Teacher bot's utterances.
-  $QA_{F1}$  and  $QA_{\mathrm{CONF}}$  are two variants of SummaQA (Scialom et al., 2019), a question answering-based evaluation metric. If a conversation is rich in information, it could be used as a surrogate for the passage to answering important questions. To this end, SummaQA generates Cloze-style questions from a passage by masking out entities, then employs a QA system to answer those questions based on a conversation. A higher QA performance suggests the conversation has better coverage. Particularly,  $QA_{F1}$  reports the F1 score for question answering;  $QA_{\mathrm{CONF}}$  measures the confidence of the QA system in predicting answers.
- WoW-Coherence and InferConv are neural coherence scoring models ( $\S 2.4.2$ ) repurposed for evaluation. These models quantitatively assess if Teacher bot has produced a coherent response given the conversation history, or not.
-  $DPR_{\mathrm{RELV}}$  provides a new perspective on dialogue coherence evaluation (Zhang et al., 2018a). It draws on the Dense Passage Retriever model (DPR; Karpukhin et al., 2020) to predict if a Teacher bot's response is relevant to Student bot's input. A higher relevance score means the input and response share the same topic, suggesting a coherent conversation.
Subjective Metrics. We recruit 24 human evaluators to interact with Teacher bots. Each evaluator is asked to converse with bots over four passages. For each passage, the evaluator chats with four different Teacher bots for three turns, where Teacher bot initiates the conversation and responds twice to the evaluator's input. We randomly select 48 passages for evaluation, i.e., 16 passages from each of the three test sets. To evaluate conversations produced from Paper Abstracts, we require evaluators, 8 in total, to be either PhD students or have
Dataset	Model	Coverage Metrics	Coherence Metrics	Subjective Metrics	Avg Len
R-1	R-2	R-L	QACONF	QAF1	WoW-Coh	InferConv	DPRRELV	QAHUMAN	Coh	Read	Overall
Wikipedia	DoHA	48.09	41.27	44.31	19.76	19.39	0.503	0.550	0.555	30.0	2.07	2.79	1.84	15.51
+Cov	74.62	71.66	72.25	30.90	34.38	0.307	0.466	0.543	45.63	1.85	2.85	2.12	28.11
+COH	44.87	35.87	39.64	18.96	17.17	0.807	0.694	0.578	33.12	2.42	2.81	2.34	17.11
FULL	62.78	58.83	60.61	25.94	27.01	0.617	0.630	0.556	38.74	2.26	2.88	2.37	20.69
CNN-DM	DoHA	38.89	30.09	32.91	15.90	15.80	0.521	0.567	0.538	28.12	2.28	2.67	1.87	16.29
+Cov	81.52	78.52	73.73	30.98	38.46	0.253	0.381	0.525	58.59	2.14	2.79	2.37	36.08
+COH	30.45	18.61	24.0	13.31	11.31	0.845	0.692	0.561	40.15	2.40	2.48	1.93	16.23
FULL	65.77	60.45	57.83	25.5	30.36	0.604	0.692	0.559	52.5	2.57	2.71	2.53	29.76
Papers	DoHA	36.27	28.20	30.60	10.29	5.34	0.565	0.452	0.557	30.5	1.58	2.55	1.4	15.61
+Cov	72.63	69.69	49.18	20.32	17.61	0.271	0.141	0.529	57.5	1.73	2.77	1.96	33.55
+COH	32.96	21.24	26.21	8.18	3.95	0.806	0.547	0.576	25.47	1.82	2.26	1.25	16.3
FULL	59.65	54.22	47.88	15.46	13.81	0.766	0.501	0.560	51.76	2.16	2.53	2.09	27.37
Table 4: We compare Teacher bots based on naive DoHA(Prabhumoye et al., 2021) model to variants fine-tuned using different reward functions, Avg len refers to average utterance length.
obtained a PhD degree. For fair comparison, we shuffle and hide the order of Teacher bots presented to evaluators. Human evaluators were suggested to feed the same or similar inputs across Teacher bots on the same passage whenever possible. Throughout the conversation, the passage was not shown to the evaluators. After the conversation, human evaluators were asked to complete the following evaluation tasks:
-  $QA_{\text{HUMAN}}$ : Five sentences are randomly selected from each passage and one important entity is masked out in each sentence. The evaluators are presented with each corrupted sentence and asked if the sentence could be recovered by referencing the conversation with Teacher bot. We report the ratio of sentences that could be correctly recovered.
- Linguistic Quality: We ask human evaluators to rate each conversation along three dimensions: Coherence: Does Teacher bot provide coherent responses to the evaluator's input? Readability: Are Teacher bot's utterances easy to read, containing no grammatical or semantic errors? Overall Quality: How will the conversation score in terms of informativeness, coherence, readability and all aspects considered? The scoring rubric provided to human evaluators is shown in Table 3.
4 Objective Results