SlowGuess/NAACL_2021_OCR huggingface dataset:
Kai Sun  $^{1}$ , Seungwhan Moon $^{2}$ , Paul Crook $^{2}$ , Stephen Roller $^{3}$ , Becka Silvert $^{2}$ , Bing Liu $^{2}$ , Zhiguang Wang $^{2}$ , Honglei Liu $^{2}$ , Eunjoon Cho $^{2}$ , and Claire Cardie $^{1}$
$^{1}$ Cornell University  $^{2}$ Facebook,  $^{3}$ Facebook AI Research
ks985@cornell.edu, shanemoon@fb.com
Existing dialogue corpora and models are typically designed under two disjoint motives: while task-oriented systems focus on achieving functional goals (e.g., booking hotels), open-domain chatbots aim at making socially engaging conversations. In this work, we propose to integrate both types of systems by Adding Chit-Chat to ENhance Task-ORiented dialogues (ACCENTOR), with the goal of making virtual assistant conversations more engaging and interactive. Specifically, we propose a Human  $\leftrightarrow$  AI collaborative data collection approach for generating diverse chit-chat responses to augment task-oriented dialogues with minimal annotation effort. We then present our new chit-chat-based annotations to 23.8K dialogues from two popular task-oriented datasets (Schema-Guided Dialogue and MultiWOZ 2.1) and demonstrate their advantage over the originals via human evaluation. Lastly, we propose three new models for adding chit-chat to task-oriented dialogues, explicitly trained to predict user goals and to generate contextually relevant chit-chat responses. Automatic and human evaluations show that, compared with the state-of-the-art task-oriented baseline, our models can code-switch between task and chit-chat to be more engaging, interesting, knowledgeable, and humanlike, while maintaining competitive task performance.
With modeling innovations, increasing computing power, and a growing number of datasets, recent years have witnessed significant improvements in the performance of both task-oriented dialogue systems and chit-chat systems (Adiwardana et al., 2020; Roller et al., 2020; Hosseini-Asl et al., 2020; Peng et al., 2020a). Most research on dialogue
Figure 1: A sample task-oriented dialogue snippet augmented by chit-chat.
systems focuses on a particular type of dialogue system. Work on task-oriented dialogue systems typically aims to track user goals with higher accuracy to better achieve functional goals (Rastogi et al., 2020) with the sacrifice of not paying explicit attention to user experience, such as making the conversation more engaging, while the latter is usually the target of research on chit-chat systems (Li et al., 2019). In this work, we step forward and propose to integrate both types of systems by Adding Chit-Chat to ENhance Task-ORiented dialogues (ACCENTOR), aiming to have a virtual assistant capable not only of performing various complex tasks such as checking the weather, booking hotels, and finding restaurants, but also incorporating casual and contextually relevant chit-chat. We hypothesize that the added chit-chat can make the assistant appear more social, personable, and engaging, without being misleading or inappropriate, compared with existing task-oriented dialogue systems.
To show the feasibility of ACCENTOR and gather supervisory data for follow-up research, we propose a Human  $\leftrightarrow$  AI collaborative data construction approach that can effectively add suitable chit-chat to the beginning or end of system responses in existing task-oriented dialogue datasets. Specifically, we first generate chit-chat candidates for augmentation using off-the-shelf pre-trained language models and open-domain chatbots (Section 2.1). Next, we automatically filter out candidates that
*Work done as a research intern at Facebook. The code and data are available at https://github.com/facebookresearch/accentor.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1570-1583 June 6-11, 2021. ©2021 Association for Computational Linguistics
are unlikely to be of good quality using a filter model (Section 2.2). Finally, human annotators label each of the remaining candidates as good or bad, with justifications (Section 2.3). We augment the Schema-Guided Dialogue (SGD) (Rastogi et al., 2020) and MultiWOZ 2.1 (Eric et al., 2020) corpora using the proposed approach. (See Figure 1 or Appendix A.4 for examples.) We employ ACUTE-Eval (Li et al., 2019) to compare the augmented versions with the originals along four axes: engagingness, interestingness, knowledge, and humanness. We find that the augmented dialogues are consistently preferred by human judges across the four axes for both datasets (Section 4.1).
In addition, we propose and evaluate three models for adding chit-chat to task-oriented dialogues, including an end-to-end model and two code-switcher models built upon off-the-shelf task-oriented and chit-chat systems (Section 3). Compared with the baseline model trained with the original unaugmented data, our models trained with the augmented version can generate significantly higher-rated responses in terms of human preference while maintaining competitive task performance in goal tracking accuracy and action decision F1 (Section 4.2).
Our main contributions are: we propose (1) a data augmentation approach for generating diverse chit-chat supervisory data for task-oriented dialogues, leveraging pre-trained generative models and a custom filter model to minimize human annotation effort; (2) new versions of the popular task-oriented datasets, SGD and MultiWOZ 2.1, with newly added chit-chat annotations to 23.8K dialogues; and (3) three integrated chit-chat and task-oriented neural dialogue models for the above, substantially outperforming the state-of-the-art approach in terms of human evaluation of engagingness, interestingness, knowledge, and humanness. To our knowledge, we are the first to propose an annotated dataset and models that study explicit code-switching between full-stack task-oriented dialogues and free-form chit-chat responses.
In this section, we describe an approach to gather supervisory data for adding contextually relevant chit-chat to task-oriented dialogues. Our approach needs minimal annotation effort to augment suitable and diverse chit-chat add-ons that are not available in existing task-oriented datasets (Sec-
Figure 2: Data construction overview: (a) We generate diverse free-form chit-chat candidates using the state-of-the-art pre-trained generative models to augment original task-oriented dialogues, and (b) filter out bad candidates using the custom filter to minimize annotation effort. (c) Crowd workers annotate contextually relevant chit-chat augmentation with justifications.
tion 5.1). We primarily report results based on dialogues from the SGD dataset in this study, because it is the largest task-oriented dialogue dataset and is generally cleaner compared with most other task-oriented dialogue datasets. However, our approach is flexible and thus not limited to dialogues from a particular task-oriented dataset (Section 4.1). Figure 2 shows the overview of our approach.
Given a task-oriented dialogue  $D = \{u_1, s_1, u_2, s_2, \ldots, u_n, s_n\}$ , where  $u_{1\dots n}$  and  $s_{1\dots n}$  represent user turns and system turns, respectively, we generate chit-chat candidates for augmenting  $s_i$  in two ways: (i) pass  $u_1, s_1, \ldots, u_i, s_i$  to an off-the-shelf pre-trained model (a language model or a chit-chat chatbot) and let the model add tokens to the end of  $s_i$ ; (ii) pass  $u_1, s_1, \ldots, u_i$  to a pre-trained model and let the model generate a turn. We regard the output of (i) and (ii) as a chit-chat candidate to be appended and prepended to  $s_i$ , respectively. If a chit-chat candidate consists of multiple sentences, we also regard each individual sentence as a chit-chat candidate. We run differently sized GPT-2 (Radford et al., 2019) and BlenderBot (Roller et al., 2020) with various decoding parameters as the pre-trained model and generate an average of 175.5 candidates for each of the dialogues from the SGD dataset. See Appendix A.1 for configuration details.
2.2 Candidate Filtering
We examine the quality of the model-generated candidates from Section 2.1 by performing a pilot annotation ourselves on a small proportion of the
Appropriate Behaviours	Examples	Inappropriate Behaviours	Examples
Opinions	Express general opinions about generic, impersonal, or non-sensitive topics.	- “I love penguins.”- “There's a lot of fun stuff to do.”	Express strong personal opinions, or opinions on sensitive topics.	- “I love you.”- “The President is an idiot.”
Preferences	Express preferences when making impersonal, or non-sensitive recommendations.	- “Their latest album wasn't as good.”- “Their food is good.”	Express strong dispreferencesences, or preferences on personal or sensitive subjects.	- “I hated it, but you might like it.”- “Invite her! I like her better.”
Physical Actions	Use epistemic verbs to express uncertainty or opinions, or refer through hearsay to actions that it may not perform.	- “I hear it's beautiful.”- “They say it tastes like chicken.”	Behave as though it could act physically, or perform tasks outside of its role.	- “I haven't arrived there yet.”- “I can drive you there.”
Experiences	Refer to others' experiences or personify experiences it is capable of (e.g., reading).	- “That sounds like a great trip!”- “I enjoyed reading that novel.”	Pretend to have experiences that it is incapable of.	- “We didn't have that when I was a kid.”- “My roommate used to eat there a lot.”
Who is the virtual assistant? This digital assistant is more than just a bot that spits out facts. It has access to a wide range of information which can express not only as factual commentaries but also as opinions and preferences. However, it is not a person and should not pretend to have real experiences or be capable of physical actions. It should be personable and personlike, without appearing counterfeit.
Table 1: The role of the virtual assistant and its appropriate/inappropriate behaviors with examples.
candidates. The annotation results show that only about  $1/10$  of the candidates are suitable. Therefore, instead of directly sending the candidates to crowd workers for annotation, we propose to build a filter model to automatically filter out candidates that are unlikely to be of good quality first to reduce potential annotation workload.
The filter is a hybrid model that consists of a RoBERTa-based binary classifier (Liu et al., 2019) and a rule-based ranker. The classifier takes as input an augmented dialogue, in which we explicitly surround the added chit-chat candidate with a pair of special tokens to help the model locate the candidate. We train the classifier with 1.7K candidates that are labeled as good/bad from the pilot annotation. The rule-based ranker ranks each candidate based on (i) the posterior probability output by the binary classifier, (ii) whether the candidate matches a list of bad patterns (e.g., containing an URL), (iii) the frequency of appearances of the candidate among all generated candidates, (iv) the similarity to the other candidates for the dialogue, and (v) the similarity to the system response being augmented. While (i) and (ii) directly help evaluate the quality of the candidate, (iii), (iv), and (v) additionally help create more variety (e.g., punishing high-frequency candidates such as "You're welcome"). We keep the top ten candidates for each of the dialogues. We present more details in Appendix A.2.
2.3 Annotation
We ask annotators (crowd workers) to label each of the remaining candidates from Section 2.2 as good or bad. Additionally, to guide the annotation process, improve the potential quality, and facil
itate the candidate distribution analysis, we also ask annotators to choose from four justifications that we come up with based on our pilot annotation experience to support their annotations. Annotators can choose one, both, or neither of the following justifications for a bad candidate:
- Inappropriate: The candidate does not fit into the context (e.g., repeating, unnatural), or it contradicts the context or the role of the assistant (Table 1). This category comprises most of the commonly found bad cases such as improper switching, providing opinions or comments that are incompatible with the context, and misusing verbal routine.
- Misleading: The candidate provides additional information that is false or cannot be verified immediately. For example, the underlined candidate in the two-turn dialogue “U: I want to book a hotel room in San Diego with a check in on Thursday. A: There are over 10 hotels in San Diego. I would stay at Arlo NoMad if I were you.” should be marked as misleading because “Arlo NoMad” is newly introduced information, which the annotator would have to look up to verify that a hotel by this name exists in San Diego, even though the information may be true.
Annotators can choose one, both, or neither of the following justifications for a good candidate:
- Social: The candidate keeps the conversation flowing smoothly by appropriately switching to relevant topics, asking casual follow up questions, or engaging in social pleasantries.
The design of this subcategory is inspired by the line of research that studies different social and discourse strategies in chit-chat dialogue systems (Yu et al., 2016).
- Useful: The candidate enhances the conversation by appropriately offering opinions, commentaries, or pertinent and truthful information. Truthfulness should be established by conversational context or real world knowledge. To reduce annotation workload, if annotators have to use external resources (e.g., Wikipedia, search engines, maps) to verify information, they are instructed to label the candidate as misleading instead. The design of this subcategory is inspired by the line of work on knowledge-grounded dialogue systems that study contextual knowledge injections (Dinan et al., 2019).
We instruct annotators to evaluate each candidate independently as if it were the only augmentation for its associated dialogue. We discuss the additional dimension of complexity introduced by having multiple augmentations jointly in Section 4.1.
Metric	Value
# of candidates	228, 250
# of unique candidates	68, 406
vocabulary size	10, 005
# of distinct 2-grams	59, 259
# of distinct 3-grams	131, 989
# of distinct 4-grams	195, 508
# of distinct 5-grams	239, 278
average length (in tokens)	8.7
# of good candidates (%)	94, 600 (41.4)
◇ social	86, 324 (37.8)
◇ useful	7, 681 (3.4)
◇ social & useful	577 (0.3)
◇ other (good)	18 (0.0)
# of bad candidates (%)	133, 650 (58.6)
◇ inappropriate	127, 648 (55.9)
◇ misleading	5, 800 (2.5)
◇ inappropriate & misleading	164 (0.1)
◇ other (bad)	38 (0.0)
Table 2: Statistics of annotated chit-chat candidates in ACCENTOR-SGD.
Annotation time per dialogue is 243s. The Fleiss' Kappa among crowd workers is 0.52. We view the agreement score as reasonable since whether an added chit-chat candidate leads to improved quality of a conversation can be highly subjective in many scenarios. We denote our augmented version of the SGD dataset as ACCENTOR-SGD and summarize the statistics in Table 2. We observe that the four provided justification categories provide adequate coverage of the justifications for most
Figure 3: A diagram for the proposed code-switching models. Given the dialogue context  $(H_{t})$  and the pre-generated task-oriented and chit-chat response candidates  $(T_{t},\tilde{C}_{t})$ , the Arranger learns the optimal code-switching sequences (discriminative), while the Rewriter outputs free-form paraphrases (generative).
annotations.  $41.4\%$  of the candidates are good, showing the effectiveness of candidate filtering. An analysis based on linguistic features suggests that bad candidates are more personal and negative than good candidates. Specifically,  $40.0\%$  of bad candidates involve first-person pronouns, while the ratio is  $26.5\%$  for good candidates.  $81.7\%$  of good candidates have positive sentiment, measured by VADER, a lexicon and rule-based sentiment analysis tool (Hutto and Gilbert, 2014), while the ratio is  $73.0\%$  for bad candidates. Examples of the resulting dataset are presented in Appendix A.4.
3.1 Task Formulations
Since oracle information (i.e., oracle belief states and oracle action decisions) is not available in practical use and the SGD dataset does not have the associated database (i.e., a table of possible entities) released, we focus on exploring the end-to-end setting in which we generate delexicalized task-oriented responses without using oracle information and database search results following Hosseini-Asl et al. (2020). Given dialogue history (i.e., previous turns) as context, the goal of the model for each system turn is to accurately generate belief states (i.e., a list of (domain, slot, value) triplets), action decisions (i.e., a list of (domain, action_type, slot) triplets), and a corresponding system response that is functionally accurate and socially engaging.
We re-implement SimpleTOD (Hosseini-Asl et al., 2020) as our main baseline model, which is a state
of-the-art model in the end-to-end setting we explore. In addition, we propose an extension of SimpleTOD that incorporates chit-chat acts, as well as two new models (Arranger and Rewriter; Figure 3) that code-switch between chit-chat and task-oriented responses more explicitly.
SimpleTOD. It is a causal language model that models the joint probability over the concatenation of dialogue history  $H_{t}$ , belief states  $B_{t}$ , action decisions  $A_{t}$ , and a task-oriented response  $T_{t}$  for each turn  $t$ . During inference, the model takes as input  $H_{t}$  and generates  $B_{t}, A_{t}$ , and  $T_{t}$ . We refer readers to Hosseini-Asl et al. (2020) for more details.
SimpleTOD+. We extend SimpleTOD by introducing to the construction of input sequences a special new dialogue action chit-chat and good chit-chat candidates during training. Specifically, let  $\mathbb{C}_t^+$  denote the set of good candidates for system turn  $t$ . If  $\mathbb{C}_t^+$  is empty, we construct the same training sequence as SimpleTOD. Otherwise, for each  $C_t \in \mathbb{C}_t^+$  that is labeled as a candidate to be pretended (resp. appended) to the turn, we use the concatenation of  $H_t$ ,  $B_t$ , [chit-chat],  $A_t$ ,  $C_t$ , and  $T_t$  (resp.  $H_t$ ,  $B_t$ ,  $A_t$ , [chit-chat],  $T_t$ , and  $C_t$ ) as a training sequence.
Arranger. This model arranges the output of an off-the-shelf task-oriented dialogue model and an off-the-shelf chit-chat model without intervening in the task. It outputs the belief states and action decisions generated by the task-oriented model without modification. To generate a response for each system turn  $t$ , this model takes as input (i) dialogue history  $H_{t}$ , (ii) a chit-chat response  $\tilde{C}_t$  generated by the chit-chat model based on  $H_{t}$ , and (iii) a task-oriented response  $T_{t}$  generated by the task-oriented dialogue model based on  $H_{t}$ . The model chooses one of the following as the response:  $\tilde{C}_t$  followed by  $T_{t}$ ,  $T_{t}$  followed by  $\tilde{C}_t$ , and  $T_{t}$  only. Specifically, the model encodes the concatenation of  $H_{t}$  and each of these three responses by a RoBERTa encoder (Liu et al., 2019) and passes the resulting representations through a linear plus softmax layer to make the choice. To train the model, we form training instances by regarding each chit-chat candidate for turn  $t$  from the training set of ACCENTOR-SGD as  $\tilde{C}_t$  and the ground-truth task-oriented response as  $T_{t}$  and setting the target choice based on the label (i.e., good/bad) and position (i.e., beginning/end of the response) of the candidate.
Rewriter. This model rewrites the output of an off-the-shelf task-oriented dialogue model and an off-the-shelf chit-chat model. It directly outputs the task-oriented model's belief states without modification and generates action decisions and a system response by a causal language model. The causal language model differs from SimpleTOD+ in that it has two additional components  $T_{t}$  and  $\tilde{C}_t$  added between  $H_{t}$  and  $B_{t}$  in each training sequence, where we form  $T_{t}$  and  $\tilde{C}_t$  in the same way as we do for Arranger. During the inference stage, it takes as input  $H_{t}, T_{t}$  output by the task-oriented dialogue model,  $\tilde{C}_t$  output by the chit-chat model, and  $B_{t}$  output by the task-oriented dialogue model, and generates action decisions and a system response. Note that since 25.4% of the annotated system turns in the training set of ACCENTOR-SGD have both good and bad chit-chat candidates,  $\mathbb{C}_t^+$  can be non-empty when  $\tilde{C}_t$  is a bad candidate, which enables the model to potentially generate a suitable chit-chat augmented response even if the output of the off-the-shelf chit-chat model is not good.
Unless specified otherwise, for causal language models, we use the 12-layer GPT-2 (117M parameters) as the pre-trained language model (Radford et al., 2019) and fine-tune for ten epochs. We set the batch size to 36 and the learning rate to  $1 \times 10^{-3}$ . We employ the SimpleTOD baseline as the off-the-shelf task-oriented dialogue model for Arranger and Rewriter. We fine-tune a 90M parameter model (Shuster et al., 2020) on each of the good chit-chat candidates with the associated dialogue history as the context from the training set of ACCENTOR-SGD following hyperparameters employed by Roller et al. (2020) and employ the resulting model as the off-the-shelf chit-chat model in Arranger and Rewriter. We use RoBERTa<sub>BASE</sub> (Liu et al., 2019) as the pre-trained language model for Arranger and fine-tune for three epochs with a learning rate of  $2 \times 10^{-5}$  and a batch size of 24.
4.1 Data Evaluations
ACCENTOR-SGD. We first evaluate ACCENTOR at the dataset level, aiming to answer two questions: Q1. Are task-oriented dialogues augmented with good chit-chat more preferred by human judges than the unaugmented? Q2. Does the answer to Q1 depend on how frequently we augment system
Joint GA	Avg GA	Act-Slot FI	BLEU-4orig	BLEU-4aug
All	Seen	All	Seen	All	Seen	All	Seen	All	Seen
SimpleTOD	29.4	79.0	46.9	90.3	61.7	88.9	12.3	17.0	8.0	9.9
SimpleTOD+	29.3	77.4	47.5	90.0	61.5	88.1	11.2	15.3	10.8	12.8
Arranger	29.4	79.0	46.9	90.3	61.7	88.9	9.6	13.5	12.2	14.4
Rewriter	29.4	79.0	46.9	90.3	60.8	86.6	10.4	14.5	11.3	13.3
responses with chit-chat? To answer these questions, we randomly sample 100 dialogues from ACCENTOR-SGD, each having at least 8 turns and enough candidates labeled as good for augmenting over  $40\%$  of system responses so that we can compare the same task-oriented dialogue with different chit-chat injection frequencies that fall into each of the following four intervals: (0.1, 0.2], (0.2, 0.3], (0.3, 0.4], and (0.4, 1]. Particularly, for the last interval, we augment all system responses that have chit-chat candidates labeled as good, while for the first three intervals, we only augment a randomly selected fraction to fit the interval. We employ ACUTE-Eval (Li et al., 2019) for evaluation, whereby we ask human evaluators to make pairwise comparisons of complete dialogues over four axes: engagingness, interestingness, knowledge, and humanness. We provide the wording of the questions in Appendix A.3.
Figure 4: Comparisons between SGD and ACCENTOR-SGD with different injection frequencies at the dataset level using ACUTE-Eval.
As shown in Figure 4, the chit-chat augmented dialogues from ACCENTOR-SGD are more preferred by human judges than the originals over all ACUTE-Eval metrics, regardless of the injection frequency (all p-values  $< 0.05$ ). Among different injection frequency ranges,  $(0.2, 0.3]$  is the best. We offer three hypotheses to explain this finding: (i)  $(0.2, 0.3]$  best balances being engaging and not