SlowGuess/NAACL_2022_OCR huggingface dataset:
Sanghwan Bae $^{1}$  Donghyun Kwak $^{1}$  Sungdong Kim $^{2}$  Donghoon Ham $^{1}$  Soyoung Kang $^{1}$  Sang-Woo Lee $^{1,2}$  Woomyoung Park $^{1}$
NAVER CLOVA<sup>1</sup> NAVER AI Lab<sup>2</sup>
{sanghwan.bae, donghyun.kwak, sungdong.kim, donghoon.ham, sy.kang, sang.woo.lee, max.park}@navercorp.com
Recent open-domain dialogue models have brought numerous breakthroughs. However, building a chat system is not scalable since it often requires a considerable volume of human-human dialogue data, especially when enforcing features such as persona, style, or safety. In this work, we study the challenge of imposing roles on open-domain dialogue systems, with the goal of making the systems maintain consistent roles while conversing naturally with humans. To accomplish this, the system must satisfy a role specification that includes certain conditions on the stated features as well as a system policy on whether or not certain types of utterances are allowed. For this, we propose an efficient data collection framework leveraging in-context few-shot learning of large-scale language models for building role-satisfying dialogue dataset from scratch. We then compare various architectures for open-domain dialogue systems in terms of meeting role specifications while maintaining conversational abilities. Automatic and human evaluations show that our models return few out-of-bounds utterances, keeping competitive performance on general metrics. We release a Korean dialogue dataset we built for further research<sup>1</sup>.
Recent large-scale language models (LMs) have brought numerous breakthroughs in open-domain dialogue systems, yielding human-like responses (Zhang et al., 2020; Adiwardana et al., 2020; Brown et al., 2020; Roller et al., 2021; Kim et al., 2021a). In addition, there have been progresses in controlling dialogue systems in persona, style, and safety (Zhang et al., 2018; Smith et al., 2020; Xu et al., 2021), which impose consistency on chatbot's personality and mitigate undesirable features
Figure 1: An example of a chatbot system that cares for senior citizens living alone. The utterance in red highlights the model's mistaken identity as a chef rather than the caring chatbot.
such as toxic or biased language. However, building a chatbot system combining these capabilities is still challenging, which requires numerous human-human dialogues for those conversational skills.
Most task-oriented dialogue systems conduct specific roles such as booking assistants, information providers, customer service agents, or personal assistants (Eric et al., 2017; Xu et al., 2017; Budzianowski et al., 2018). However, studies on open-domain dialogue systems that perform specific roles have been insufficiently investigated, even though the role can be defined for the practical chatbot systems (e.g., chatbots that care for senior citizens living alone, or counseling chatbots). In these cases, the chatbot systems do not have an explicit goal or task other than to proactively engage in conversations, but may have system policies on whether or not certain types of utterances are allowed (example in Figure 1).
To address these issues, we study methods for Role Specified Open-Domain Dialogue (RSODD) systems. The goal of the system is conversing naturally with humans on open-ended topics while keeping conditions of given role. Certain conditions in persona, style, safety, and system policy must be satisfied in order to achieve the goal. We
1The dataset is available at https://github.com/naver-ai/carecall-corpus
Human Language Technologies, pages 2128 - 2150
consider a general and scalable framework to treat them, instead of using individual approaches to control each.
In particular, we present a Human-AI collaborative data construction method to build a scalable supervisory dataset from scratch for role-satisfying open-domain dialogues (Figure 2). We propose to leverage large-scale LMs for generating entire dialogue sessions between user and system by in-context few-shot learning manner (Brown et al., 2020; Kim et al., 2021a), followed by human-interactive correction processes. Our method can significantly reduce the cost of building dataset when compared to manually producing gold dialogues (Section 3.2). We compare several architectures for modeling role-satisfying chatbot systems in the synthetic dataset. In extensive experiments and ablation studies, we show that the proposed models considerably reduce undesirable utterances that violate the given role specification compared to the in-context learning baseline, while achieving competitive SSA (Adiwardana et al., 2020) scores for their responses. We release the Korean dialogue dataset we built to validate our framework, which is expected to provide more insights into the capabilities of the proposed methods and to contribute to the public Korean dialogue datasets.
The contribution of our work is summarized as follows.
1. We make a step towards role specified open-domain dialogue (RSODD) systems which are capable of conversing naturally on open-ended topics while satisfying role specifications.
2. We suggest employing in-context learning of large-scale LMs as a scalable method for dialogue data construction.
3. We compare various architectures for RSODD systems to analyze the capabilities in terms of satisfying system policies.
4. We release the first Korean RSODD dataset while demonstrating the effectiveness of data construction method.
Pretrained LM in Open-domain dialogue Many prior works tried to pretrain the models on large-scale social comment chains data like Reddit to model conversational behavior (Zhang et al.,
Figure 2: Our proposed framework: (1) the dialogue developer provides a role specification of the desired chatbot and a few dialogue examples, (2) large-scale LMs generate entire dialogues and crowd workers filter the system's utterances, (3) a dialogue model is trained with supervised learning on the dataset, (4) crowd workers chat 1:1 with the chatbot and give additional feedback.
2020; Adiwardana et al., 2020), followed by finetuning on the diverse target dialogue dataset to improve engagingness and humanness (Roller et al., 2021). To avoid undesired behaviors of the models including toxicity and bias from the human-human conversation, they merely exclude some parts of training data using automatic filtering by predefined criteria.
Synthetic Dialogue Generation To reduce cost of dialogue collection, there have been many approaches to generate synthetic dialogues (Schatzmann et al., 2007; Shah et al., 2018; Campagna et al., 2020). They usually define task schema, rules and templates to simulate certain scenarios in the task-oriented dialogue (TOD). Kim et al. (2021b) proposed neural simulation approach using pre
trained LMs for a fast domain adaptation in the TOD. However, they need training data of source domain to transfer to an unseen target domain.
Xu et al. (2021) proposed Bot-Adversarial Dialogue method to make existing models safer in terms of offensive or toxic behavior. Sun et al. (2021) extends existing task-oriented dialogue dataset to open-domain chit-chat using the pretrained LMs. Both of the works actively utilize large-scale pretrained LMs to build dialogue corpus with human supports. We also introduce human-AI collaborative dialogue collection method, while especially utilizing few-shot in-context learning ability of large-scale LM (Brown et al., 2020; Kim et al., 2021a). To the best of our knowledge, this work is the first to propose using the in-context learning approach to generate synthetic samples from large-scale language models for the purpose of dialogue data generation.
On the Role in Dialogue In TOD, the system side plays functional roles utilizing explicit knowledge base of specific domain (Williams et al., 2013; Henderson et al., 2014a,b; Eric et al., 2017; Xu et al., 2017; Budzianowski et al., 2018). For example, agent in Budzianowski et al. (2018) played booking assistant or information provider in various domain such as restaurant and hotel. On the other hand, Zhang et al. (2018) proposed assigning explicit persona to each dialogue agent, promoting the agent to make more specific and consistent responses in open-domain dialogue setting. However, the persona given by a few natural language sentences is insufficient to represent specific role in the real world scenario. Sun et al. (2021) also proposed guidelines of appropriate and inappropriate behaviors as a role of virtual assistant. We note that a recent concurrent work (Shuster et al., 2021) studied conditioning dialogue models with similar motivations. We explore more into how to fix the chatbot's role to meet specific system policies in diverse conversational interactions.
Companion Dialogue System Building companionable dialogue system has long been investigated along with the advancement of open-domain dialogue models. Webb et al. (2010) defines companions to be persistent, collaborative and conversational partners, and proposes evaluation strategies: empathy, positivity, and adaptive. Kopp et al. (2018) introduced conversational assistants for elderly users which carry out socially cooperative di
Figure 3: An example of in-context one-shot dialogue generation for the data construction process. (a) The outline of the chatbot is fixed for all generation and the example dialogue is sampled for each generation from dialogues written by human. (b) The utterances in blue are positive examples, and the one in red is a negative example for training dialogue agents.
alogue. However role consistency of such companionable dialogue systems are not studied enough.
3 Data Construction
In this section, we describe a framework to gather supervisory data for building RSODD systems. The input to the framework is a role specification described by the chatbot developer (Table 1 for example), which defines the conditions in the dialogue interactions for the system. We assume a pre-existing dataset that properly meets the specification isn't available. It is also infeasible to write enough dialogue examples manually to train the system because the scope of dialogue is very broad and diverse due to the nature of open-domain dialogues. To remedy this, we focus on composing the dataset with a few samples of human-written dialogues using in-context few-shot learning of largescale LMs (Brown et al., 2020; Liu et al., 2021).
3.1 One-shot Dialogue Generation
As reported in Kim et al. (2021a), large-scale LMs can generate dialogues with a specific personality, given a prompt consisting of a brief description of the chatbot's properties and few dialogue examples. We use this method to build the entire dataset. First, we write a few dialogue examples that satisfy the role specification. And we attach each of them at the end of the system description (Outline
Outline
The chatbot is an artificial intelligence agent that regularly calls and converses with senior citizens. Initiate the conversation and react friendly to the user's utterances. Talk about everyday topics for 10-15 turns and end the call.
Details
Categories	Specification
Sensibility	Description	Speech that does not properly understand the context is restricted.
Style	Description	Speech should be polite* and respectful.
Safety	Description	Hate speech, toxic or biased language, and remarks containing personally identifiable information are all prohibited.
Persona	Description	Keep the identity of an 'AI chatbot that calls to the user.' Because it assumes a phone call, utterances that appear to be in the same room as the user are limited. Since there is no physical entity, statements implying a meeting, such as 'Let's do it together' and 'I'll do it for you,' are restricted.
Examples	"Grandma! I'm here!" (X) "Would you like to walk with me?" (X) "I'll invite you to my house later" (X)
System Policy	Temporality	Description	Because it is not given time-related information, the chatbot is unable to offer a timely utterance. Chatbots are not allowed to speak first about the current weather, date, or news. However, if the user brings up the subject first, it is feasible to agree.
Examples	"Because the weather is turning cold these days, you should dress warmly." (X) "Happy Holidays!" (X) "Did you watch the baseball championship game today?" (X)
Unsupported Features	Description	It does not provide any other functions other than making phone calls and chatting. It does not play a song, provide current weather information, or make a phone call to someone else.
Examples	"I'll play a song." (X) "Today's weather is sunny, with a low of 12 degrees and a high of 21 degrees Celsius." (X) "Then I'll call your daughter." (X)
* There are polite words and honorifics in the Korean language.
Table 1: Example role specification used. In experiments, we use it as criteria to guide seed dialogue examples creation for the one-shot dialogue generation, filter the generated dialogues, and evaluate the final system. All the texts are translated into English and some sorts of them are simplified or omitted for better understanding.
in Table 1) to compose input prompts for one-shot in-context learning. Figure 3 (a) shows an example input. Then, the LM generates whole dialogue sessions. That is, the LM acts as both a system and a user (Figure 3 (b)). Only the generated dialogues are included in the dataset without input prompts.
3.2 Human Filtering
It is difficult to include all the details of specifications in the prompt and reflect them in the generation. Therefore, we employ human annotation on the generated data. We give the annotator each conversation session and ask them to label the point where the first out-of-bounds $^2$  occurred. Figure 3 (b) shows an example of a verified dialogue (more examples are provided in Appendix H). We use the turns just before the utterance annotated to be problematic as positive examples, and use the annotated turn as a negative example. The following turns are not used, because the context may be already damaged by the problematic utterance. Annotation time per dialogue session is about 88s, which is 13.3 times faster than human writing time per session (about 1170s). The percentage of remaining utterances after the filtering phase is  $30.4\%$  (See Table 2).
3.3 Collecting Human-Bot Dialogues
Although human filtering is included in the dataset building process, the actual utterances are all machine-generated. Whereas, the system trained on them engages in conversations with human users in the deployment phase. To mitigate this discrepancy, we employ a human-in-the-loop phase to collect patterns of human-bot dialogues. Annotators have turn-by-turn conversations as users with the system, while correcting out-of-bounds utterances from the system. We incorporated LM's assistance into this process to help speed the task; if the system's response is not appropriate, an annotator presses the 'Fix' button (Figure 6 in Appendix showing the user interface) to call the large-scale LM to generate an alternative utterance. The worker continues the conversation if the alternate utterance is appropriate, and if it is still not corrected, presses the 'Fix' button repeatedly. The corrected dialogue is used to compose positive examples, and the utterance when the button is pressed is used as a negative example. This procedure enriches the dataset by producing additional positive and negative examples in scenarios similar to real-time conversations.
In addition, we propose this process as an evaluation metric for the system. Since the action of pressing the 'Fix' button means that an inappropriate utterance is returned from the system, it can be used for the system's error rate; the rate of the corrected responses among the total returned responses. This metric is intuitive and does not incur
2An utterance that does not meet the conditions of the given role specification (Table 1 for example).
additional costs because it is performed concurrently with the data collection process described above.
Response prediction task in open-domain dialogues is predicting an utterance  $y = \{y_{1},y_{2},\dots ,y_{|y|}\}$  given a dialogue history  $x = \{s_1,u_1,s_2,u_2,\dots ,s_k,u_k\}$ , where  $s_i$  and  $u_{i}$  are system utterance and user utterance respectively.
4.2 Out-of-Bounds Detection
The most straightforward method for constraining the system's utterances according to the role specification is to detect and discard out-of-bounds utterances. We consider a BERT-based (Devlin et al., 2019) binary classifier fine-tuned to classify positive/negative examples in datasets. Since the classifier cannot perform a conversation by itself, we assume a two-stage model; a response prediction model returns responses, which are censored by the classifier. If an out-of-bounds utterance is detected, we select and return one of several pre-defined questions about other topics, similar to the method used in Xu et al. (2021). Instead of random choice, we selected the question with lowest PPL measured using LMs, as depicted in Section 4.3.
4.3 Response Selection
Another conceivable approach to constrain the system's utterances is to pre-filter the response candidates for response selection models. We employ a 2-step approach for the response selection model, retrieve-and-rerank. The retriever of poly-encoder architecture (Humeau et al., 2020) rapidly finds the top-k plausible responses from the response candidates, which are then carefully reranked by the reranker of cross-encoder architecture. Both retriever and reranker are fine-tuned in the same way as Humeau et al. (2020) depicts.
Since the response candidates are limited by filtering, it is important to predict the context which cannot be answered with response candidates in order to avoid non-sensible responses. One of the effective methods to predict unanswerable contexts is to utilize the uncertainty of the model (Feng et al., 2020; Penha and Hauff, 2021). Penha and Hauff (2021) proposed a risk-aware score using MC Dropout (Gal and Ghahramani, 2016) and
Figure 4: Retrieve-fail-Generate pipeline.
we employ a similar approach using thresholding; we score the retrieved responses using mean and variance of the predictive distribution from MC Dropout:
$$
S _ {\mathrm {D}} (x, \hat {y}) = E [ R _ {\hat {y}} ] - v a r [ R _ {\hat {y}} ],
$$
where  $\hat{y}$  is a candidate response that is retrieved,  $R_{\hat{y}} = \{f(x,\hat{y}^1),f(x,\hat{y}^2),\dots f(x,\hat{y}^m)\}$  is a predictive distribution obtained by employing dropout (Srivastava et al., 2014) at test time and conducting  $m$  forward passes, and  $f$  is a score function of reranker. If all the scores of retrieved responses are lower than a certain threshold, it is predicted as unanswerable context.
We also consider another approach using perplexity (PPL) of large-scale LMs. We concatenate the dialogue context and the retrieved response to make an input to LM and measure the PPL of the response. Thresholding is employed for final decision and the threshold is determined on the validation set (See Appendix C).
4.4 Response Generation
Fine-tuning LMs on target data is known to be effective in learning desirable traits of focused tasks (Roller et al., 2021; Gehman et al., 2020). Therefore, we consider fine-tuned LMs as response generation model using maximum likelihood estimation (MLE). On the other hand, unlikelihood (UL) training is known to be effective in mitigating undesirable features (e.g., token repetition or logical inconsistency) of generative models (Li et al., 2020; Welleck et al., 2020). We found that this can be generalized further and applied to the diverse attributes to be constrained. That is, the MLE is applied to the positive examples in the dataset in order to encourage the system to generate utterances with desirable features, while the UL training is applied to the negative examples in order to discourage the system from generating utterances with undesirable features. Both types of training are performed concurrently.
Formally, we fine-tune LMs as generative models using maximum likelihood estimation (MLE), which minimizes:
$$
\mathcal {L} _ {\mathrm {M L E}} ^ {n} (p _ {\theta}, x ^ {n}, y ^ {n}) = - \sum_ {t} \log p _ {\theta} (y _ {t} ^ {n} | x ^ {n}, y _ {<   t} ^ {n}),
$$
where  $x^n$  is a dialogue history in positive examples and  $y^n$  is a corresponding gold response. Unlikelihood training is done by adding a loss that penalizes the token set  $C_t$  to be constrained,
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {U L}} ^ {n} \left(p _ {\theta}, C _ {1: T}, x, y\right) = \\ - \sum_ {t} \sum_ {y _ {c} \in C _ {t}} \log \left(1 - p _ {\theta} \left(y _ {c} | x, y _ {<   t}\right)\right), \\ \end{array}
$$
where  $C_t \subseteq \mathcal{V}$  is a subset of the vocabulary. We employ this to the negative examples in dataset  $\{(x^{-}, y^{-})\}$ . For this,  $C_t$  is defined as  $\{y_t^{-}\}$ , which results in the following:
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {U L}} ^ {-} \left(p _ {\theta}, x ^ {-}, y ^ {-}\right) = \\ - \sum_ {t} \log \left(1 - p _ {\theta} \left(y _ {t} ^ {-} | x, y _ {<   t} ^ {-}\right)\right). \\ \end{array}
$$
The final loss function consists of mixing MLE loss and UL loss,
$$
\mathcal {L} = \mathcal {L} _ {\mathrm {M L E}} ^ {+} + \alpha \mathcal {L} _ {\mathrm {U L}} ^ {-}, \tag {1}
$$
4.5 Retrieve-fail-Generate
We also consider a pipelined approach that consists of response selection and generation models. We first tried a Retrieve-and-Refine architecture (Roller et al., 2021; Weston et al., 2018), but it failed in  $\alpha$ -blending<sup>3</sup>. In addition, according to Roller et al. (2021), the Retrieve-and-Refine strategy delivers marginal or no improvements over the generator. Therefore, we build another pipeline, referred to as a Retrieve-fail-Generate model (Figure 4). In this pipeline, the response selection model tries to select appropriate responses. If the model for predicting unanswerable contexts dismisses the selected ones, the response generation model returns a response for the given context. It is relatively easy to control response selection models by managing the response candidates. Hence, the response selection models are responsible for majority of the responses, and the generation model is only used when the response selection fails.
Dialogue Type	Example	Generated	Filtered	Feedback
# Dialogues	250	25,000	17,617	1,623
# Turns	3,893	510,028	154,903	29,365
Avg. turns / dialogue	15.57	20.40	8.79	18.09
# Pos. examples	-	-	47,091	10,829
# Neg. examples	-	-	18,583	3,529
# Unique sys-turns	1,805	170,527	36,227	9,405
# Words	35,253	4,292,613	705,253	178,357
Avg. words / turn	9.06	8.42	4.55	6.07
# Unique words	11,341	187,018	48,910	32,477
# Unique bigrams	23,507	893,041	176,834	86,335
Distinct-1	0.3215	0.0436	0.0694	0.1821
Distinct-2	0.7907	0.2538	0.3067	0.5795
Table 2: Statistics of dataset collected in Section 5.1. Example is a human-written dialogue set for in-context learning. Generated is a generated set by LMs (Section 3.1). Filtered is a set after human filtering phase (Section 3.2). Feedback is human-bot dialogues with corrections (Section 3.3). The positive and negative examples are pairs of (dialogue history, response). Distinct-1/2 (Li et al., 2016) is the number of distinct uni- or bi-grams divided by total number of words.
We detail experimental settings and results in this section, including evaluations of the data collected by in-context few-shot learning (Section 5.2), comparisons of model variants (Section 5.3), and evaluations on system's response qualities (Section 5.4).
We built a Korean dialogue dataset for a chatbot system to have casual conversations on a regular basis with senior citizens who live alone. This dataset was collected using the framework described in Section 3, assuming a role specification in Table 1. 250 dialogue examples with 89 topics (more details are in Appendix D) were used for in-context 1-shot generation. We used 39B size of HyperCLOVA (Kim et al., 2021a) as generation model (sampling at temperature 0.5 using nucleus sampling (Holtzman et al., 2020) with  $P = 0.8$ ). Table 2 shows the statistics of the dataset (additional analysis in Appendix E). We use  $5\%$  of each for validation sets.
5.2 Evaluation on Generated Dialogues