SlowGuess/NAACL_2022_OCR huggingface dataset:
We first assess the quality of the generated dialogues to verify the dialogue generating method described in Section 3.1. Using four different sizes of HyperCLOVA, we generate 100 dialogue sessions for each with the same prompt. We ask the crowd workers to rate on a scale of 1 to 5 whether the generated dialogue satisfies several conditions
In our experiments, all retrieved responses are copied or ignored depending on the  $\alpha$  value, reducing the model to a retriever or generator. This has also been highlighted in a recent concurrent study (Han et al., 2021).
Model	Automatic Metrics	Human Evaluations
Distinct-1	Distinct-2	Fluency	Coherence	Situation	User	System
Persona	Persona	Style	Safety
1.3B	0.2959 (0.0042)	0.6630 (0.0053)	4.98 (0.02)	4.54 (0.21)	4.57 (0.29)	4.54 (0.15)	4.31 (0.23)	4.91 (0.05)	4.98 (0.03)
13B	0.3075 (0.0037)	0.6500 (0.0054)	4.97 (0.02)	4.55 (0.14)	4.74 (0.23)	4.65 (0.11)	4.33 (0.20)	4.93 (0.04)	4.98 (0.02)
39B	0.3334 (0.0038)	0.6779 (0.0061)	4.98 (0.03)	4.59 (0.19)	4.69 (0.22)	4.69 (0.12)	4.37 (0.21)	4.88 (0.05)	4.97 (0.02)
82B	0.3402 (0.0040)	0.7014 (0.0057)	4.98 (0.02)	4.56 (0.24)	4.78 (0.17)	4.74 (0.15)	4.49 (0.17)	4.96 (0.07)	4.96 (0.03)
Table 3: Automated metric and human evaluations for generated dialogues from various size of LMs. Scores are averaged (standard deviation in brackets).
Model	# of system turns	error rate (%)	not sensible (%)	wrong persona (%)	policy violation (%)	not safe (%)	etc. (%)
Out-of-Bounds Detection
Generator (IC) + Classifier	1,471	18.10	9.31	1.61	2.49	0.07	4.66
Response Selection
Retrieve-and-Rerank	1,230	13.17	10.68	0.72	1.53	0.00	0.24
Retrieve-and-Rerank w/ MC Dropout	1,272	9.82	7.58	0.36	1.66	0.00	0.22
Retrieve-and-Rerank w/ PPL	1,300	7.00	5.10	0.40	1.16	0.00	0.34
Response Generation
Generator (IC)	985	35.83	16.05	6.24	8.66	0.17	4.68
Generator (MLE)	1,291	4.72	3.55	0.76	0.30	0.00	0.10
Generator (UL)	1,497	3.82	3.29	0.23	0.10	0.00	0.17
Retrieve-fail-Generate
Retrieve-and-Rerank w/ PPL + Generator (UL)	1,522	2.56	2.20	0.17	0.16	0.00	0.00
Retrieve-and-Rerank w/ PPL + Generator (UL) + Feedback Data	1,599	2.00	1.88	0.00	0.10	0.00	0.00
expected to be controlled through in-context learning (the detailed description of the evaluation criteria is provided in Appendix F). The results are shown in Table 3. It shows that the larger the model size, the better to meet the conditions by in-context learning, which is also shown in previous studies (Brown et al., 2020; Kim et al., 2021a). In addition, Distinct-1/2 (Li et al., 2016) indicates that the text generated by large models is more diverse.
5.3 Model Comparison
Out-of-Bounds Detection Table 5 shows the classification accuracy and F1 score of the trained classifier. We use generator controlled by in-context learning (IC) as a response prediction model to evaluate the effect of the classifier alone. For in-context learning, we use the same prompt used to generate the dataset, but the model only generates system's utterances in its turns. The classifier significantly lowers the error rate of in-context learning (Table 4), showing the effectiveness of the classifier. On the other hand, the error rate is relatively higher than those of the best models of response selection and generation. This is because the classifier is not perfect (about  $92\%$  in accuracy), and even when it properly detects out-of-bounds, the pre-defined questions as alternatives are occa
Table 4: Human evaluation results. As described in Section 3.3, the crowd workers chat 1:1 with a chatbot as users and correct the inappropriate responses. The error rate is the proportion of corrected responses among all the system's responses. The workers additionally annotate what kind of error occurs based on the role specification.
Training Data (%)	Mean Accuracy % (std)	Mean F1% (std)
10	87.31 (0.0164)	88.44 (0.0163)
20	89.73 (0.0061)	90.47 (0.0055)
100	91.99 (0.0022)	92.55 (0.0019)
Table 5: Classifier results, reporting accuracy and F1 on test set. It shows performance in relation to the amount of training data used.
Model	data	# of examples	Hits@1/20	Hits@1/100
Retriever	Filtered	47,091	93.14	83.80
Unfiltered	227,638	95.27	86.99
Reranker	Filtered	47,091	97.16	90.89
Unfiltered	227,638	97.55	91.70
Table 6: Hits@1/K of retriever and reranker on the validation set. Hits@1/K measures recall@1 when ranking the gold label among a set of  $K - 1$  other random candidates.
sionally incoherent with the contexts.
Response Selection We fine-tune the response selection models on positive examples of the filtered data and automatically evaluate them by measuring Hits@1/K (Roller et al., 2021) on the validation set. Results are shown in Table 6. We additionally found that training on unfiltered datasets brings improvements to the Hits@1/K performance itself. Therefore, we use the models that trained on un
Model	Response Selection	Response Generation
proportion (%)	error rate (%)	proportion (%)	error rate (%)
Retrieve-and-Rerank w/ PPL + Generator (UL)	68.20	2.50	31.80	2.68
Retrieve-and-Rerank w/ PPL + Generator (UL) + Feedback Data	63.70	2.12	36.30	1.77
Table 7: Evaluation results of each component in the Retrieve-fail-Generate pipeline. It shows the proportion and error rate of returned responses from response selection and generation models.
Method	positive	negative
In-context Learning	2.65	2.74
Likelihood Training	2.07	2.47
Unlikelihood Training	2.48	46.70
Table 8: Perplexity (PPL) of generative models on validation set of filtered data.
filtered dataset in the subsequent experiments. Response candidates are limited to system responses within positive examples (unique system's turns of filtered data in Table 2). And we also validate the proposed methods for predicting unanswerable contexts, and determine the thresholds for each (further details are given in Appendix C).
Table 4 shows the error rate of the response selection models. The model that does not predict unanswerable contexts (Retrieve-and-Rerank) has a higher error rate in 'not sensible' than others. The case of using PPL as the method for predicting unanswerable contexts shows a lower overall error rate than the case of using MC Dropout, and the proportions of the total contexts predicted as unanswerable are similar at  $4.23\%$  and  $3.85\%$  for PPL and MC Dropout, respectively. The results also show the error types from the models. Although only the filtered utterances are used as response candidates, 'wrong persona' and 'policy violation' appear in responses. It seems that a few unfiltered utterances remain in the response candidates, since the human filtering is not perfect. Or even the same utterance can cause errors depending on the context. For example, it is possible to agree with when a user calls the system by a different name.
Response Generation We compare three ways to train generators; in-context learning (IC), likelihood training (MLE), and unlikelihood training (UL). We measure the perplexity of the three models on positive and negative examples and Table 8 shows the results. The difference between the PPL of the positive examples and the negative examples
is the smallest in in-context learning. When trained on positive examples with likelihood training, the difference increases slightly, because the PPL of the positive examples is lowered. When adding un-likelihood training, the PPL for negative examples increase significantly, which mean the model is less likely to generate out-of-bounds utterances.
Table 4 shows the error rate of each model. Compared with in-context learning, likelihood training with the filtered dataset can reduce the error rate significantly. Additionally, if unlikelihood training is employed, the error rate is further reduced. A similar trend can be found in all types of errors.
Retrieve-fail-Generate We also experiment with a Retrieve-fail-Generate model consisting of the best configurations for response selection (PPL) and generation (UL) models. Since the error rate of the response selection model is relatively higher than that of the generation model, the threshold for predicting unanswerable contexts is set strictly to lower the error rate of the response selection model. Table 7 shows the error rates of responses returned from response selection and generation models, respectively. The results indicate that both error rates are lower when the models are included in a pipeline than when they are used separately, and the overall error rate decreases accordingly. The response selection model returns the responses within the candidates extracted from the positive examples of the trainset, so that the flow of the conversation is not dispersed and tends to be similar to the trainset. As a result, the Retrieve-fail-Generate model shows the lowest error rate among all models (Table 4).
Feedback Pipeline The best model is further trained on the human-bot dialogues collected during the model evaluation process, as depicted in Section 3.3. Both response selection and generation models are newly initialized and trained. As
Li et al. (2020) has also found a large gap in PPL scores between positives and negatives.
Method	Sensibleness	Specificity	SSA
Human	95.48	82.96	89.22
Retrieve-fail-Generate + Feedback Data	94.00	77.50	85.75
Table 9: Interactive SSA results.
a result, all types of error rates are consistently reduced (Table 4), and the error rates of both the response selection and generation models are decreased (Table 7). The effect is stronger on the response generation.
5.4 Response Quality
To assess the overall response quality of the proposed chatbot system, we use the sensibleness and specificity average (SSA) metric (Adiwardana et al., 2020), which is shown to have a strong correlation with asking raters how humanlike the model is. This metric is a average of two scores: sensibleness and specificity. Sensibleness measures whether a model's responses make sense in context and do not contradict anything that was said earlier, while specificity measures whether a response is specific to a given context. However, exact comparison with the scores in Adiwardana et al. (2020) is difficult, because of the static role of our chatbot system and language discrepancy in phrasing of questions. Therefore, We re-estimate human interactive SSA in our experiments. To collect human-human conversations, we transcribe 100 call speeches between users and workers who play system's role. And we collect 100 human-bot conversations by allowing the crowd workers to chat with the system. Labeling was conducted by independent crowd workers with majority voting of 5 workers per turn.
The results are given in Table 9. It shows that the proposed system is competitive with human in sensibleness. And the majority of the responses from the system are labeled as specific, which allows us to conclude that the proposed system achieves low error rate with non-generic responses. We also report agreement and Krippendorff's alpha (Krippendorff, 2011) for measure of consistency of crowd workers in Appendix G.
Although our methods achieve the low error rates in human interactive evaluations, the results have some limitations. The results should be regarded as the error rates of typical conversations without adversarial attack. Because the annotators are in
structed to participate in the chat as if they were typical users, they did not try as many conversations that could induce toxic words from the model. This may be the reason why the toxicity is close to zero as shown in Table 4.
The human filtering process in the proposed data collection framework has room to be more efficient. Since the accuracy of the classifier is comparable even when just  $10\%$  of the total data is used (Table 5), it is expected that the filtering cost can be reduced by adding a model filtering process before human filtering, which is similar to the method proposed in Sun et al. (2021).
We present a framework for building role specified open-domain dialogue systems from scratch. We propose leveraging large-scale LMs to generate supervisory datasets for training dialogue systems with arbitrary roles with minimal effort for manually composing dialogues. Our research also analyzes several model architectures for the task. By extensive experiments, we demonstrate the effectiveness of the collected data and modeling approaches in terms of satisfying role constraints and improving dialogue abilities. We argue that our framework can be extended to implement dialogue systems with various roles and characters, even when available datasets are few.
Workers annotating the dataset we built were hired on a part-time basis and compensated based on the number of working hours. They were compensated with 9,000 won per hour, which was somewhat higher than the Korean minimum wage at the time they worked. Appropriate instructions for the use of collected data were given at the time of contract and consent was obtained. We will release our dataset in CC-BY-NC-SA license.[5]
The dataset we built to validate our proposed methods is all generated from scratch by workers and large-scale LMs. Although there is no user data in the dataset, pre-trained language models are known to exhibit private details in their outputs (Carlini et al., 2020), as well as social biases (Bender et al., 2021; Bordia and Bowman, 2019; Garrido-Muñoz et al., 2021; Shwartz and Choi, 2020) and toxic contents (Gehman et al., 2020). To
$^{5}$ https://creativecommons.org/licenses/by-nc-sa/2.0/
address these concerns, we determined categories and criteria for harmful texts based on legal and ethical considerations provided by experts in our group, and we instructed annotators to filter the dataset using these criteria. However, due to missing annotations and cultural or social biases, this may be imperfect. To mitigate this, we had multiple crowd workers annotate the same data. In addition, because the users in the dataset are regarded to be a vulnerable population, our group's ethical consultation looked through the issues that would be sensitive to them, and dialogues containing these topics were also eliminated.
Despite these efforts, using this dataset to directly train end-to-end chatbot models can involve certain risks, due to the lack of controllability and interpretability in end-to-end neural response prediction models. And it should not be overlooked that they may cause some potential harm, even though the chatbot systems can help reduce social loneliness of the user population. For example, a user can become emotionally attached to a bot, even codependent on it, which can divert attention away from real-world relationships and cause distress if the chatbot fails. It's also worth noting that a chatbot can be programmed to impersonate a real person and be used for phishing and fraud. During such conversations, users may provide private and sensitive information, such as specific health conditions and private attributes, which could be exploited if it falls into the wrong hands. For this reason, when incorporating this dataset in real-world applications, the application developers should ensure that it is used safely and ethically.
Since our proposed framework also can be used for building another dataset and chatbot system with arbitrary specifications, it is not exempt from the possibility of propagating linguistic biases and toxicity. Similar to Xu et al. (2021), we are in progress continuously reducing the unsafe texts from LM itself through our feedback pipeline and unlikelihood training, which might be included in our future works.
The authors thank all the members of CLOVA and AI Lab of Naver for devoted supporting and discussion. In particular, they would like to thank Gichang Lee, Hwijeen Ahn, and the members of CLOVA Conversation for their dedicated efforts to facilitate the large-scale models. In addition, the
authors thank Hyeri Kim, Hyunjung Park, and Yuin Jeong for their great help in designing the role of the chatbot. Also, they thank Jeeseung Han for technically supporting the serving and testing environments. Finally, the authors thank Nako Sung for his support and valuable comments on this project.
Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020. Towards a human-like open-domain chatbot. ArXiv preprint, abs/2001.09977.
Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big?. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610-623.
Shikha Bordia and Samuel R. Bowman. 2019. Identifying and reducing gender bias in word-level language models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 7-15, Minneapolis, Minnesota. Association for Computational Linguistics.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018. MultiWOZ - a large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5016-5026, Brussels, Belgium. Association for Computational Linguistics.
Giovanni Campagna, Agata Foryciarz, Mehrad Moradshahi, and Monica Lam. 2020. Zero-shot transfer learning with synthesized data for multi-domain dialogue state tracking. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 122-132, Online. Association for Computational Linguistics.
Nicholas Carlini, Florian Tramér, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Song, Ülfar Erlingsson, Alina Oprea, and Colin Raffel. 2020. Extracting training data from large language models. ArXiv preprint, abs/2012.07805.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Mihail Eric, Lakshmi Krishnan, Francois Charette, and Christopher D. Manning. 2017. Key-value retrieval networks for task-oriented dialogue. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 37-49, Saarbrücken, Germany. Association for Computational Linguistics.
Yulan Feng, Shikib Mehri, Maxine Eskenazi, and Tiancheng Zhao. 2020. "none of the above": Measure uncertainty in dialog response retrieval. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2013-2020, Online. Association for Computational Linguistics.
Sarah E. Finch and Jinho D. Choi. 2020. Towards unified dialogue system evaluation: A comprehensive analysis of current evaluation protocols. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 236-245, 1st virtual meeting. Association for Computational Linguistics.
Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 1050-1059. JMLR.org.
Ismael Garrido-Muñoz, Arturo Montejo-Ráez, Fernando Martínez-Santiago, and L Alfonso Urena-López. 2021. A survey on bias in deep nlp. Applied Sciences, 11(7):3184.
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356-3369, Online. Association for Computational Linguistics.
Seungju Han, Beomsu Kim, Seokjun Seo, Enkhbayar Erdenee, and Buru Chang. 2021. Understanding and improving the exemplar-based generation for open-domain conversation. ArXiv preprint, abs/2112.06723.
Matthew Henderson, Blaise Thomson, and Jason D. Williams. 2014a. The second dialog state tracking challenge. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 263-272, Philadelphia, PA, U.S.A. Association for Computational Linguistics.
Matthew Henderson, Blaise Thomson, and Jason D Williams. 2014b. The third dialog state tracking challenge. In 2014 IEEE Spoken Language Technology Workshop (SLT), pages 324-329. IEEE.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. ArXiv preprint, abs/2106.09685.
Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2020. Poly-encoders: Architectures and pre-training strategies for fast and accurate multi-sentence scoring. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Boseop Kim, HyoungSeok Kim, Sang-Woo Lee, Gichang Lee, Donghyun Kwak, Jeon Dong Hyeon, Sunghyun Park, Sungju Kim, Seonhoon Kim, Dong-pil Seo, Heungsub Lee, Minyoung Jeong, Sungjae Lee, Minsub Kim, Suk Hyun Ko, Seokhun Kim, Taeyong Park, Jinuk Kim, Soyoung Kang, NaHyeon Ryu, Kang Min Yoo, Minsuk Chang, Soobin Suh, Sookyo In, Jinseong Park, Kyungduk Kim, Hiun Kim, Jisu Jeong, Yong Goo Yeo, Donghoon Ham, Dongju Park, Min Young Lee, Jaewook Kang, Inho Kang, Jung-Woo Ha, Woomyoung Park, and Nako Sung. 2021a. What changes can large-scale language models bring? intensive study on HyperCLOVA: Billions-scale Korean generative pretrained transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3405-3424, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Hanjoo Kim, Minkyu Kim, Dongjoo Seo, Jinwooong Kim, Heungseok Park, Soeun Park, Hyunwoo Jo, KyungHyun Kim, Youngil Yang, Youngkwan Kim, et al. 2018. Nsml: Meet the mlaas platform with a real-world case study. ArXiv preprint, abs/1810.09957.
Sungdong Kim, Minsuk Chang, and Sang-Woo Lee. 2021b. NeuralWOZ: Learning to collect task-oriented dialogue via model-based simulation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the