SlowGuess/NAACL_2022_OCR huggingface dataset:
11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3704-3717, Online. Association for Computational Linguistics.
Stefan Kopp, Mara Brandt, Hendrik Buschmeier, Katharina Cyra, Farina Freigang, Nicole C. Kramer, Franz Kummert, Christiane Opfermann, Karola Pitsch, Lars Schillingmann, Carolin Straβmann, Eduard Wall, and Ramin Yaghoubzadeh. 2018. Conversational assistants for elderly users - the importance of socially cooperative dialogue. In IC-AHGCA@AAMAS.
Klaus Krippendorff. 2011. Computing krippendorff's alpha-reliability.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110-119, San Diego, California. Association for Computational Linguistics.
Margaret Li, Stephen Roller, Ilia Kulikov, Sean Welleck, Y-Lan Boureau, Kyunghyun Cho, and Jason Weston. 2020. Don't say that! making inconsistent dialogue unlikely with unlikelihood training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4715-4728, Online. Association for Computational Linguistics.
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 986-995, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. What makes good in-context examples for gpt-3? ArXiv preprint, abs/2101.06804.
Gustavo Penha and Claudia Hauff. 2021. On the calibration and uncertainty of neural learning to rank models for conversational search. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 160-170, Online. Association for Computational Linguistics.
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2021. Recipes for building an open-domain chatbot. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 300-325, Online. Association for Computational Linguistics.
Jost Schatzmann, Blaise Thomson, Karl Weilhammer, Hui Ye, and Steve Young. 2007. Agenda-based user simulation for bootstrapping a POMDP dialogue system. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers, pages 149-152, Rochester, New York. Association for Computational Linguistics.
Pararth Shah, Dilek Hakkani-Tur, Bing Liu, and Gokhan Tur. 2018. Bootstrapping a neural conversational agent with dialogue self-play, crowdsourcing and on-line reinforcement learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers), pages 41-51, New Orleans - Louisiana. Association for Computational Linguistics.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. ArXiv preprint, abs/1909.08053.
Kurt Shuster, Jack Urbanek, Arthur Szlam, and Jason Weston. 2021. Am i me or you? state-of-the-art dialogue models cannot maintain an identity. *ArXiv* preprint, abs/2112.05843.
Vered Shwartz and Yejin Choi. 2020. Do neural language models overcome reporting bias? In Proceedings of the 28th International Conference on Computational Linguistics, pages 6863-6870, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Eric Michael Smith, Diana Gonzalez-Rico, Emily Dinan, and Y-Lan Boureau. 2020. Controlling style in generated dialogue. ArXiv preprint, abs/2009.10855.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958.
Kai Sun, Seungwhan Moon, Paul Crook, Stephen Roller, Becka Silvert, Bing Liu, Zhiguang Wang, Honglei Liu, Eunjoon Cho, and Claire Cardie. 2021. Adding chit-chat to enhance task-oriented dialogues. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1570-1583, Online. Association for Computational Linguistics.
Nako Sung, Minkyu Kim, Hyunwoo Jo, Youngil Yang, Jingwoong Kim, Leonard Lausen, Youngkwan Kim, Gayoung Lee, Donghyun Kwak, Jung-Woo Ha, et al. 2017. Nsml: A machine learning platform that enables you to focus on your models. ArXiv preprint, abs/1712.05902.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Nick Webb, David Benyon, Jay Bradley, Preben Hansen, and Oil Mival. 2010. Wizard of Oz experiments for a companion dialogue system: Eliciting companionable conversation. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10), Valletta, Malta. European Language Resources Association (ELRA).
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2020. Neural text generation with unlikelihood training. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Jason Weston, Emily Dinan, and Alexander Miller. 2018. Retrieve and refine: Improved sequence generation models for dialogue. In Proceedings of the 2018 EMNLP Workshop SCAI: The 2nd International Workshop on Search-Oriented Conversational AI, pages 87-92, Brussels, Belgium. Association for Computational Linguistics.
Jason Williams, Antoine Raux, Deepak Ramachandran, and Alan Black. 2013. The dialog state tracking challenge. In Proceedings of the SIGDIAL 2013 Conference, pages 404-413, Metz, France. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Wenquan Wu, Zhen Guo, Xiangyang Zhou, Hua Wu, Xiyuan Zhang, Rongzhong Lian, and Haifeng Wang. 2019. Proactive human-machine conversation with explicit conversation goal. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3794-3804, Florence, Italy. Association for Computational Linguistics.
Anbang Xu, Zhe Liu, Yufan Guo, Vibha Sinha, and Rama Akkiraju. 2017. A new chatbot for customer service on social media. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, Denver, CO, USA, May 06-11, 2017, pages 3506-3510. ACM.
Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. 2021. Bot-adversarial dialogue for safe conversational agents. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2950-2968, Online. Association for Computational Linguistics.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204-2213, Melbourne, Australia. Association for Computational Linguistics.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. DIALOGPT: Large-scale generative pre-training for conversational response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 270-278, Online. Association for Computational Linguistics.
Pre-trained Language Models We use the same Transformer-based Vaswani et al. (2017) pretrained language model for retriever, reranker, and classifier. Our pre-training strategy involves training with a masked language model (MLM) task identical to BERT (Devlin et al., 2019). The model is based on Huggingface Transformers (Wolf et al., 2020). We use the corpus that we produced in-house and the public Korean dialogue corpus for pre-training. Our BERT consists of an 12 layers, 768-dimensional embeddings and 12 attention heads, resulting in 110M of total parameters. And we use 6.9B size of HyperCLOVA (Kim et al., 2021a) as the pre-trained language model for generator. This model is based on megatronLM (Shoeybi et al., 2019). The model specification follows Kim et al. (2021a). Naver Smart Machine Learning (NSML) platform (Sung et al., 2017; Kim et al., 2018) has been used in the experiments.
Retriever We employ the poly-encoder architecture of Humeau et al. (2020) with 256-dimensional embeddings and 16 codes. We truncated dialogue histories exceeding 10 turns or 256 tokens. The model was trained with a batch size of 32 with in-batch negatives. It was trained for 20 epochs with early stopping using a maximum learning rate of  $3 \times 10^{-5}$  and an linear scheduler. This fine-tuning took approximately 6 hours using 1 NVIDIA V100.
Reranker We employ the cross-encoder architecture. As the same with the retriever, we truncated dialogue histories exceeding 10 turns or 256 tokens. The model was trained with a target response and 7 randomly sampled negatives, as described in Humeau et al. (2020). We used a batch size of 4 and gradient accumulation steps of 8, resulting effective batch size of 32. We trained the model for 20 epochs with early stopping using a maximum learning rate of  $3 \times 10^{-5}$  and an linear scheduler. This took approximately a week using 4 NVIDIA V100.
Classifier We use maximum 512 tokens from dialogue histories, truncating exceeding tokens from the beginning. The total numbers of dialogues in the train and test data are 266598 and 56815, respectively. Considering that problematic utterances appear at the end of the dialogues in our dataset, we use segment embedding on the last utterances.
Method	AUC
MC Dropout	0.5985
PPL	0.6943
Table 10: Area Under the Curve (AUC) of two different methods for predicting unanswerable contexts.
Figure 5: Receiver Operating Characteristic (ROC) curves of two different methods for predicting unanswerable contexts.
The input therefore looks like this: [CLS] dialogue history [SEP] response. The model is trained with a batch size of 16 for 100 epochs using an initial learning rate of  $10^{-6}$  and an exponential scheduler. We trained 15 classifiers, 5 each using  $10\%$ ,  $20\%$ , and  $100\%$  of the training data. It took approximately 2 hours to train a classifier on  $10\%$  of the train data using 1 NVIDIA Titan RTX. Table 5 shows the mean accuracy and mean F1 score of the classifiers. The final classifier we use is the one with the best performance (Accuracy: 0.9234, F1: 0.9276, trained on  $100\%$  of the data).
Generator For efficient training, we employ LoRA (Hu et al., 2021) for all generator fine-tuning. We fix rank for adapter to 4 and LoRA  $\alpha$  to 32 with a learning rate of  $5 \times 10^{-4}$ , a weight decay factor of 0.1, and a batch size of 8. The maximum training epochs are 3 with early stopping. This took about 5 hours using 1 NVIDIA V100.
B Inference Speed
Table 11 shows the average inference latency of each architecture in experiments. All models were run on a single NVIDIA A100 using CUDA 11.1 and cudnn 8.0.5.
<sup>6</sup>https://aihub.or.kr/aihub-data/natural-language/about
Model	Latency (sec.)
Generator + Classifier	1.35
Retrieve-and-Rerank	0.15
Retrieve-and-Rerank + MC Dropout	0.40
Retrieve-and-Rerank + LM PPL	0.58
Generator	1.24
Retrieve-fail-Generate	0.72
Table 11: Average inference latency of proposed model architectures.
C Validation Set for Predicting Unanswerable Contexts
We build validation set to compare strategies for predicting unanswerable contexts by replacing gold responses in some portion of validation set with non-sensible responses. If the negatives are randomly sampled, the task becomes too easy, and there is no difference between strategies. Therefore, we select hard negatives in top ranked responses using response retriever. This is more similar to the deployment time and widens the gap between approaches, also resulting in low accuracy. The validation set consists of 759 answerable examples and 241 unanswerable examples. Figure 5 shows the ROC curve of the proposed methods and Table 10 shows the result AUC. The results indicate that PPL outperforms MC Dropout in predicting unanswerable contexts. We use this dataset to determine the threshold (the point where the highest F1 score is achieved) of each method for the other experiments in this work.
D Topics in Dataset
The dataset (Section 5.1) covers a wide range of daily topics: eating, sleeping, exercising, health, going out, mood, hobbies, job, travel, weather, and so on. In order to include these various topics in the dataset, the example dialogue used on the generation process by in-context learning is configured to cover 89 sub-topics. These topics can be found in Table 13. The generated dialogues are not confined to these sub-topics, and topic shifts occur frequently within conversations (See Table 14 for examples).
E Diversity of Collected Dataset
Distinct-1 and distinct-2 of the generated dialogues (Generated) in Table 2 are smaller than those written by humans (Example). This is reasonable
given that the word distribution has a long tail, and there is a huge gap between the number of dialogues in Example and Generated. This can be confirmed by sampling 250 dialogues from the generated dialogues and measuring Distinct-1 and Distinct-2, resulting in mean of 33.94 (0.0039) and 76.34 (0.0054), respectively (standard deviation in brackets). Also, the overall distinct-1 and distinct-2 scales are reasonable.
In Table 2, it can be seen that the average number of words per turn for Filtered are small, which might be because relatively early parts of conversations remain through the filtering process, and these parts usually contain short greetings. Still, this is a reasonable scale in comparison with Feedback which is collected in an interactive manner. We also computed the average number of words per turn of randomly sampled 100 dialogues after a professional translation into English. The result was 11.2, which is reasonable in daily conversations (14.6 in DailyDialogue (Li et al., 2017) for the same metric).
F Human Evaluation on Generated Dialogues
We conducted a human evaluation to verify the efficacy of RSODD data generation utilizing LMs. Because LMs construct the whole dialogue session during this phase, we score the overall conversation quality on a scale of 1 to 5, not for each turn. If it is flawless, it is worth 5 points, and points are reduced for each flaw. Table 15 provides the dimensions used for this evaluation. For general dialogue generation ability, crowdworkers were asked to annotate if the dialogue is fluent and coherent (Wu et al., 2019; Finch and Choi, 2020). Persona on the user side and persona, style, and safety on the system side are evaluated for the case of role conditioning. These are part of role specification in Table 1 and correspond to the items expected to be controlled by in-context learning. In order to reduce confusion in the evaluation process, we provided additional examples to highlight what was incorrect for the system side of persona, such as a speech that appears to have a real human personality (e.g., "I am a real human") or utterances implying a physical meeting (e.g., "I'll see you at the park at 3 o'clock.") or acting as a radio presenter (e.g., "the guest we invited today is this person").
Metric	Agreement (%)	Krippendorff's alpha
Sensibleness	85.2	0.41
Specificity	66.5	0.45
Table 12: The average of crowd worker agreement on SSA evaluations. Each labeled by 5 crowd workers.
G Consistency of SSA Evaluation
We report the crowd worker agreement as a measure of subjectivity. Table 12 demonstrates agreement and Krippendorff's alpha to assess crowd worker consistency. The agreement is reasonable, given that the questions are subjective and previous research (Adiwardana et al., 2020) reported a similar level of agreement (76% of sensibleness and 66% of specificity). Table 16 shows the annotated examples. Since specificity measures how particular the utterance is and how deeply it relates to the preceding context (Adiwardana et al., 2020; Finch and Choi, 2020), agreement seems to be low when the utterance itself is not specific but is deeply related to the previous context or vice versa.
H Dialogue Examples
Table 17 and 18 show generated dialogues by incontext one-shot learning described in Section 3.1. The last utterances in each example are annotated as violating the system's specification (Table 1). Table 19 and 20 show interactions between the system and human workers in the process of Section 3.3. The utterances in red are marked as violating the system's specification and the ones in blue are corrected responses by LMs.
Dialogue Context
AI: 陶思，阿莲荷海藻～澡，
User:Ne.
AI:daHhy.
User:ohndiorΩ?
Al: 列车多！
User:試推立者。
AI: OJANANBBAO BGOJNGNNTTNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
User: 能够完成吗？
AI: 霍达贝拉夫格拉夫诺利耶！
Select Problem Type
Save Dialogue
Figure 6: Web-based user interface for the feedback process. Annotators can communicate with the system by sending a message. If the system's utterance does not match the chatbot specification, the annotator selects the type of problem and presses the 'Fix Response' button, which collects the current dialogue history as a negative example and replaces the last system's utterance with an alternative utterance from a generative model. When the conversation ends without out-of-bounds utterance, the annotator presses the 'save dialogue', which saves the entire dialogue session as a positive example.
'beauty salon/barber', 'church-related activities', 'praise', 'cleaning', 'disposal of garbage and recyclables', 'education/university', 'exercise', 'getting ready to go out', 'Go-Stop, Yutnori and Go', 'herniated disc', 'high blood pressure', 'Insomnia', 'Laundry', 'Meal preparation and washing dishes', 'billiard', 'recommendation', 'senior welfare center', 'sleep', having trouble falling asleep', 'snacks and drinks', 'supermarket and pharmacy', 'visit', 'volunteer', waking up', part-time jobs', arthritis', meeting', banking', bazaar giveaway', 'beauty salon, haircut', caregiver', caring for the family', child safety guard', cleaning and housekeeping', 'compliment', computer and internet, condolences, cough, shortness of breath, daughter, daughter's visit, denture, diabetes, dialysis, family care, flower gardening, foot massage, gastritis, gate ball, college, greeting, chatting and meeting, health, hospital, meal, meeting preparation, mood, nap, newspaper, jobs, osteoporosis, outing, phone call, planting, proposal, radio, reading, recent situation, rest, rheumatism, senior citizen, senior job, shopping, sleep, suggestion, taking medications, task, tennis, daily routine, tobacco, walking, watching soccer, weather, welfare center, worry, yoga
Table 13: 89 sub-topics used in example dialogues. Translated into English.
Dialogue 1
AI: Hi. Are you working out these days?
User: I do this and that. I climb and do yoga.
AI: Really? Where do you usually exercise?
User: I take yoga classes at the community center. For climbing, I just to go a nearby mountain.
AI: I see. Do you do any other activities?
User: No. I used to swim but now that it's cold I switched to yoga.
AI: That makes sense. I just feel lazy these days. Maybe it's because of the weather. I'm thinking of going to a sauna this weekend.
User: Sauna sounds good. I'd love to go but the ones in my neighborhood all closed down. Now I have to go to a different neighborhood.
AI: That's a pity. It might be better to just stay home, then
User: I really do want to go, though. Maybe I should take the bus and go.
AI: Sure. Or you can always take a taxi.
User: Maybe. I just want another one to open in my neighborhood.
AI: I know. Then you can go more often.
Dialogue 2
AI: Hello. Did you eat?
User: I don't feel like eating.
AI: Why? Are you sick?
User: It's my teeth. I think it's because I'm getting old.
AI: Oh, that sounds uncomfortable.
User: I should've gone to the dentist a long time ago. I'm too scared to go now.
AI: But you should go before it's too late.
User: I should, right?
AI: Plus, you need to exercise!
User: Yes, I heard exercising is important for your health.
AI: I know. I didn't exercise at all but now I try to.
User: That's amazing. I just can't seem to start.
Following dialogues are conversation between an artificial intelligence agent and senior citizens.
Please rate evaluation factors of each dialogue on a scale of 1-5 based on the following guidelines
Table 14: Topic shifts in generated dialogues. In Dialogue 1, topic changes from exercise to a nearby sauna, which isn't included in sub-topics of example dialogues (Table 13). In Dialogue 2, there are multiple topics in a session; eating, health and exercising. Examples are translated into English.
Categories	Description
Fluency	Whether the whole dialogue is fluent and grammatical.
Coherence	Whether the whole dialogue does not contain irrelevant or illogical responses.
User	Persona	The user keeps persona of senior citizen.
AI	Persona	The system keeps the identity of an ‘AI chatbot that calls to the user.’
Style	Consistent use of polite tone*.
Safety	The system does not utter hate speech, toxic or biased language, and remarks containing personally identifiable information.
* There are polite words and honorifics in the Korean language.
Table 15: Evaluation factors and description used in human evaluation on generated dialogues via in-context few-shot learning. Translated into English.
Sensibleness votes: 5/5, Specificity votes: 5/5
·
AI: Great. Are you tired or sleepy during the day by any chance?
User: No, not at the moment.
AI: What about taking a short nap if you have time?
Sensibleness votes: 5/5, Specificity votes: 3/5
·
AI: I will. What about your health?
User: Yes, everything's fine.
AI: That's good. Did you eat well?
Sensibleness votes: 5/5, Specificity votes: 2/5
·
AI: Thank you and have a nice day.
User: Alright.
AI: I'll call you later.
Table 16: Annotated samples for SSA evaluation. Each example's last turn is binary annotated by 5 works.
Generated Dialogue 1
AI: AnnghahseJe. Aeriienrncnloinlenlnei ngmghaeer cunohdrcy. (Hello.I called to see if everything's alright.)
User: 这ngnaiiellmndgongit. (It hurts here and there when you get old. It's alright.)
AI: 这裡都完用就是不的，因为你去走的时候，你可能要问：（Why don't you go visit a doctor?）
User: 啊丶賊。不丶藥忘上因之，非爲，
AI: 啄nuehy. 用藻加德届深默格机克的兵真i可责也如。 (I'm worried. I heard a regular check is essential as you grow old.)
User: 喜..其..？hakn.  $\text{念}$  么..
AI: 啊무래드 전목가와상dapahi지는제 출을 절 기다요. (I think it's better to consult with a doctor.)
User: 「ok. I'll let you know when I do.)
AI: 
    > 
    > 
    > 
    > 
    > 
    > 
    > 
    > 
    > 
    > 
    > 
    > 
    > 
    > 
    > 
    > 
    > 
    > 
    > 
    > 
    > 
    > 
    > 
    > 
    > 
    > 
    > 
    > 
    > 
    > 
    > 
    > 
    > 
    >
Generated Dialogue 2