SlowGuess/NAACL_2022_OCR huggingface dataset:
Zhichao Yang, Pengshan Cai, Yansong Feng, Fei Li, Weijiang Feng, Elena Suet-Ying Chiu, and Hong Yu. 2019. Generating classical Chinese poems from vernacular Chinese. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6155-6164, Hong Kong, China. Association for Computational Linguistics.
Tangming Yuan, David Moore, and Alec Grierson. 2008. A human-computer dialogue system for educational debate: A computational dialectics approach. International Journal of Artificial Intelligence in Education, 18(1):3-26.
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 93-104, Brussels, Belgium. Association for Computational Linguistics.
Hainan Zhang, Yanyan Lan, Jiafeng Guo, Jun Xu, and Xueqi Cheng. 2018a. Reinforcing coherence for sequence to sequence model in dialogue generation. In *IJCAI*, pages 4567-4573.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018b. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204-2213, Melbourne, Australia. Association for Computational Linguistics.
Tiancheng Zhao, Kaige Xie, and Maxine Eskenazi. 2019. Rethinking action spaces for reinforcement learning in end-to-end dialog agents with latent variable models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1208-1218, Minneapolis, Minnesota. Association for Computational Linguistics.
Xueliang Zhao, Wei Wu, Chongyang Tao, Can Xu, Dongyan Zhao, and Rui Yan. 2020. Low-resource knowledge-grounded dialogue generation. In International Conference on Learning Representations.
Chunting Zhou, Graham Neubig, Jiatao Gu, Mona Diab, Francisco Guzmán, Luke Zettlemoyer, and Marjan Ghazvininejad. 2021. Detecting hallucinated content in conditional neural sequence generation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1393-1404, Online. Association for Computational Linguistics.
Kangyan Zhou, Shrimai Prabhumoye, and Alan W Black. 2018. A dataset for document grounded conversations. In Proceedings of the 2018 Conference
on Empirical Methods in Natural Language Processing, pages 708-713, Brussels, Belgium. Association for Computational Linguistics.
9.1 Implementation Details
9.1.1 Pre-tuning
Key hyper-parameters for both models.
- source max len: 1024
-target max len: 128
- batch size: 8
train epoch: 3
learning rate: 2e-5
- Both Teacher and Student bots adopt the initialized weights of bart-base from huggingface.
- Both DoHA models and BART models are based on the implementation presented in the DoHA paper (Prabhumoye et al., 2021).
Note we initialize our models with bart-base instead of bart-large as our self-play fine-tuning is very computational intensive and time consuming. With our current setting, the self-play fine-tuning takes about 2.5 days on one single NVIDIA Tesla V100 GPU.
9.1.2 Self-Play Reinforced Fine-Tuning
Mixed Loss. Optimizing our mixed loss is achieved by intermittently minimizing the MLE loss for a  $a$  batches and then minimizing the RL loss for  $b$  batches. In other words, the parameter  $\gamma$  in Section 2.3 is determined by  $a$  and  $b$ . Specifically, in each fine-tuning step, the parameters of Teacher bot are updated for 4 times, once over an RL batch, 3 times over MLE batches.
Key Hyper-Parameters. We apply the same set of basic hyper-parameters for all Teacher bots during all fine-tuning process:
- MLE batch size: 8
- RL batch size: 5
maximum coverage score: 0.5
train epoch: 3
learning rate: 1e-6
- Fine-tuning steps: 10,000
-  $\beta$  for three reinforced Teacher bots: FULL- $\beta = 0.7$ , +Cov- $\beta = 1.0$ , +Coh- $\beta = 0.0$
9.2 Coherence Scoring Models
Both WoW-coherence and InferConv are trained based on bert-base-cased.
max length: 256
- batch size: 32
learning rate 2e-5
- epochs 3
For InferConv classification model, the constant scores are  $s_e = 1.0$ ,  $s_n = 0.2$  and  $s_c = 0.0$
The WoW-coherence and InferConv classification model achieves  $82.1\%$  and  $88.4\%$  accuracy on respective test sets.
All other hyper-parameter settings for Teacher bot, Student bot and coherence scoring models are based on the system's default setting. All our experiments were run on servers with Nvidia A100 and V100 GPUs.
9.3 Dataset Details
The passages in train/validation/test set of our Wikipedia corpus are randomly sampled from passages in train/validation/test set of WoW respectively. Similarly, passages in train/validation/test set of our CNN-DM corpus are randomly sampled from the train/validation/test set of the original CNN-DM dataset. For Paper Abstracts, we randomly distribute all collected paper abstracts into train/validation/test sets.
9.4 Rewards function Details
During our exploration, we have explored multiple variations of coverage reward functions:
1. Reward score is gained at the end of each turn of conversation, and it is calculated the ROUGE score improvement of the teacher bot's utterance (As we applied in the paper);
2. Reward score is gained at the end of each turn of conversation, and it is calculated the ROUGE score improvement of both the teacher bot and student bot's utterances;
3. Reward score is gained at the end of the entire conversation, and is calculated as the ROUGE score improvement of all teacher bot's utterances.
According to our experience, the performance of 1 is similar than 2. Specifically, we notice in most cases, our student bot contribute marginally in terms of ROUGE score improvement. In addition, in practice, we observe 1 shows better performance than 3.
9.5 Human Evaluation Details
All human evaluation passages are collected randomly, the length of the Wikipedia and CNN passages are tailored to 100-150 words to conform to the length of passages in the fine-tuning stage. During conversations, we suggest our evaluators to use
utterances relevant to the topic, so that their utterances could be appropriately addressed by Teacher bots referencing content in the passage. All human evaluators we recruit have at least a bachelor degree, each evaluator is rewarded with a $20 gift card for participation.
The evaluation results are collected using Google Colab, an example evaluation page is available through this link (Personal information are anonymized during reviewing stage).
Note that in our human evaluation, Teacher bot chat about a passage with two evaluators. We use Pearson correlation coefficient to measure Interannotator agreement, the average Pearson correlation coefficient among each pair of evaluators is 0.16, implying weakly positive correlation. We note different evaluators use different utterances to chat with the same Teacher bot, thus it is reasonable that the conversation with the same Teacher bot over the same passage have large variance.
9.6 Examples
We provide more examples of conversations in self-play in Table 7 and human evaluations in Table 8.
Table 7: Conversation examples of Teacher bot fine-tuned with both rewards chat with Student bot during self-play. Example F demonstrates an example where Student bot's question seems too broad to be responded using content in the passage.
Table 8: Conversation examples of Teacher bot fine-tuned with both rewards chat with human evaluators.