SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 4 gives results for RPA-based re-ranking of generation models. Automated results show a slight bump in F1 on the LIGHT valid set, and indeed a bump in RPA. Including the intra-generation re-ranking with PACER yields an even higher RPA score. Table 3 contains the results of varying the candidate tokens re-ranked per intra-generation step (#Toks) and number of partial re-ranking steps (Freq), both in terms of generation metrics/RPA and relative computational cost compared to re-ranking. Increasing # of toks or increasing the frequency can lead to improved F1 and RPA, but with significant latency increase for too high values (e.g. over 11x when applying re-ranking for every partial step using the top 10 tokens each time). Applying both partial and final complete ranking helps performance. Note that re-ranker models use the same model to re-rank that is being used to measure RPA afterwards, making that metric biased. Hence, human evaluations are required for this model, which will be detailed in Section 4.7, and which will indicate that re-ranking does in fact help.
4.4 Unlikelihood
Results of unlikelihood (UL) training are also given in Table 4. We apply UL loss to the 128-truncation model in two different ways: (1) Top-1: apply the loss on the token that yields the most incorrect partial sequence RPA classification; (2) All: apply the loss to all tokens that yield an incorrect RPA classification on partial sequences. The RPA UL methods suffer compared to the baselines in terms of PPL and F1, yet they retain similar RPA metrics. We hypothesize that while the UL loss can adjust the model to refrain from generating out-of-character responses, there are still far too many other tokens that may yield similar outcomes that are not penalized. Table 12 in Appendix D includes similar results with the 1024-truncation model.
4.5 Multi-Objective Training
Multi-objective training results are in Table 5, where the base model is a 1024-truncation model. We measure generation metrics in terms of RPA (with PPL and F1 in Table 13 in Appendix E), and classification metrics in terms of Hits@1/427 as before. The model is able to predict the appropriate character using either the decoder outputs or the encoder+decoder outputs. For each case,  $n_{MO} = 2$  yielded better results than  $n_{MO} = 0$ . Interestingly, despite the relatively strong performance of the model in classifying the right character (87.42
Input	nMO	Stage	RPA	Hits@1
Human	N/A	-	92.8	-
None	0	0	88.4	-
Multi-Objective
Dec. only	2	1	88.4	39.3
Dec. only	2	2	87.7	87.4
Enc+Dec	2	1	88.4	70.9
Enc+Dec	2	2	88.8	71.6
Multi-Objective + Automated Expanded Attention
Dec. Only	0	1	89.1	86.4
Dec. Only	0	2	89.1	89.1
Enc+Dec	2	1	88.4	83.3
Enc+Dec	2	2	89.1	88.5
hits@1 for the best model), this does not translate to substantial RPA improvements over the baseline.
4.6 Expanded Attention
Profile Grounding Expanding the decoder attention yields significant gains across all automated metrics, as seen in Table 6 for a 1024-truncate model (and in Table 15 in Appendix F for a 128-truncate model). As a baseline we explore simply re-attending to the full context again; this indeed improves metrics across the board for the short-context model, but the long-context model actually suffers. However, both models improve substantially over the baseline when including the full LIGHT context without the dialogue history, and attention over sub-components of the LIGHT context still yields strong improvements.
To see how much this expanded attention matters, we explored varying the number of rounds  $r \in \{1,2,3\}$  of expanded attention, i.e., how many times the model attends to this additional context. In Table 6, we also see that a second expanded attention round yields even better results, but performance drops off after applying a third round.
Automated Grounding We show results for the automated grounding of expanded attention in Table 7. Attempting to use the decoder attention weights to select expanded attention context yields no additional benefits, which is not surprising: if the model could identify the pertinent components of the input beforehand, it would not require a re-attention. The trainable mask does not yield any benefits either. However, using the RPA classifier attention weights to inform the model which tokens to re-attend to yields improved performance across all three metrics compared to the baseline, and PPL
Table 5: Models trained with varying multi-objective setups, evaluated on the valid set. Models are initialized from a (1024-truncation) model fine-tuned on LIGHT.
Expanded Attention	r	1024-Truncate	Model
PPL	F1	RPA
Human	0	-	-	92.80
None	0	12.35	15.85	88.42
ABCD + Dialogue Hist.	1	12.47	15.82	88.34
ABCD	1	12.18	16.01	91.82
ABCD	2	12.17	15.95	92.60
ABCD	3	12.19	15.99	91.73
ABC	1	12.22	15.94	91.83
ABC	2	12.24	15.99	92.24
ABC	3	12.25	15.93	92.25
AB	1	12.27	15.87	90.97
A	1	12.30	15.80	89.13
B	1	12.34	15.76	89.46
Table 6: Models trained with expanded attention (profile grounding), evaluated on the valid set. Expanded attention input: A = Self Persona, B = Self Name, C = Partner Name, D = Setting Description. We also vary the number of rounds  $r$  of expanded attention.
Exp. Attn. Grounding	PPL	F1	RPA
Human	-	-	92.80
None	12.35	15.85	88.42
Profile Ground Best (2 rounds)	12.17	15.95	92.60
Profile Ground Best (1 round)	12.18	16.08	91.79
Profile Ground Random	12.43	15.74	87.62
Decoder Attn.	12.39	15.40	87.59
Trainable Mask	12.40	15.75	88.43
Classifier Attn. (top-k)	12.19	15.90	91.11
Classifier Attn. (bottom-k)	12.31	15.89	88.71
Table 7: Models trained with expanded attention (automated grounding), evaluated on the valid set. We vary the method for selecting the extra context to re-attend to. All models are long-truncation (1024).
is nearly the same as profile grounding (12.19 vs. 12.18), while RPA trails slightly behind (91.11 vs. 91.79). We also include the usage of the bottom- $k$  tokens from the classifier weights to emphasize that there is indeed signal from the top- $k$ , as using the bottom tokens does not help.
Automated Grounding + Multi-Objective Table 5 shows that combining automated grounding with the multi-objective task yields higher hits@1 compared to not using the trainable mask, especially in the first stage of multi-objective training. However, RPA scores are only fractionally better than the baseline. Appendix E includes results across more settings (see Table 13 and Table 14).
Expanded Attention + RPA Re-ranking The expanded attention and RPA re-ranker methods can also both be applied to obtain effective models. Results are in Table 4; indeed, the combination yields the highest F1 and RPA scores.
4.7 Human Evaluations
We performed human evaluation on our models. For each model we collected 100 human-model conversations, set up similarly to the original LIGHT dataset conversations. During the conversation, crowdworkers were asked to annotate the model's response for the following attributes: 1) Mistaken Identity: your partner says something that would imply they believe they're someone other than who they're noted to be; 2) Contradiction: your partner says something that contradicts something they've said before; 3) Wrong Location: your partner says something that would imply they believe they are in a different location than the provided one; 4) Unrelated: your partner says something that doesn't follow the previous turns; and 5) Repetitive: your partner says something they've already said, or are driving the conversation in circles. Utterances that do not contain any of the negative attributes are denoted "all good". Finally, we collect an engagingness score on a scale of 1-5 at the end of the conversation. More details in Appendix I. Results are given in Table 4, with more details results (and comparison with a retrieval baseline) in Table 19 in the Appendix. The baseline model displays mistaken identity  $6.45\%$  of the time, and has an average engagingness score of 4.04. Longer context increases engagingness to 4.16 but also increases mistaken identity. Unlikelihood and multi-objective training similarly increase mistaken identity. The successful methods, then, are the beam reranking methods and the expanded attention models. The long-context beam re-ranker decreases mistaken identity to  $4.81\%$ , while the profile expanded attention model improves to  $4\%$ , and has the best engagingness of 4.34. Combining RPA Re-ranking with expanded attention yields the lowest mistaken identity  $(2.38\%)$ , while adding PACER leads to the highest all-good percentage  $(85.7\%)$ . Correlations between automatic metrics and human evaluations are measured in Appendix K, where we find that RPA and mistaken identity are indeed strongly correlated.
5.1 Re-rankers & Generation Settings
We further explored three decoding settings: standard beam-search, delayed beam search (Massarelli et al., 2020) and nucleus sampling (Holtzman et al., 2020), both in a re-ranking setting and not. When considering performance on automated metrics
provided in Table 20 in the Appendix), we see that generation settings other than beam search, when using a re-ranker, yield lower F1 scores but higher RPA scores, as the RPA re-ranker has more diversity of candidate responses from which to choose; however, these methods perform worse in human evaluations, with nucleus sampling reranking yielding far more problems and far lower engagingness ratings. Qualitative analysis of outputs on the test set are in Appendix J.1.
5.2 Classifier Failure Modes
We note that the human dialogue data is classified as being "in character" only  $92.8\%$  of the time on the validation set by the LTR RPA classifier. We examine the scenarios in which the classifier is incorrect, with example input/output pairs in Table 21 in the Appendix. First, there are instances where either character could have said the output response (row 1). Second, there are instances where there are not enough clues in the context to provide an estimation of who said the response, for example at the beginning of the conversation (row 2). And, there are still some small amount of instances that the classifier simply fails (row 3).
5.3 Model Failure Modes
We analyze the results of turn annotation to understand what failure modes contribute to mistaken identity. A full list of such modes is in Table 16; the baseline model most often mistakes its partner for itself (i.e., the model thinks it is talking to itself). Other common failures include the model thinking that it is its partner's character, or emulating irrelevant characteristics.
5.4 Per-Turn Character Accuracy Analysis
We consider the RPA of various models when evaluated across the turns of conversation. Intuitively, baseline models would suffer as the conversation goes on for a variety of reasons (character roles are truncated out of context, more input yields noisier outputs, etc.). Appendix Figure 1 shows the perturn results for a set of representative models. The human outputs are most often correct on the first turn, with gradual RPA decay throughout the conversation. The 128-truncate baseline, as expected, suffers a dramatic performance drop after the first couple of turns. Meanwhile, with the profile expanded attention, we see near-human performance, with better RPA in later turns. Including RPA re-ranking improves dramatically over all turns.
5.5 Expanded Attention Visualization
To gain some insight into what is happening with the expanded attention, we mapped out the attention between context and response tokens for both a baseline model with no expanded attention, and a model with profile expanded attention. Figures 4 and 5 in the Appendix display the heat maps for an example context and response, with details on heat map construction given in Appendix M.
We find that the baseline model spreads its attention out across both the LIGHT context and the dialogue history, with the majority of the attention looking at overlapping words in the context and the response and almost no attention on the character names. The expanded attention model concentrates on the recent dialogue history heavily in the first level of attention, and then concentrates on pertinent words in the context related to the character information (i.e., the character names) in the second round of attention.
In this work we explored the problem of maintaining one's character in open dialogue, and showed that state-art-of-the-art models have a fundamental weakness in this regard. We provided a clear framing of the problem and showed one can build automatic metrics (RPA) that evaluate models using a classifier. We then explored a variety of methods throughout this paper. While a wide variety of well-known techniques, such as multi-objective or unlikelihood training, have little impact, we found that expanded attention and re-ranking are two approaches that can help to a degree, and their combination also improves results. Our introduced method PACER performs well and may be suitable for other tasks beyond the focus of this paper. Nevertheless, our best methods still lag behind human (crowdworker) performance in several regards, e.g.  $1.34\%$  vs.  $2.23\%$  in terms of mistaken identity per turn, or  $5\%$  vs.  $14.7\%$  per conversation. Therefore considerable progress still has to be made on this challenging problem.
Limitations We note in the conclusion that the problem is not solved; our best models still lag behind human performance in maintaining character identity. All results are tested in the LIGHT environment, comprising open-domain dialogue
within constrained settings with assigned characters. The application of these methods to other role-playing (or otherwise) settings is left for future work, though we believe that such methods could be beneficial outside of LIGHT.
Potential Risks We provide methods for mitigating mistaken identity in dialogue models. It follows that such methods yield models that are more convincingly role-playing as a given character. With more convincingly in-character models, someone with bad intentions could have a model imitate real-world people without consent, or worse, can say negative/harmful things while impersonating someone else. We note that our methods are orthogonal to improvements in dialogue safety (Xu et al., 2020; Dinan et al., 2019a), and so can be used in tandem to mitigate these potential risks.
Scientific Artifacts We make use of LIGHT in this work (Urbanek et al., 2019) (released under CC-BY license), an English-language crowdsourced dataset. We also plan to release the code and models (will be released under MIT license), with the intended use being for others (and ourselves) to reproduce and build upon the research discussed in this paper.
Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020. Towards a human-like open-domain chatbot. arXiv preprint arXiv:2001.09977.
Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020. The pushshift reddit dataset. arXiv preprint arXiv:2001.08435.
Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. 2019a. Build it break it fix it for dialogue safety: Robustness from adversarial human attack. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4537-4546, Hong Kong, China. Association for Computational Linguistics.
Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, Shrimai Prabhumoye, Alan W. Black, Alexander Rudnicky, Jason Williams, Joelle Pineau, Mikhail Burtsev, and Jason Weston. 2020. The second conversational intelligence challenge (ConvAI2). In *The
NeurIPS '18 Competition, pages 187-208, Cham. Springer International Publishing.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019b. Wizard of wikipedia: Knowledge-powered conversational agents. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Karthik Gopalakrishnan, Behnam Hedayatnia, Qinglang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, Dilek Hakkani-Tür, and Amazon Alexa AI. 2019. Topical-chat: Towards knowledge-grounded open-domain conversations. In INTERSPEECH, pages 1891-1895.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2020. Poly-encoders: Architectures and pre-training strategies for fast and accurate multi-sentence scoring. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Mojtaba Komeili, Kurt Shuster, and Jason Weston. 2021. Internet-augmented dialogue generation.
Ilia Kulikov, Alexander Miller, Kyunghyun Cho, and Jason Weston. 2019. Importance of search and evaluation strategies in neural dialogue modeling. In Proceedings of the 12th International Conference on Natural Language Generation, pages 76-87, Tokyo, Japan. Association for Computational Linguistics.
Margaret Li, Stephen Roller, Ilia Kulikov, Sean Welleck, Y-Lan Boureau, Kyunghyun Cho, and Jason Weston. 2020. Don't say that! making inconsistent dialogue unlikely with unlikelihood training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4715-4728, Online. Association for Computational Linguistics.
Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.
Tianyu Liu, Yizhe Zhang, Chris Brockett, Yi Mao, Zhifang Sui, Weizhu Chen, and Bill Dolan. 2021. A token-level reference-free hallucination detection benchmark for free-form text generation.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations.
Luca Massarelli, Fabio Petroni, Aleksandra Piktus, Myle Ott, Tim Rocktäschel, Vassilis Plachouras, Fabrizio Silvestri, and Sebastian Riedel. 2020. How decoding strategies affect the verifiability of generated text. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 223-235, Online. Association for Computational Linguistics.
Pierre-Emmanuel Mazare, Samuel Humeau, Martin Raison, and Antoine Bordes. 2018. Training millions of personalized dialogue agents. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2775-2779, Brussels, Belgium. Association for Computational Linguistics.
Alexander Miller, Will Feng, Dhruv Batra, Antoine Bordes, Adam Fisch, Jiasen Lu, Devi Parikh, and Jason Weston. 2017. ParlAI: A dialog research software platform. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 79-84, Copenhagen, Denmark. Association for Computational Linguistics.
Yixin Nie, Mary Williamson, Mohit Bansal, Douwe Kiela, and Jason Weston. 2021. I like fish, especially dolphins: Addressing contradictions in dialogue modeling. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1699-1713, Online. Association for Computational Linguistics.
Shrimai Prabhumoye, Kazuma Hashimoto, Yingbo Zhou, Alan W Black, and Ruslan Salakhutdinov. 2021. Focused attention improves document-grounded generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4274-4287, Online. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8).
Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open-domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5370-5381, Florence, Italy. Association for Computational Linguistics.
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2021. Recipes for building an open-domain chatbot. In Proceedings of the 16th Conference of
the European Chapter of the Association for Computational Linguistics: Main Volume, pages 300-325, Online. Association for Computational Linguistics.
Sashank Santhanam, Behnam Hedayatnia, Spandana Gella, Aishwarya Padmakumar, Seokhwan Kim, Yang Liu, and Dilek Hakkani-Tur. 2021. Rome was built in 1776: A case study on factual correctness in knowledge-grounded response generation.
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. In *Findings of the Association for Computational Linguistics: EMNLP* 2021, pages 3784-3803, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. 2021. Masked language modeling and the distributional hypothesis: Order word matters pre-training for little. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2888-2913, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Jack Urbanek, Angela Fan, Siddharth Karamcheti, Saachi Jain, Samuel Humeau, Emily Dinan, Tim Rocktäschel, Douwe Kiela, Arthur Szlam, and Jason Weston. 2019. Learning to speak and act in a fantasy text adventure game. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 673-683, Hong Kong, China. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Xinyi Wang, Jason Weston, Michael Auli, and Yacine Jernite. 2019. Improving conditioning in context-aware sequence to sequence models.
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2020. Neural text generation with unlikelihood training. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. 2020. Recipes for safety in open-domain chatbots. arXiv preprint arXiv:2010.07079.
Jing Xu, Arthur Szlam, and Jason Weston. 2021. Beyond goldfish memory: Long-term open-domain conversation.