SlowGuess/NAACL_2022_OCR huggingface dataset:
Kurt Shuster Jack Urbanek Arthur Szlam Jason Weston
State-of-the-art dialogue models still often stumble with regards to factual accuracy and self-contradiction. Anecdotally, they have been observed to fail to maintain character identity throughout discourse; and more specifically, may take on the role of their interlocutor. In this work we formalize and quantify this deficiency, and show experimentally through human evaluations that this is indeed a problem. In contrast, we show that discriminative models trained specifically to recognize who is speaking can perform well; and further, these can be used as automated metrics. Finally, we evaluate a wide variety of mitigation methods, including changes to model architecture, training protocol, and decoding strategy. Our best models reduce mistaken identity issues by nearly  $65\%$  according to human annotators, while simultaneously improving engagingness. Despite these results, we find that maintaining character identity still remains a challenging problem.
The exchange of stories from one's past, or descriptions of activities in one's present, are a fundamental part of human discourse. Trustworthy human conversationalists keep their stories roughly straight within a conversation. An interlocutor taking on your own stories and persona as theirs is especially jarring and unnatural. However, despite the improvements in state-of-the-art open-domain dialogue modeling, both in terms of distributional accuracy metrics like perplexity, and subjectively in terms of human judgements (Adiwardana et al., 2020; Roller et al., 2021), interactions with those agents reveal that they cannot keep their stories straight. In particular, they are likely to take on the role of their interlocutor; for example, if an agent's partner says they are a software engineer, the agent is likely to say it is a software engineer too (Roller et al., 2021), or worse, appropriate their partners just told tale of a trip to NAACL as their own. Some
Setting: Barn, Farm
Partner: Thief Self: Guest
Guest: Hail who goes there?!
Thief: Hello friend I mean you no harm
Model Response
Guest: I am not your friend, I am your thief! I am here to steal from you!
Character Token Probabilities
Thief: 0.39 Guest: 0.23
Setting: Castle Kitchens, Inside Palace
Partner: Hunter Self: Helper
Hunter: Hi
Helper: Hello. What brings you to the kitchen?
Hunter: I am chopping some meat that is to be sold.
Helper: Could you spare me a piece? I'm starving.
Hunter: It's 3 cents a pound, if you have 3 cents then yeah
Model Response
Helper: I don't have any money. I'm just a poor hunter trying to make a living.
Character Token Probabilities
Hunter: 0.39 Helper: 0.12
Table 1: Failures of a 2.7B parameter transformer pretrained on 1.5B examples from a large dialogue corpus (Roller et al., 2021) and fine-tuned on LIGHT (Urbanek et al., 2019). The bold words in red highlight the model mistaking its identity for its partner's. (Top) The model believes it is a thief, rather than a guest. (Bottom) The model believes it is a hunter rather than a helper. Token probabilities are given at the position of the mistake for the two names.
example failure cases are given in Table 1, where models incorrectly take on the name, role or activities of their partner instead of their assigned role. These failures are related to the general problems of repetition in language models (Holtzman et al., 2020), the weak influence of word order (Sinha et al., 2021) and inability to avoid contradictions (Nie et al., 2021).
In this work we formalize and quantify this behavior, show that to some extent it can be detected automatically with a specifically trained classifier, and then study a wide variety of mitigations. These include multi-objective training, likelihood training (Li et al., 2020; Welleck et al., 2020),
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2367 - 2387 July 10-15, 2022 ©2022 Association for Computational Linguistics
classifier-assisted re-ranking based generation, and several forms modifying the attention mechanisms of the decoder in a sequence to sequence model. Our best methods can reduce mistaken identity issues by  $65\%$ , while simultaneously improving inconversation engagingness; indeed, our models that can stick to their role in conversation are judged by humans to be significantly more engaging than their baseline counterparts. Despite these advances, we find that there is still considerable space to improve these results further in future work.
We make publicly available both our trained models and code to reproduce results<sup>1</sup>.
Role-Playing in Open-Domain Dialogue Much recent work has explored training open-domain dialogue models on large and small dialogue corpora, with the former imbuing raw conversational ability and the latter providing necessary conversational skills. Most crowd-sourced datasets require acting out a role to some capacity in conversation (though indeed Mazaré et al. (2018) study extraction of roles from raw data). Some involve providing persona lines that a model must assume throughout the conversation (Zhang et al., 2018; Dinan et al., 2020; Xu et al., 2021); others require more subtle "roles", such as a listener (Rashkin et al., 2019), or a teacher and student (Dinan et al., 2019b; Gopalakrishnan et al., 2019; Zhou et al., 2018; Komeili et al., 2021). Zheng et al. (2020) explore using a discriminative model to predict whether model responses contain similarity with their persona, similar to methods we employ in our work.
Consistency in Open-Domain Dialogue A common paradigm in the state of the art of open-domain dialogue involves concatenating all relevant contextual information as input to a sequence to sequence neural model (e.g., transformers (Vaswani et al., 2017)) to obtain a conditioned response. Such models can yield human-like and engaging responses (Adiwardana et al., 2020; Roller et al., 2021). Nevertheless, various consistency issues still plague such models. Recent studies have indicated that hallucination of incorrect knowledge is still far from a solved issue (Shuster et al., 2021; Santhanam et al., 2021), with some proposing specific datasets and tools for measuring precisely the levels of this undesired attribute (Liu et al., 2021). Another clear
example of failure is the short-term memory of state-of-the-art models (Xu et al., 2021), sometimes due to the lack of long-form training data or long-context models but often due to simply the modeling itself.
To address consistency issues, a variety of methods have been explored. In the context of knowledge-grounded dialogue, different ways to attend most effectively over provided contextual information have been explored (Zheng and Zhou, 2019; Ye et al., 2020; Prabhumoye et al., 2021; Wang et al., 2019). These works find that considering factual documents separately (in some capacity) improves model grounding. We explore such methods, but in the context of character identity.
Another general problem is that of contradictions. Nie et al. (2021) collect a dataset of contradictions in dialogue, and train classifiers that help re-rank model outputs at inference time; Li et al. (2020) explore unlikelihood training (Welleck et al., 2020) to reduce repetition and contradiction, among other undesired traits, in model generations. The character identity issue we study in this work can be seen as an important class of contradictions, but to the best of our knowledge, has not been explicitly focused on.
We consider a two-party chat setting. The context provided to a model includes: (i) the name of its character and the partner's character; (ii) an extended description of its own character; (iii) and, information about the area in which the conversation takes place. The responsibility of the model is to engage its conversational partner, with no other goal prescribed; however, it should stay within character and within the bounds of the defined setting.
We operate in the context of LIGHT (Urbanek et al., 2019), consisting of grounded fantasy roleplaying game conversations. The LIGHT environment involves humans and models interacting with thousands of objects in hundreds of locations, all while assuming the roles of one of hundreds of characters. The dataset consists of roughly 8.5k dialogues spanning 111k utterances. It is an ideal setting for this study because of the rich and varied personas with explicit backstories.
To quantify the character identity problem, we take a state-of-the-art dialogue agent (specifically, BlenderBot (Roller et al., 2021)) fine-tuned on the
1URL will appear here.
LIGHT dialogue dataset and ask human annotators if the agent mistakes its identity based on its utterances in context. The agent conditions its response on the LIGHT context and prior utterances in the dialogue history. We see in Table 4 that in roughly 6.5 percent of utterances the model mistakes its identity; this corresponds to a mistake in approximately 35 percent of conversations.
BlenderBot uses a Byte-Level BPE tokenizer (Radford et al., 2019); an artifact from the BlenderBot pre-training is that it only considers 128 such tokens in the past, and thus has no mechanism for recovering truncated information about the LIGHT context in later conversational turns. Our second baseline lengthens the input context to 1024 BPE tokens, which allows the entire context for every example to fit into the truncation length of the model; we follow methods employed in Xu et al. (2021) to extend the positional embeddings of the model. We see in Table 4 that this actually makes the problem worse, resulting in 7.4 percent of utterances with mistaken identity (corresponding to a failure in approximately 38 percent of conversations).
3.2 Measuring Role-Playing Accuracy: RPA
We first define a metric, role-playing accuracy (RPA), to denote how often a model's responses are "in-character"; by this, we mean how often the model's response could feasibly be said by their character, given their assigned character identity. Measuring RPA is a non-trivial task for a variety of reasons. First, some conversations involve pairs that can reasonably say similar things (priest vs. priestess, man vs. woman, wizard vs. witch). Second, opening lines are often more generic ("hello", "how fare your travels today"), so either character can say it in conversation. The third reason stems from the data that we study; we are relying on crowdsourced data in which humans are required to portray their characters. Some crowdworkers may be better than others, and there may be some noise in the dataset in which, e.g., a horse may proclaim its love for a queen, or a knight may discuss at length the kingdom's tax collecting.
Given the difficulties above, our primary measure of RPA involves human annotation of model responses, specifically evaluating whether a candidate response fits a given model's character. We thus have human crowdworkers chat with each model in a LIGHT setting; each is given a character and asked to role-play, while the human an
notates each model response, determining whether the model is in character: we denote this metric as "Mistaken Identity" in our experiments, and other utterance-level annotations are collected. Further details regarding human evaluation are outlined in Section 4.7.
Despite the efficacy of human evaluation, it is both costly and slow; as a proxy, we thus train models specifically designed to identify whether a candidate response from a model fits the model's role, and denote these as "RPA Classifiers". We employ poly-encoder transformers (Humeau et al., 2020) to learn this metric, and structure the task as a ranking one; the model receives the LIGHT setting and prior utterances of dialogue as input, as well as the response currently under consideration, and the model must choose the correct character from a fixed set of candidates. We also explore RPA classifiers trained on all partially complete sequences of labels, such that the classifiers can determine the character speaking without requiring the full utterance; we call these left-to-right (LTR) RPA classifiers. Further details about how our RPA classifiers are built are given in Appendix B.
3.3 Mitigations
In this section we describe several strategies for improving the role-playing accuracy of dialogue agents, specifically ways to improve our transformer baselines.
3.3.1 Re-ranking Model Outputs via RPA
We can employ an RPA classifier in response generation by using it to rank candidate model outputs.
Utterance Re-ranking: Given a set of candidate responses, the RPA classifier can re-score the set and return the response yielding the highest probability of staying in character (according to the RPA score on the complete candidate generations). The dialogue models employ beam-search to generate responses, and the candidates for re-ranking are the beams within beam-search. We also try nucleus sampling (Holtzman et al., 2020) and delayed beam-search (Massarelli et al., 2020) to see whether more diverse candidates have any effect.
Partial And Complete Efficient Re-ranking (PACER): Re-ranking only the final beam candidates may be suboptimal because it is well known that those candidates are not very diverse (Kulikov et al., 2019), meaning there may not be any good candidates to choose from in this final set. In order
to generate utterances that agree with our classifiers, a possible improvement is to generate the utterance such that partial generations also agree with the classifier when generating left-to-right, ensuring that good candidates are surfaced. With access to LTR RPA classifiers, we can apply re-ranking to partial sequences.
Unfortunately, re-ranking at every step of beam search, for every token, requires significant computation, such as in the recent FUDGE method (Yang and Klein, 2021). FUDGE re-scores tokens at each decoding step by multiplying the classifier probability with each token probability, and renormalizing, which is used for control tasks with lightweight classifiers in order to be tractable.
In our proposed approach, called PACER, we re-score candidate tokens, for each beam, according to the probability that their inclusion yields the appropriate character classification, and then finally re-rank the complete candidate beams. To make this efficient, we crucially score only a small proportion of decoding steps (e.g.,  $5\%$  of token positions) as well as for only a few candidate rescored tokens (e.g., top 10 only). We can control these hyperparameters to explore the speed vs. accuracy trade-off.
3.3.2 Unlikelihood
We explore utilizing an unlikelihood (UL) loss (Li et al., 2020; Welleck et al., 2020) to force the model to stay in character during training. Unlikelihood training works as a counter to the standard maximum likelihood (MLE) training of language models; while MLE training pushes the model to generate the correct tokens, UL training pushes the model to not generate incorrect tokens.
While training on the LIGHT dataset with standard NLL loss, with some fixed probability we consider a candidate model generation for UL loss. The full generation is sent to the RPA classifier; if the generation is classified as coming from the incorrect character, we examine each partial generated sequence of the output, and send these sequences to the LTR RPA classifier to determine whether the candidate partial sequences match the model's character. We apply UL loss to tokens that yield the wrong character classification.
3.3.3 Multi-objective Training
The RPA classifiers utilize the LIGHT setting and prior utterances of dialogue history to determine which character generates a candidate response.
We hypothesize that the generation models themselves should be able to pick out and utilize these components as well. However, the RPA classifier models are trained explicitly for this task, whereas the seq2seq models are trained only to generate a plausible continuation of a dialogue history.
We thus explore a setup in which the generation models are trained to identify the speaker of an utterance as well. To do this, we use the output representations from the model (either encoder + decoder, or decoder only) as inputs to  $n_{MO}$  additional transformer layers, where we vary  $n_{MO} \in \{0,2\}$ . The final outputs are used to compute a character score, similarly to the RPA classifier.
The model can then be trained piece-wise. After initializing the model weights with those trained on the LIGHT response generation task, we then train only the extra layers with only the character classification objective; once the classifier achieves suitable performance on the task, we can begin to back-propagate the character classification objective multi-tasking with the dialogue task itself to the generation model directly, in the hope that the model learns to update its internal representations of the context and/or the decoded response.
3.3.4 Expanded Decoder Attention
Maintaining identity relies on the model's capacity to understand which inputs from the conversational history are pertinent when generating a continuation of the preceding dialogue. In a standard, open-domain chit-chat scenario, the model has free reign to decide which elements of the context it would like to condition on when generating a response, as we are dealing with a nearly unconstrained output space (so long as the output follows plausibly from the input). In LIGHT, however, we want to emphasize certain components of the context more so than others; specifically, when role-playing as a character, we want the model to always be reminded of its role, so that it can conditionally generate an optimal response while staying in character. In this lens, one can view the task as "grounding" on one's character information when conversing.
Profile Grounding Inspired by models demonstrating good performance in knowledge-grounded dialogue (Zheng and Zhou, 2019; Ye et al., 2020; Prabhumoye et al., 2021; Wang et al., 2019), we propose a simple extension to the transformer seq2seq architecture, specifically the decoder, to ensure the model knows to condition on the pro
file. The standard transformer decoder first uses self-attention over the decoded response, and then cross-attention over the encoder outputs. We add a third attention step, expanded attention, that attends again over an extracted subset of the input context (encoded separately from the normal context). We explore various subsets of the context to determine which are most important for both RPA and other automated metrics, and call this method "Profile" grounding as the subsets generally include the character and role description. We utilize the exact same (shared) parameters for both the normal cross-attention and the expanded attention; thus, model size is not affected.
Automated Grounding Instead of directly telling the model what to re-attend to, we also explore whether the model can learn to do this automatically, based on its own (or other) representations of the context. The first method we consider is examining the decoder attention weights. Specifically, we use the attention weights from the decoder over the full context to choose  $k$  tokens to re-attend to. This operation is done on a per-layer basis, and thus allows different decoder layers to re-attend to (potentially different) components of the input.
The second method we consider is a trainable mask; this involves feeding the encoded context through a "mask" layer to select various tokens to re-attend to. Specifically, we feed the context through a linear projection layer followed by a softmax to select the top- $k$  tokens. This set of tokens is then re-encoded by the encoder and fed to the decoder as the expanded attention context.
Finally, we explore using the classifier attention weights over the context from the RPA classifier itself. Intuitively, the RPA classifier has learned what components of the input are necessary for determining which character is speaking; if we look at these attention weights when considering the model's character, we know what the classifier thinks is important to use.
Combined Methods We also consider combining expanded attention with re-ranking methods, or with multi-objective training, to see if the combination can improve results. For the latter we use the automated grounding trainable mask method.
Train split	Ranking Accuracy (Hits@1/427)
# Eval Contextual Utterances
Full Datasplit	LTR Datasplit
0	4	All	0	4	All
LTR	64.8	84.3	83.9	61.7	80.5	80.5
Full	31.0	86.5	84.9	27.8	75.3	74.9
Table 2: RPA classifier performance on the validation set, comparing a partial-sequence trained model ("LTR") to one trained only on full sequences ("Full"). Models were trained with 4 prior utterances of context.
Re-ranker	Params	F1	RPA	Cost
# Tokens	Freq.
None	0	0	15.8	88.4	1x
Complete-only	0	0	16.0	93.0	1.1x
Partial-only	10	5%	15.9	88.6	1.3x
Partial-only	10	33%	158	91.1	4.2x
Partial-only	10	100%	15.6	93.6	11.2x
Partial-only	50	5%	15.9	88.9	3.0x
PACER	10	5%	16.1	93.3	1.2x
PACER	10	33%	15.9	94.6	3.8x
PACER	10	100%	15.8	96.3	11.5x
Table 3: Models (128-truncated) evaluated with various re-ranking schemes on the validation set. Cost is relative speed compared to no re-ranking at all.
4.1 RPA Classifiers
We first assess the quality of our RPA classifiers. We measure hits@1/427, where the model must correctly identify the character speaking out of 427 characters from the validation set, comparing the standard and left-to-right (LTR) models in Table 2. We experiment with either 0, 4, or All prior context utterances. The LTR classifiers perform nearly as well as the full classifiers on the full datasplit, and outperform them on the LTR split. Given the robustness of the LTR RPA classifiers, we use this model for computing RPA throughout the remaining results, unless otherwise specified. Further results are given in Appendix Table 10.
4.2 Baseline Generation Performance
We next train baseline models for the dialogue generation task itself. Performance on the LIGHT dataset test split for our baseline models can be found in Table 4, with results on the validation set in Table 17 in the Appendix. While lengthening the context from 128 to 1024 tokens yields better perplexity, the model obtains worse F1 scores and does not improve much if at all on role playing ability, both when measured by the RPA classifiers and in human evaluations (see also Table 19). Further
Model	Automatic Metrics	Human Evaluations
PPL↓	F1↑	RPA↑	Mistaken Identity↓	All-Good↑	Mis. Id. in Conv.↓	Engaging↑
Human	-	-	92.68	1.34%	-	5.0%	-
Baselines
128-Truncate Vanilla Baseline	12.64	15.69	87.61	6.45%	76.0%	35.1%	4.04
1024-Truncate Vanilla Baseline	12.43	15.68	87.71	7.35%	75.0%	38.4%	4.16
Re-rankers
128-Truncate Baseline + RPA Re-ranker	-	15.87	92.09	5.56%	80.3%	34.7%	4.14
128-Truncate Baseline + PACER	-	15.85	92.78	4.27%	73.9%	33.7%	3.96
Modified Training Objectives
RPA Unlikelihood (Top-1 Token)	13.10	15.18	87.48	7.13%	72.8%	39.4%	3.87
RPA Unlikelihood (All Tokens)	13.31	14.77	88.07	10.51%	67.7%	43.0%	3.87
Multi-Objective (Vanilla, Dec. Only)	12.86	15.67	87.67	10.00%	74.8%	49.0%	4.21
Expanded Attention Methods
Profile (128, 2 rounds over ABC)	12.37	15.74	91.70	4.82%	81.6%	28.4%	4.18
Profile (1024, 2 rounds over ABCD)	12.23	15.66	92.18	4.00%	83.8%	23.8%	4.34
Automated (1024, Classifier Attn)	12.27	15.75	90.93	5.51%	76.0%	29.1%	4.04
Automated + MO (1024, Dec. Only)	13.01	15.52	88.95	4.43%	78.6%	23.0%	4.12
Expanded Attention + Re-ranker Methods
Profile (128) + RPA Re-ranker	-	15.88	95.16	2.23%	84.4%	14.7%	4.24
Profile (128) + PACER	-	15.79	95.31	4.07%	85.7%	24.5%	4.32
Table 4: Automated metrics and human evaluations for various models considered throughout the paper on the LIGHT test set. RPA (Role-Playing Accuracy) is measured by the 4-utterance LTR classifier, see Sec. 3.2. The human evaluations are per utterance, except for Engaging and Mistaken Identity in Conversation (with the latter indicating % of conversations with mistaken identity).
detailed training and optimization specifications are given in Appendix A.
4.3 RPA Re-ranker Performance