SlowGuess/NAACL_2022_OCR huggingface dataset:
Kevin Yang and Dan Klein. 2021. FUDGE: Controlled text generation with future discriminators. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3511-3535, Online. Association for Computational Linguistics.
Hao-Tong Ye, Kai-Lin Lo, Shang-Yu Su, and Yun-Nung Chen. 2020. Knowledge-grounded response generation with deep attentional latent-variable model. Computer Speech & Language, 63:101069.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204-2213, Melbourne, Australia. Association for Computational Linguistics.
Wen Zheng and Ke Zhou. 2019. Enhancing conversational dialogue models with grounded knowledge. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM '19, page 709-718, New York, NY, USA. Association for Computing Machinery.
Yinhe Zheng, Rongsheng Zhang, Minlie Huang, and Xiaoxi Mao. 2020. A pre-training based personalized dialogue generation model with persona-sparse data. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):9693-9700.
Kangyan Zhou, Shrimai Prabhumoye, and Alan W Black. 2018. A dataset for document grounded conversations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 708-713, Brussels, Belgium. Association for Computational Linguistics.
All models are trained with the  $\mathrm{ParlAI}^2$  framework (Miller et al., 2017). Due to the large number of experimental setups and computational cost, we do not consider multiple training runs.
Base Models RPA classifier Poly-encoders are initialized with the 622M parameter models from Roller et al. (2021); we also use this architecture for dialogue response (retrieval) models which we also evaluate (see Table 19). All generative models are initialized with BlenderBot, also from Roller et al. (2021), a 2.7B parameter transformer encoder/decoder model. Each model was pre-trained on 1.5B training examples from pushshift.io Reddit (Baumgartner et al., 2020), with BlenderBot additionally fine-tuned on the BST tasks (see Roller et al. (2021) for more details), before training on LIGHT.
RPA Classifiers The RPA classifier models are trained with a cross-entropy loss over the correct label, with 99 random negatives chosen from the training set; we ensured that each character in conversation showed up in the set of candidate labels. The models were trained with a batch size of 16 on 432GB GPUs, with early stopping on the validation set according to valid accuracy. We used the Adam optimizer (Kingma and Ba, 2015) with weight decay (Loshchilov and Hutter, 2019), sweeping over learning rates  $\{1e - 5,5e - 6\}$ .
Generative Models All variants of generative models were trained using 832GB GPUs, with early stopping on perplexity on the validation set. We used the Adam optimizer, sweeping over learning rates  $\{1e - 5,7e - 6\}$ , training with a batch size of 128 for the short-truncation models, and 32 for the long-truncation models. For the multi-objective models, we used the same loss (and negative-sampling) setup as the RPA classifiers for the character accuracy objective. During inference, unless otherwise specified, we generated using beam-search with beam size of 10, enforcing a minimum length of 20, and with tri-gram blocking with respect to both the context and the current generation.
B RPA Classifier Training
We build the training data for the RPA classifiers from the LIGHT dataset. The input is a concate
Dataset	Train	Valid	Test
LIGHT (Urbanek et al., 2019)	111k	6k	13k
RPA, 0-Utterance	212k	12k	26k
RPA, 4-Utterance	748k	45k	90k
RPA, All-Utterance	34k	2k	4k
RPA LTR, 0-Utterance	3.3M	205k	414k
RPA LTR, 4-Utterance	12M	747k	1.5M
RPA LTR, All-Utterance	516k	31k	64k
Table 8: Number of training, valid, and test examples for the LIGHT dataset, as well as the RPA training splits (both normal and LTR).
nation of (1) the LIGHT context (set of characters, setting, etc.); (2) a fixed number of previous utterances in the conversation; and (3) a candidate utterance from any point later in the conversation (a special token separates the candidate utterance from the prior context). We experiment with either 0, 4, or  $N - 2$  prior utterances (dubbed "All" in relevant tables), where  $N$  is the total number of utterances ( $N - 2$  allows the last turn for each speaker to be a candidate utterance). The left-to-right (LTR) data split is built similarly, except each example  $i$  becomes  $w_{i}$  examples, where  $w_{i}$  is the number of tokens in the candidate utterance for example  $i$ . Statistics of the training dataset are given in Table 8.
Suppose we choose  $n$  as the number of prior utterances to include in the input, and let us denote  $D = 8538$  to represent all the dialogues in the LIGHT train split, and  $U = 110877$  to represent all the utterances in those dialogues. For the RPA classification dataset, each dialogue is presented twice, once from each character's POV. When  $n = N - 1$ , where  $N$  is the length of a conversation, then we have roughly  $2D$  training examples. When  $n = 0$ , we have roughly  $2U$  training examples.
For any value  $0 < n < N - 1$ , we build out several examples from several slices of each conversation. Suppose we have dialogue  $d_{i}$  with  $N$  utterances  $\{u_0, u_1, \dots, u_N\}$ . To build the training data from dialogue  $d_{i}$ , we select all continuous subsets of  $n$  utterances within  $d_{i}$ , forming contexts
$$
c _ {i} = \left\{u _ {i},..., u _ {i + n} \right\} \quad \forall \quad 0 \leq i \leq N - i
$$
Then, we look at all  $N - i$  utterances following utterance  $u_{i + n}$ , and use these as target utterances in the task. The goal of this is to build the model to be robust to dataset artifacts; without this modification, the model could trivially pick out the character just by looking at the number of alternating utterances.
$^{2}$ https://parl.ai
# Prior Utterances	No LIGHT Context	LIGHT Context
H@1/427	H@1/2	H@1/427	H@1/2
0	10.4	60.3	77.6	77.7
4	87.3	87.4	86.5	86.5
All	85.7	86.7	89.3	89.8
Table 9: RPA classifier performance on the validation set, as measured by Hits@1/427 and Hits@1/2 (all characters and participant characters as candidates, respectively). Each model is trained and evaluated with that # of prior utterances.
# Train-time
Prior Utterances	Hits@1/427
# Eval Prior Utterances
0	4	All
Without LIGHT Context
0	10.35	18.58	17.71
4	2.10	87.31	84.35
All	7.02	81.26	85.70
With LIGHT Context
0	77.64	66.20	58.61
4	31.04	86.48	84.90
All	32.54	82.73	89.26
Table 10: RPA classifier performance on the validation set, as measured by hits@1/427. Highlighted numbers indicate models evaluated on the split on which they were trained.
These measures force the model to fully understand the task and react accordingly.
C RPA Classifier Performance: Additional Results
In Table 10, we see how each RPA classifier performs on the various datasplits, varying the number of prior utterances used during training and evaluation. Each model performs best on the split on which it was trained (the highlighted numbers).
C.1 Left-to-Right Dynamic Classification
We find that the left-to-right RPA classifiers are correctly sensitive to per-token perturbations in the input, and can accurately predict the speaker at the token level. In Table 11, we give an example where the classifier changes its character prediction, depending on the candidate utterance.
D Unlikelihood: Additional Results
In Table 12, we compare UL models across different truncation lengths; the same story applies to the 1024-truncation models. We additionally include a third method, Random-3, where we apply the loss randomly to 3 tokens that yield incorrect RPA classifications. This method performs about the
same as the Top-1 method, but the RPA is lower, indicating that the Top-1 method at least is providing some signal.
E Multi-Objective: Additional Results
E.1 Perplexity & F1
Table 13 displays full PPL and F1 scores corresponding to the models in Table 5.
E.2 Multi-Objective + Automated Grounding
In Table 14, we see how, when using either the encoder+decoder or just the decoder outputs, we do not require additional multi-objective layers (as we did in the non-automated-grounding case).
F Expanded Attention: Additional Results
We provide results for both the 128-truncate and 1024-truncate models with profile grounding in Table 15. Trends remain the same for both models.
G Full Valid Results
Table 17 includes results on the LIGHT validation set for models in Table 4.
H Retrieval Re-rankers
We evaluated a Poly-encoder baseline model with an RPA re-ranker as well. The Poly-encoder scores utterances from the full training set as candidates, and the candidates for re-ranking are the top- $k$  ranked utterances; results are in Table 18. Retrieval models benefit dramatically from the re-ranking, improving to almost  $99\%$  RPA as measured by the LTR classifier. As the candidate responses for retrieval models come from the set of all training utterances, and due to overlap between the set of characters appearing in the train and valid sets, we can examine how often the model output was originally spoken by its partner's character; this can be seen as a proxy for mistaken identity. We find that the re-ranker reduces the amount of time that the model returns a message its partner said, indicating some viable and promising results.
I Full Human Evaluation Results
In Table 19, we display the full results of human evaluations across all dimensions. We note that the Poly-encoder model is best at not mistaking location or being repetitive, but this is expected given its retrieving over human-written utterances. In
Figure 1: Per-turn RPA classifications, for a variety of models. Error bars show the difference between the model's RPA value and the human's RPA value.
Setting: Turquoise Shore, Shore
A beautiful turquoise color water by the shore. It is filled with many gems and gold.
Character 1: Sea Witch. I am a sea witch. I pray on young sailors who hope to find adventure and treasures on the open sea. I lure them in with magic spells and promise of riches.
Character 2: Mermaid. I am one of the most beautiful mermaids to live in the sea. I like to watch the other sea creatures swim by me, including dolphins, who are my favorite creatures because they are so friendly. I fear the people who live on land because they hunt my kind
Classified Utterance: Hey there Mermaid! Long time, no see.
Correct Speaker: Sea Witch	Classified Utterance: Hey there Sea Witch! Long time, no see.
Correct Speaker: Mermaid
Word	Predicted Speaker	Confidence	Word	Predicted Speaker	Confidence
Hey	sea witch	0.5156	Hey	sea witch	0.5156
there	sea witch	0.5467	there	sea witch	0.5467
Mermaid!	sea witch	0.9978	sea	mermaid	0.9968
witch	mermaid	1.000
Long	sea witch	0.9981	Long	mermaid	1.000
time,	sea witch	0.9979	Time	mermaid	1.000
no	sea witch	0.9982	no	mermaid	1.000
see.	sea witch	0.9985	see.	mermaid	1.000
Table 11: Left-to-right dynamic classification examples. A candidate utterance is shown, along with the classifier's predictions at each partial decoded sequence. Left: The true next utterance in the dialogue, with the RPA classifier's predictions and confidence token by token. Right: A perturbed utterance. If we switch the name being addressed, the model switches its predictions immediately.
Unlikelihood Method	PPL	F1	RPA
Human	-	-	92.80
None (128)	12.54	15.80	88.54
None (1024)	12.35	15.85	88.42
128-Truncation
RPA UL: Top-1 Token	13	15.35	88.54
RPA UL: All tokens	12.86	15.28	88.86
RPA UL: Random-3	12.99	15.37	87.85
1024-Truncation
RPA UL: Top-1 Token	12.49	15.66	88.12
RPA UL: All tokens	12.57	15.83	88.06
Table 12: Models trained with unlikelihood loss, evaluated on the valid set. We vary the tokens to which we apply UL loss.
Input	nMO	Stage	PPL	F1	RPA	Hits@1/427
Human	0	-	-	92.80
None	0	0	12.35	15.85	88.42
Dec. Only	0	1	13.22	15.66	89.08	86.37
Dec. Only	0	2	12.92	15.88	89.10	89.10
Enc+Dec	0	1	13.24	15.55	88.83	85.78
Enc+Dec	0	2	13.44	15.61	89.29	89.22
Enc+Dec	2	1	12.94	15.80	88.39	83.25
Enc+Dec	2	2	12.69	15.77	89.05	88.49
Table 14: Models trained with varying multi-objective + automated grounding setups, evaluated on the valid set. The base model in all cases is initialized from a generation model fine-tuned on LIGHT.
Input	nMO	Stage	PPL	F1	RPA	Hits@1
Human	N/A	-	-	92.8
None	0	0	12.4	15.9	88.4
Multi-Objective
Dec. only	2	1	12.4	15.9	88.4	39.3
Dec. only	2	2	12.8	16.0	87.7	87.4
Enc+Dec	2	1	12.4	15.9	88.4	70.9
Enc+Dec	2	2	12.5	15.8	88.8	71.6
Multi-Objective + Automated Expanded Attention
Dec. Only	0	1	13.2	15.7	89.1	86.4
Dec. Only	0	2	12.9	15.9	89.1	89.1
Enc+Dec	2	1	12.9	15.8	88.4	83.3
Enc+Dec	2	2	12.7	15.8	89.1	88.5
Table 13: Models trained with varying multi-objective setups, evaluated on the valid set. Models are initialized from a (1024-truncation) model fine-tuned on LIGHT.
Exp. r	128-Truncate Model	1024-Truncate Model
Attn.	PPL	F1	RPA	PPL	F1	RPA
Human	0	-	-	92.80	-	-	92.80
None	0	12.59	15.80	88.28	12.35	15.85	88.42
ABCD+	1	12.23	15.87	90.59	12.47	15.82	88.34
ABCD	1	12.25	15.97	90.94	12.18	16.01	91.82
ABCD	2	12.23	15.89	90.83	12.17	15.95	92.60
ABCD	3	12.26	15.81	90.44	12.19	15.99	91.73
ABC	1	12.33	15.82	91.50	12.22	15.94	91.83
ABC	2	12.31	16.03	92.03	12.24	15.99	92.24
ABC	3	12.33	15.90	91.59	12.25	15.93	92.25
AB	1	12.42	15.92	90.31	12.27	15.87	90.97
A	1	12.46	16.05	90.22	12.30	15.80	89.13
B	1	12.53	15.85	89.85	12.34	15.76	89.46
Table 15: Models trained with expanded attention (profile grounding), evaluated on the valid set. Expanded attention input: A = Self Persona, B = Self Name, C = Partner Name, D = Setting Description, + = dialogue history. We also vary the number of rounds  $r$  of expanded attention.
Model:	Percentage
Beam Baseline	Delayed Beam Baseline	Delayed Beam with Re-ranker
thinks it is someone else/partner	0%	13.04%	19.23%
Thinks partner's character is its character (i.e., thinks it is talking to itself)	57.69%	56.52%	11.54%
emulates partner's characteristic	0%	4.35%	0%
incorrectly identifies partner	19.23%	17.39%	30.77%
talks about its character in the 3rd person	0%	4.35%	0%
emulates irrelevant characteristic	3.85%	0%	7.69%
combines self and partner persona	7.69%	0%	9.62%
incorrectly identifies 3rd party character	0%	0%	1.92%
claims it does not know who it is	0%	0%	1.92%
noise	11.54%	4.35%	17.31%
Table 16: Turn annotation analysis of RPA Re-rankers.
Model	PPL	F1	RPA
Human	-	-	92.80
Baselines
128-Truncate Vanilla Baseline	12.54	15.80	88.54
1024-Truncate Vanilla Baseline	12.35	15.85	88.42
Re-rankers
128-Truncate Baseline + Re-ranker	-	16.14	92.99
128-Truncate Baseline + PACER	-	16.13	93.31
RPA UL (Top-1 Token)	13.00	15.35	88.54
RPA UL (All Tokens)	12.86	15.28	88.86
Multi-Objective (Vanilla, Dec. Only)	12.78	16.00	87.71
Expanded Attention Methods
Profile Grounding (128, 2 Rounds over ABC)	12.31	16.03	92.03
Profile Grounding (1024, 2 Rounds over ABCD)	12.17	15.95	92.60
Automated Grounding (1024, Classifier Attn.)	12.19	15.90	91.11
Automated Grounding + MO (1024 Dec. Only)	12.92	15.88	89.10
Expanded Attention + Re-ranker Methods
Profile (128) + RPA Re-ranker	-	16.21	95.62
Profile (128) + PACER	-	16.18	95.82
Table 17: Validation statistics for various models considered throughout the paper.
Metric	Baseline	Re-ranker
RPA (normal)	85.47	94.29
RPA (LTR)	86.31	99.76
% Partner Said Response	3.20	2.02
Table 18: Retrieval models with character output rankers; performance on the validation set.
Figure 2, we show a screenshot of the instructions for the evaluation task provided to crowdworkers on Amazon Mechanical Turk.
J Generation Settings
J.1 Test Output Analysis
We provide qualitative analysis of the various generation methods below.
No Re-ranking When examining the baseline with no re-ranking, we found that nucleus sampling can help when beam search does not work; however, both can go out of character the farther one goes in conversation.
Beam Search Re-rankers The beam outputs in standard beam search are at times too similar, in
which case re-ranking does next to nothing, unless a viable response is available.
Nucleus Sampling Re-rankers Using nucleus setting in a re-ranking setup yields more diverse choices to choose from; however, sometimes the model simply does not address *any* character within the conversation.
Delayed Beam Search Re-rankers This strikes a nice balance between sensible outputs from beam search and diversity from nucleus sampling.
Mixed-Decoding Re-ranker Using mixed decoding (re-ranking several decoding schemes) can work quite well, as it is a nice blend of different generation methods.
J.1.1 Turn Annotation Analysis
Qualitative analysis of the turn annotation results are in Table 16. We generally found that beam search fails the vast majority of the time when the model thinks that it is talking to itself; i.e., it confuses its partner for its own character. The re-rankers can help shift the hallucination away from this regime.
(You can keep accepting new HITs after you finish your current one, so keep working on these if you like the task!)
You will have a natural conversation with a partner, and you will also evaluate your partner's responses for conversational problems such as contradiction, repetition, and others.
You will be given a location and setting from a fantasy text adventure. Your partner also has a character. In this chat, have a conversation with your partner pretending to be your given character. Some ideas of what to talk about:
- Talk about the location you and your partner are in
- Discuss elements of yourself as your given character
- Ask your partner about themselves
Possible labels:
- Contradiction: your partner says something that contradicts something they've said before
- Mistaken Identity: your partner says something that would imply they believe they're someone other than who they're noted to be
- Wrong Location: your partner says something that would imply they believe they are in a different location than the provided one
- Unrelated: Your partner says something that doesn't follow the previous turns.
- Repetitive: Your partner says something they've already said, or are driving the conversation in circles.
- Have a conversation: do not trivially copy your partner or ignore them.
The conversation will continue for at least 7 turns for each partner, and then you or your partner can end the conversation when you wish.
There is a 3-minute time limit for each turn.
- Please do not send messages that are either too short or too long (messages cannot exceed 30 words).
- Please do not reference the task or MTurk, HITs, Requestors, or other Mechanical Turk specific vocabulary itself during the conversation.
Note that the user who you are chatting with may be either a human or a bot.
- Keep in mind that the conversations will eventually be made public, so act as you would on a public social network (e.g. Twitter).
- No racism, sexism or otherwise offensive comments, or the submission will be rejected and we will report to Amazon.
m  t   t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t
Figure 2: Instructions provided to annotators in human evaluations.
Model	Contradiction	Mistaken Identity	Mistaken Location	Off-Topic	Repetitive	All-Good	Clean Convo	Mistaken Identity In Convo	Avg. Engagingness
Human	-	1.34%	-	-	-	-	-	5%	-
Baselines
Poly-Encoder	5.50%	6.14%	0.77%	12.02%	1.92%	75.45%	16.33%	34.69%	3.42
128-Truncate Vanilla Baseline	8.26%	6.45%	2.71%	4.26%	4.00%	76.00%	26.80%	35.05%	4.04
1024-Truncate Vanilla Baseline	7.48%	7.35%	2.66%	6.21%	4.31%	75.03%	22.22%	38.38%	4.16
Re-rankers
128-Truncate Baseline + RPA Re-ranker (Beam)	4.83%	5.56%	3.62%	4.35%	3.26%	80.31%	20.19%	34.65%	4.14
128-Truncate Baseline + RPA Re-ranker (Nucleus)	9.07%	8.68%	2.33%	5.31%	3.89%	73.70%	31.96%	37.11%	3.83
1024-Truncate Baseline + RPA Re-ranker (Beam)	5.55%	4.81%	1.60%	3.45%	2.71%	82.98%	33.33%	24.45%	4.14
128-Truncate Baseline + PACER	8.28%	4.27%	4.89%	3.14%	3.14%	73.90%	21.78%	33.66%	3.96
1024-Truncate Baseline + PACER	7.63%	7.13%	2.38%	3.63%	3.75%	79.25%	28.00%	36.00%	4.18
Modified Training Objectives
RPA Unlikelihood (Top-1 Token)	8.70%	7.13%	3.38%	7.25%	3.74%	72.83%	14.42%	39.40%	3.87
RPA Unlikelihood (All Tokens)	11.64%	10.51%	3.13%	4.88%	5.38%	67.71%	19.00%	43.00%	3.87
Multi-Objective (Vanilla, Dec-Only)	8.13%	10.00%	1.88%	5.63%	2.63%	74.75%	18.00%	49.00%	4.21
Expanded Attention Methods
Profile Grounding (128, 2 Rounds over ABC)	5.32%	4.82%	3.21%	4.45%	2.84%	81.58%	27.45%	28.43%	4.18
Profile Grounding (1024, 2 Rounds over ABCD)	4.13%	4.00%	3.38%	3.13%	3.25%	83.75%	36.63%	23.76%	4.34
Automated Grounding (Classifier Attn.)	10.17%	5.51%	2.57%	6.13%	2.33%	75.98%	24.27%	29.13%	4.04
Automated Grounding + MO (Dec. Only)	8.23%	4.43%	2.03%	3.80%	5.19%	78.61%	38.00%	23.00%	4.12
Expanded Attention + Re-ranker Methods
Profile Grounding (128) + RPA Re-ranker	5.33%	2.23%	1.61%	4.22%	2.98%	84.37%	36.27%	14.71%	4.25
Profile Grounding (1024) + RPA Re-ranker	6.00%	3.60%	1.20%	0.42%	0.30%	85.25%	40.00%	21.90%	4.35
Profile Grounding (128) + PACER	5.43%	4.07%	2.84%	2.47%	1.23%	85.70%	41.18%	24.51%	4.32
Profile Grounding (1024) + PACER	6.21%	4.38%	1.10%	3.65%	2.56%	83.56%	40.78%	22.33%	4.13
Table 19: Human evaluations. Annotators chatting with models were asked to annotate whether model utterances contained any of the problem attributes listed, with "All-Good" indicating that there were no issues. "Clean Convo" is the percentage of conversations without any issues.
J.2 Automated Metrics
We experiment with various generation settings, with or without re-rankers; results are in Table 20. For the baseline and re-ranker models, beam search yields the highest F1 scores; RPA can be improved with the other inference methods when combined with a re-ranker. We believe this may be due to the higher diversity of candidate responses generated from those methods.
K Human + Automatic Eval Correlation
We analyze the correlation between human annotations and the automatic metrics collected on the LIGHT validation set, as shown in Figure 3; we note some interesting trends:
Perplexity perplexity appears to be positively correlated with mistaken identity, and negatively correlated with engagingness. So, perplexity is
Generation Setting	Normal	Re-ranking
F1	RPA	F1	RPA
Human	-	92.80	-	-
128-Truncation Model
Beam Search	15.80	88.54	16.14	92.99
Delayed Beam Search	15.46	88.74	15.48	93.18
Nucleus Sampling	15.70	89.25	15.44	97.12
Top-K Sampling	14.47	88.16	14.14	97.01
1024-Truncation Model
Beam Search	15.85	88.42	16.08	92.92
Delayed Beam Search	15.03	88.00	15.39	92.89
Nucleus Sampling	15.42	88.22	15.25	97.24
Top-K Sampling	14.45	86.91	14.06	97.15
Table 20: Performance on the LIGHT valid set for the baseline models with different generation settings, with or without re-rankers. All settings use tri-gram blocking with respect to the context and current generation, and have a minimum length of 20. We set  $topp = 0.3$  for Nucleus sampling,  $topk = 50$  for Top-K sampling, and use a beam-delay of 10 with  $topk = 10$  for delayed beam search.
Figure 3: Correlation between human evaluations and automated metrics computed on the test set.
a good indicator of how fluent and engaging the model is in conversation, and can indirectly point to a better understanding of the role-playing task. An important note is that we only tested this amongst models of the same size, and only for the models we tested, so it is not clear that larger models will necessarily bring improvements.
F1 F1 word overlap is positively correlated with engagingness as well, so F1 may be a good proxy of model performance. Correlation with mistaken identity is negative here, implying that better F1 corresponds with better role-playing ability. However, we note that F1 is not a catch-all metric (Liu et al., 2016).