SlowGuess/ACL_2020_OCR huggingface dataset:
Ryan Lowe, Michael Noseworthy, Iulian V. Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017. Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses. arXiv.
A. H. Miller, W. Feng, A. Fisch, J. Lu, D. Batra, A. Bordes, D. Parikh, and J. Weston. Parlai: A dialog research software platform. arXiv.
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. In Proceedings of the Third Conference on Machine Translation (WMT).
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of ACL.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv.
Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. 2019. What makes a good conversation? How controllable attributes affect human judgments. arXiv.
Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2017. A hierarchical latent variable encoder-decoder model for generating dialogues. In Proceedings of AAAI.
Chongyang Tao, Lili Mou, Dongyan Zhao, and Rui Yan. 2017. RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems. arXiv.
Jason Weston, Emily Dinan, and Alexander H Miller. 2018. Retrieve and refine: Improved sequence generation models for dialogue. arXiv.
John Wieting, Taylor Berg-Kirkpatrick, Kevin Gimpel, and Graham Neubig. 2019. Beyond BLEU: Training Neural Machine Translation with Semantic Similarity. In Proceedings of ACL, Florence, Italy.
Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017. A broad-coverage challenge corpus for sentence understanding through inference. arXiv.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. arXiv.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? arXiv.
A Temporal Structure
We hypothesize that a good encoding function can capture the structure that exists in dialogue. Often this translates to capturing the semantics, coherency in dialogue which are some of the key attributes of a conversation. Formally, we propose using a function  $f_{t}^{D_{i}}$  which maps one utterance to the next.
$$
\mathbf {h} _ {u _ {i + 1}} = f _ {t} ^ {D _ {i}} \left(\mathbf {h} _ {u _ {i}}\right) \tag {5}
$$
To define a good encoding function, we turn to pre-trained language models. These models are typically trained on large corpus and achieve state-of-the-art results on a range of language understanding tasks (Ott et al., 2018). To validate our hypothesis, we use a pre-trained (and fine-tuned) BERT (Devlin et al., 2018) as  $f_{e}$ . We compute  $h_{u_i} = f_e(u_i)\forall u_i\in D$ , and learn a linear classifier to predict an approximate position of the  $u_{i}\in D_{i}$ . The task has details in its design, in the case of goal-oriented dialogues the vocabulary differs in different parts of the conversation and in chitchat dialogues it cannot be said. To experiment, we choose PersonaChat (Zhang et al., 2018) and DailyDialog (Li et al., 2017) to be nominal of chit-chat style data, and Frames (Asri et al., 2017) and MultiWOZ (Budzianowski et al., 2018) for goal-oriented data.
We encode every consecutive pairs of the utterances with a  $\%$  score,  $t$ , that denotes its occurrence after the completion of  $t\%$  of dialogue.
$$
t _ {u _ {p}} = \frac {\operatorname {i n d e x} _ {u _ {p}} + 1}{k} \tag {6}
$$
where  $\mathrm{index}_{u_p}$  denote the average of the indices in the pair of the utterances and  $k$  denote the total number of utterances in dialogue.
Now, we pre-define the number of bins  $B$ . We split the range 0-100 into  $B$  non-overlapping sets (every set will have min and max denoted by  $s_{min}^{i}$  and  $s_{max}^{i}$  respectively). We parse every dialogue in the dataset, and place the encoding of every utterance pair in the corresponding bin.
$$
b i n _ {u _ {p}} = \left\{i \mid t _ {u _ {p}} > s _ {m i n} ^ {i} \& s _ {m a x} ^ {i} > t _ {u _ {p}} \right\} \quad (7)
$$
We then use Linear Discriminant Analysis (LDA) to predict the bin of each utterance  $u_{i}$  in the dialogue after converting the high dimensional embedding into 2 dimensions. LDA provides the
best possible class conditioned representation of data. This gives us a downsampled representation of each utterance  $u_{i}$  which we plot as shown in Figure 3. The reduction on BERT encoding to 2-dimensions shows that BERT is useful in nudging the encoded utterances towards useful structures. We see well defined clusters in goal-oriented but not-so-well-defined clusters in open domain dialogues. This is reasonable to expect and intuitive.
B Generalization on unseen dialog datasets
In order for a dialogue evaluation metric to be useful, one has to evaluate how it generalizes to unseen data. We performed the evaluation using our trained models on PersonaChat dataset, and then evaluated them zero-shot on two goal-oriented datasets, Frames (Asri et al., 2017) and MultiWoz (Budzianowski et al., 2018), and one chit-chat style dataset: Daily Dialog (Li et al., 2017) (Table 3). We find BERT-based models are significantly better at generalization than InferSent or RUBER, with MAUDE marginally better than DistilBERT-NLI baseline. MAUDE has the biggest impact on generalization to DailyDialog dataset, which suggests that it captures the commonalities of chit-chat style dialogue from PersonaChat. Surprisingly, generalization gets significantly better of BERT-based models on goal-oriented datasets as well. This suggests that irrespective of the nature of dialogue, pre-training helps because it contains the information common to English language lexical items.
C Noise Contrastive Estimation training ablations
The choice of negative samples (Section 3) for Noise Contrastive Estimation can have a large impact on the test-time scores of the metrics. In this section, we show the effect when we train only using syntactic negative samples (Table 4) and only semantic negative samples (Table 5). For comparison, we show the full results when trained using both of the sampling scheme in Table 6. We find overall training only using either syntactic or semantic negative samples achieve less  $\Delta$  than training using both of the schemes. All models achieve high scores on the semantic positive samples when only trained with syntactical adversaries. However, training only with syntactical negative samples results in adverse effect on detecting semantic negative items.
Datasets	DailyDialog	Frames	MultiWOZ
Model	Eval Mode	Score	Δ	Score	Δ	Score	Δ
RUBER	+	0.173 ±0.168	0.211 ±0.172	0.253 ±0.177
-	0.063 ±0.092	0.11	0.102 ±0.114	0.109	0.121 ±0.123	0.123
InferSent	+	0.163 ±0.184	0.215 ±0.186	0.277 ±0.200
-	0.050 ±0.085	0.113	0.109 ±0.128	0.106	0.127 ±0.133	0.15
DistilBERT NLI	+	0.885 ±0.166	0.744 ±0.203	0.840 ±0.189
-	0.575 ±0.316	0.31	0.538 ±0.330	0.206	0.566 ±0.333	0.274
MAUDE	+	0.782 ±0.248	0.661 ±0.293	0.758 ±0.265
-	0.431 ±0.300	0.351	0.454 ±0.358	0.207	0.483 ±0.345	0.275
Table 3: Zero-shot generalization results on DailyDialog, Frames and MultiWOZ dataset for the baselines and MAUDE. + denotes semantic positive responses, and - denotes semantic negative responses.
PersonaChat Dataset	Model	RUBER	InferSent	DistilBERT NLI	MAUDE
Training Modes	Only Semantics	Only Semantics	Only Semantics	Only Semantics
Evaluation Modes	Score	Δ	Score	Δ	Score	Δ	Score	Δ
Semantic Positive	Gold Truth Response	0.443±0.197	0	0.466±0.215	0	0.746±0.236	0	0.789±0.244	0
BackTranslation	0.296±0.198	0.147	0.273±0.195	0.192	0.766±0.235	-0.02	0.723±0.277	0.066
Seq2Seq	0.082±0.163	0.361	0.10±0.184	0.367	0.46±0.357	0.286	0.428±0.390	0.361
Semantic Negative	Random Utterance	0.299±0.203	0.144	0.287±0.208	0.178	0.489±0.306	0.257	0.388±0.335	0.40
Random Seq2Seq	0.028±0.077	0.415	0.036±0.082	0.429	0.237±0.283	0.529	0.16±0.26	0.629
Syntactic Negative	Word Drop	0.334±0.206	0.109	0.308±0.217	0.158	0.802±0.224	-0.056	0.73±0.29	0.059
Word Order	0.472±0.169	-0.029	0.482±0.19	-0.016	0.685±0.284	0.061	0.58±0.35	0.209
Word Repeat	0.255±0.24	0.188	0.153±0.198	0.312	0.657±0.331	0.089	0.44±0.39	0.349
Table 4: Metric score evaluation between InferSent, DistilBERT-NLI and MAUDE on PersonaChat dataset, trained on  $P(\hat{r}) =$  Semantics. Bold scores represent the best individual scores, and bold with blue represents the best difference with the true response.
PersonaChat Dataset	Model	RUBER	InferSent	DistilBERT NLI	MAUDE
Training Modes	Only Syntax	Only Syntax	Only Syntax	Only Syntax
Evaluation Modes	Score	Δ	Score	Δ	Score	Δ	Score	Δ
Semantic Positive	Gold Truth Response	0.891±0.225	0	0.893±0.231	0	0.986±0.088	0	0.99±0.07	0
BackTranslation	0.687±0.363	0.204	0.672±0.387	0.221	0.877±0.268	0.109	0.91±0.23	0.08
Seq2Seq	0.929±0.187	-0.038	0.949±0.146	-0.055	0.996±0.048	-0.01	0.99±0.05	0.00
Semantic Negative	Random Utterance	0.869±0.248	0.022	0.835±0.294	0.058	0.977±0.116	0.009	0.97±0.13	0.02
Random Seq2Seq	0.915±0.196	-0.024	0.904±0.206	-0.011	0.994±0.057	-0.008	0.99±0.08	0
Syntactic Negative	Word Drop	0.119±0.255	0.772	0.105±0.243	0.788	0.373±0.414	0.613	0.41±0.44	0.584
Word Order	0.021±0.101	0.87	0.015±0.0915	0.878	0.064±0.194	0.922	0.07±0.21	0.928
Word Repeat	0.001±0.007	0.89	0.001±0.020	0.893	0.006±0.057	0.980	0.01±0.06	0.981
Table 5: Metric score evaluation between InferSent, DistilBERT-NLI and MAUDE on PersonaChat dataset, trained on  $P(\hat{r}) =$  Syntax. Bold scores represent the best individual scores, and bold with blue represents the best difference with the true response.
2437
Figure 3: From left to right, LDA downsampled representation of BERT on Frames (Goal oriented), MultiWOZ (Goal oriented), PersonaChat (chit-chat) and DailyDialog (chit-chat)
PersonaChat Dataset	Model	RUBER	InferSent	DistilBERT NLI	MAUDE
Training Modes	All	All	All	All
Evaluation Modes	Score	Δ	Score	Δ	Score	Δ	Score	Δ
Semantic Positive	Gold Truth Response	0.432±0.213	0	0.462±0.254	0	0.824±0.154	0	0.909±0.152	0
BackTranslation	0.183±0.198	0.249	0.184±0.218	0.278	0.8±0.19	0.024	0.838±0.227	0.070
Seq2Seq	0.09±0.17	0.342	0.10±0.184	0.362	0.65±0.287	0.174	0.6008±0.38	0.308
Semantic Negative	Random Utterance	0.28±0.21	0.152	0.252±0.236	0.209	0.677±0.255	0.147	0.621±0.344	0.287
Random Seq2Seq	0.03±0.09	0.402	0.026±0.079	0.435	0.48±0.313	0.344	0.323±0.355	0.585
Syntactic Negative	Word Drop	0.09±0.16	0.342	0.094±0.17	0.367	0.563±0.377	0.261	0.609±0.401	0.3
Word Order	0.04±0.10	0.392	0.052±0.112	0.409	0.153±0.29	0.671	0.182±0.327	0.726
Word Repeat	0.00±0.01	0.432	0.001±0.010	0.461	0.041±0.153	0.782	0.036±0.151	0.872
Table 6: Metric score evaluation between InferSent, DistilBERT-NLI and MAUDE on PersonaChat dataset, trained on  $P(\hat{r}) =$  Syntax + Semantics. Bold scores represent the best individual scores, and bold with blue represents the best difference with the true response.
D Qualitative Evaluation
We investigate qualitatively how the scores of different models are on the online evaluation setup on See et al. (2019)'c collected data. In Figure 4, we show a sample conversation where a human evaluator is pitched against a strong model. Here, MAUDE scores correlate strongly with raw likert scores on different metrics. We observe that RUBER and InferSent baselines overall correlate negatively with the response. In Figure 5, we show another sample where a human evaluator is pitched against a weak model, which exhibits degenerate responses. We see both MAUDE and DistilBERT-NLI correlate strongly with human annotation and provides a very low score, compared to RUBER or InferSent.
Since we essentially cherry-picked good results, its only fair to show a similarly cherry-picked negative example of MAUDE. We sampled from responses where MAUDE scores are negatively correlated with human annotations on Inquisitiveness metric (5% of cases), and we show one of those responses in Figure 6. We notice how both DistilBERT-NLI and MAUDE fails to recognize the duplication of utterances which leads to a low overall score. This suggests there still exists room for improvement in developing MAUDE, possibly by training the model to detect degeneracy in the
E Hyperparameters and Training Details
We performed rigorous hyperparameter search to tune our model MAUDE. We train MAUDE with downsampling, as we observe poor results when we run the recurrent network on top of 768 dimensions. Specifically, we downsample to 300 dimensions, which is the same used by our baselines RUBER and InferSent in their respective encoder representations. We also tested with the choice of either learning a PCA to downsample the BERT representations vs learning the mapping  $\mathbf{D}_g$  (Equation 4), and found the latter producing better results. We keep the final decoder same for all models, which is a two layer MLP with hidden layer of size 200 dimensions and dropout 0.2. For BERT-based models (DistilBERT-NLI and MAUDE), we use HuggingFace Transformers (Wolf et al., 2019) to first fine-tune the training dataset on language model objective. We tested with training on frozen finetuned representations in our initial experiments, but fine-tuning end-to-end lead to better ablation scores. For all models we train using Adam optimizer with 0.0001 as the learning rate, early stopping till validation loss doesn't improve. For the sake of easy reproducibility, we use Pytorch Lightning (Falcon, 2019) framework. We used 8 Nvidia-TitanX GPUs
Figure 4: An example of dialogue conversation between human and a strong model, where MAUDE (M) score correlates positively with human annotations. Raw Likert scores for the entire dialogue are: Engagingness : 3, Interestingness : 3, Inquisitiveness : 2, Listening : 3, Avoiding Repetition : 3, Fluency : 4, Making Sense : 4, Humanness : 3, Persona retrieval : 1. Baselines are RUBER (R), InferSent (I) and BERT-NLI (B).
on a DGX Server Workstation to train faster using Pytorch Distributed Data Parallel (DDP).
Figure 5: An example of dialogue conversation between human and a weak model, where MAUDE (M) score correlates positively with human annotations. Raw Likert scores for the entire dialogue are: Engagingness : 1, Interestingness : 4, Inquisitiveness : 1, Listening : 1, Avoiding Repetition : 3, Fluency : 1, Making Sense : 2, Humanness : 1, Persona retrieval : 1. In our setup we only score responses only following a human response. Baselines are RUBER (R), InferSent (I) and BERT-NLI (B).
Human: Hello .. How are things ?
Human: Oh, that's exciting news.
Human: I would like to have children someday. I'm happy for you.
Human: what do you mean.
Human: You said that already
Human: I know you told me twice
greedy_model: I'm doing great. Just got pregnant.
R:0.723 I:0.786 B:0.952 M:0.998
greedy_model: Yes it is. I am pregnant with my first child.
R: 0.118 I: 0.412 B: 0.959 M: 0.933
greedy_model: I am not sure if I could handle that.
R:0.002 I:0.004 B:0.950 M:0.738
greedy_model: I am pregnant with my first child.
R:0.440 I:0.268 B:0.979 M:0.995
greedy_model: No. I am pregnant.
R:0.014 I:0.079 B:0.978 M:0.994
R:0.006 I:0.011 B:0.980 M:0.991
Figure 6: An example of dialogue conversation between human and a model, where MAUDE (M) score correlates negatively with human annotations. Raw Likert scores for the entire dialogue are: Engagingness : 1, Interestingness : 1, Inquisitiveness : 2, Listening : 2, Avoiding Repetition : 2, Fluency : 3, Making Sense : 4, Humanness : 2, Persona retrieval : 1. Baselines are RUBER (R), InferSent (I) and BERT-NLI (B).