Venue: ACL 2017

# Towards an Automatic Turing Test:

Learning to Evaluate Dialogue Responses

 Ryan Lowe\({}^{\heartsuit}\)1 Michael Noseworthy\({}^{\diamondsuit*}\) &Iulian V. Serban\({}^{\diamondsuit}\) Nicolas A.-Gontier\({}^{\heartsuit}\)2 Yoshua Bengio\({}^{\diamondt}\)3 Joelle Pineau\({}^{\heartsuit\dagger}\)4

\({}^{\heartsuit}\) Reasoning and Learning Lab, School of Computer Science, McGill University

\({}^{\diamondsuit}\) Montreal Institute for Learning Algorithms, Universite de Montreal

\({}^{\ddagger}\) CIFAR Senior Fellow

Footnote 1: Indicates equal contribution.

###### Abstract

Automatically evaluating the quality of dialogue responses for unstructured domains is a challenging problem. Unfortunately, existing automatic evaluation metrics are biased and correlate very poorly with human judgements of response quality. Yet having an accurate automatic evaluation procedure is crucial for dialogue research, as it allows rapid prototyping and testing of new models with fewer expensive human evaluations. In response to this challenge, we formulate automatic dialogue evaluation as a learning problem. We present an evaluation model (adem) that learns to predict human-like scores to input responses, using a new dataset of human response scores. We show that the adem model's predictions correlate significantly, and at a level much higher than word-overlap metrics such as BLEU, with human judgements at both the utterance and system-level. We also show that adem can generalize to evaluating dialogue models unseen during training, an important step for automatic dialogue evaluation.

## 1 Introduction

Building systems that can naturally and meaningfully converse with humans has been a central goal of artificial intelligence since the formulation of the Turing test [6]. Research on one type of such systems, sometimes referred to as non-task-oriented dialogue systems, goes back to the mid-60s with Weizenbaum's famous program _ELIZA_: a rule-based system mimicking a Rogerian psychotherapist by persistently either rephrasing statements or asking questions [5]. Recently, there has been a surge of interest towards building large-scale non-task-oriented dialogue systems using neural networks [7, 8, 9, 10, 11, 12]. These models are trained in an end-to-end manner to optimize a single objective, usually the likelihood of generating the responses from a fixed corpus. Such models have already had a substantial impact in industry, including Google's Smart Reply system [10], and Microsoft's Xiaoice chatbot [13], which has over 20 million users.

One of the challenges when developing such systems is to have a good way of measuring progress, in this case the performance of the chatbot. The Turing test provides one solution to the evaluation of dialogue systems, but there are limitations with its original formulation. The test requires live human interactions, which is expensive and difficult to scale up. Furthermore, the test requires carefully designing the instructions to the human interlocutors, in order to balance their behaviour and expectations so that different systems may be ranked accurately by performance. Although unavoidable, these instructions introduce bias into the evaluation measure. The more common approach of having

Figure 1: Example where word-overlap scores fail for dialogue evaluation; although the model response is reasonable, it has no words in common with the reference response, and thus would be given low scores by metrics such as BLEU.



humans evaluate the quality of dialogue system responses, rather than distinguish them from human responses, induces similar drawbacks in terms of time, expense, and lack of scalability. In the case of chatbots designed for specific conversation domains, it may also be difficult to find sufficient human evaluators with appropriate background in the topic Lowe et al. (2015).

Despite advances in neural network-based models, evaluating the quality of dialogue responses automatically remains a challenging and understudied problem in the non-task-oriented setting. The most widely used metric for evaluating such dialogue systems is BLEU Papineni et al. (2002), a metric measuring word overlaps originally developed for machine translation. However, it has been shown that BLEU and other word-overlap metrics are biased and correlate poorly with human judgements of response quality Liu et al. (2016). There are many obvious cases where these metrics fail, as they are often incapable of considering the semantic similarity between responses (see Figure 1). Despite this, many researchers still use BLEU to evaluate their dialogue models Ritter et al. (2011); Sordoni et al. (2015); Li et al. (2015); Galley et al. (2015); Li et al. (2016a), as there are few alternatives available that correlate with human judgements. While human evaluation should always be used to evaluate dialogue models, it is often too expensive and time-consuming to do this for every model specification (for example, for every combination of model hyperparameters). Therefore, having an accurate model that can evaluate dialogue response quality automatically -- what could be considered an _automatic Turing test_ -- is critical in the quest for building human-like dialogue agents.

To make progress towards this goal, we make the simplifying assumption that a 'good' chatbot is one whose responses are scored highly on appropriateness by human evaluators. We believe this is sufficient for making progress as current dialogue systems often generate inappropriate responses. We also find empirically that asking evaluators for other metrics results in either low inter-annotator agreement, or the scores are highly correlated with appropriateness (see supp. material). Thus, we collect a dataset of appropriateness scores to various dialogue responses, and we use this dataset to train an _automatic dialogue evaluation model_ (adem). The model is trained in a semi-supervised manner using a hierarchical recurrent neural network (RNN) to predict human scores. We show that adem scores correlate significantly with human judgement at both the utterance-level and system-level. We also show that adem can often generalize to evaluating new models, whose responses were unseen during training, making adem a strong first step towards effective automatic dialogue response evaluation.1

Footnote 1: Code and trained model parameters are available online: [https://github.com/mike-n-7/ADEM](https://github.com/mike-n-7/ADEM).

## 2 Data Collection

To train a model to predict human scores to dialogue responses, we first collect a dataset of human judgements (scores) of Twitter responses using the crowdsourcing platform Amazon Mechanical Turk (AMT).2 The aim is to have accurate human scores for a variety of conversational responses -- conditioned on dialogue contexts -- which span the full range of response qualities. For example, the responses should include both relevant and irrelevant responses, both coherent and non-coherent responses and so on. To achieve this variety, we use candidate responses from several different models. Following Liu et al. (2016), we use the following 4 sources of candidate responses: (1) a response selected by a TF-IDF retrieval-based model, (2) a response selected by the Dual Encoder (DE) Lowe et al. (2015), (3) a response generated using the hierarchical recurrent encoder-decoder (HRED) model Serban et al. (2016a), and (4) human-generated responses. It should be noted that the human-generated candidate responses are _not_ the reference responses from a fixed corpus, but novel human responses that are different from the reference. In addition to increasing response variety, this is necessary because we want our evaluation model to learn to compare the reference responses to the candidate responses. We provide the details of our

\begin{table}
\begin{tabular}{l l} \hline \hline \# Examples & 4104 \\ \# Contexts & 1026 \\ \hline \# Training examples & 2,872 \\ \# Validation examples & 616 \\ \# Test examples & 616 \\ \hline \(\kappa\) score (inter-annotator correlation) & 0.63 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Statistics of the dialogue response evaluation dataset. Each example is in the form _(context, model response, reference response, human score)_.

 

AMT experiments in the supplemental material, including additional experiments suggesting that several other metrics are currently unlikely to be useful for building evaluation models. Note that, in order to maximize the number of responses obtained with a fixed budget, we only obtain one evaluation score per dialogue response in the dataset.

To train evaluation models on human judgements, it is crucial that we obtain scores of responses that lie near the distribution produced by advanced models. This is why we use the Twitter Corpus (Ritter et al., 2011), as such models are pre-trained and readily available. Further, the set of topics discussed is quite broad -- as opposed to the very specific Ubuntu Dialogue Corpus Lowe et al. (2015) -- and therefore the model may also be suited to other chit-chat domains. Finally, since it does not require domain specific knowledge (e.g. technical knowledge), it should be easy for AMT workers to annotate.

## 3 Technical Background

### Recurrent Neural Networks

Recurrent neural networks (RNNs) are a type of neural network with time-delayed connections between the internal units. This leads to the formation of a _hidden state_\(h_{t}\), which is updated for every input: \(h_{t}=f(W_{hh}h_{t-1}+W_{ih}x_{t})\), where \(W_{hh}\) and \(W_{ih}\) are parameter matrices, \(f\) is a non-linear activation function such as tanh, and \(x_{t}\) is the input at time \(t\). The hidden state allows for RNNs to better model sequential data, such as language.

In this paper, we consider RNNs augmented with long-short term memory (LSTM) units Hochreiter and Schmidhuber (1997). LSTMs add a set of gates to the RNN that allow it to learn how much to update the hidden state. LSTMs are one of the most well-established methods for dealing with the vanishing gradient problem in recurrent networks Hochreiter (1991); Bengio et al. (1994).

### Word-Overlap Metrics

One of the most popular approaches for automatically evaluating the quality of dialogue responses is by computing their _word overlap_ with the reference response. In particular, the most popular metrics are the BLEU and METEOR scores used for machine translation, and the ROUGE score used for automatic summarization. While these metrics tend to correlate with human judgements in their target domains, they have recently been shown to highly biased and correlate very poorly with human judgements for dialogue response evaluation Liu et al. (2016). We briefly describe BLEU here, and provide a more detailed summary of word-overlap metrics in the supplemental material.

BleuBLEU Papineni et al. (2002) analyzes the co-occurrences of n-grams in the reference and the proposed responses. It computes the n-gram precision for the whole dataset, which is then multiplied by a brevity penalty to penalize short translations. For BLEU-\(N\), \(N\) denotes the largest value of n-grams considered (usually \(N=4\)).

DrawbacksOne of the major drawbacks of word-overlap metrics is their failure in capturing the semantic similarity (and other structure) between the model and reference responses when there are few or no common words. This problem is less critical for machine translation; since the set of reasonable translations of a given sentence or document is rather small, one can reasonably infer the quality of a translated sentence by only measuring the word-overlap between it and one (or a few) reference translations. However, in dialogue, the set of appropriate responses given a context is much larger Artstein et al. (2009); in other words, there is a very high _response diversity_ that is unlikely to be captured by word-overlap comparison to a single response.

Further, word-overlap scores are computed directly between the model and reference responses. As such, they do not consider the context of the conversation. While this may be a reasonable assumption in machine translation, it is not the case for dialogue; whether a model response is an adequate substitute for the reference response is clearly context-dependent. For example, the two responses in Figure 1 are equally appropriate given the context. However, if we simply change the context to: _"Have you heard of any good movies recently?"_, the model response is no longer relevant while the reference response remains valid.

## 4 An Automatic Dialogue Evaluation Model (adem)

To overcome the problems of evaluation with word-overlap metrics, we aim to construct a dialogue evaluation model that: (1) captures semantic similarity beyond word overlap statistics, and (2) exploits both the context and the reference response to calculate its score for the model response. We 

call this evaluation model adem.

adem learns distributed representations of the context, model response, and reference response using a hierarchical RNN encoder. Given the dialogue context \(c\), reference response \(r\), and model response \(\hat{r}\), adem first encodes each of them into vectors (**c**, \(\mathbf{\hat{r}}\), and **r**, respectively) using the RNN encoder. Then, adem computes the score using a dot-product between the vector representations of \(c\), \(r\), and \(\hat{r}\) in a linearly transformed space: :

\[score(c,r,\hat{r})=(\textbf{c}^{T}M\mathbf{\hat{r}}+\textbf{r}^{T}N\mathbf{ \hat{r}}-\alpha)/\beta \tag{1}\]

where \(M,N\in\mathbb{R}^{n}\) are learned matrices initialized to the identity, and \(\alpha,\beta\) are scalar constants used to initialize the model's predictions in the range \([1,5]\). The model is shown in Figure 2.

The matrices \(M\) and \(N\) can be interpreted as linear projections that map the model response \(\mathbf{\hat{r}}\) into the space of contexts and reference responses, respectively. The model gives high scores to responses that have similar vector representations to the context and reference response after this projection. The model is end-to-end differentiable; all the parameters can be learned by backpropagation. In our implementation, the parameters \(\theta=\{M,N\}\) of the model are trained to minimize the squared error between the model predictions and the human score, with L2-regularization:

\[\mathcal{L}=\sum_{i=1:K}[score(c_{i},r_{i},\hat{r}_{i})-human_{i}]^{2}+\gamma ||\theta||_{2} \tag{2}\]

where \(\gamma\) is a scalar constant. The simplicity of our model leads to both accurate predictions and fast evaluation (see supp. material), which is important to allow rapid prototyping of dialogue systems.

The hierarchical RNN encoder in our model consists of two layers of RNNs El Hihi and Bengio (1995); Sordoni et al. (2015a). The lower-level RNN, the _utterance-level encoder_, takes as input words from the dialogue, and produces a vector output at the end of each utterance. The _context-level encoder_ takes the representation of each utterance as input and outputs a vector representation of the context. This hierarchical structure is useful for incorporating information from early utterances in the context Serban et al. (2016a). Following previous work, we take the last hidden state of the context-level encoder as the vector representation of the input utterance or context. The parameters of the RNN encoder are pretrained and are not learned from the human scores.

An important point is that the adem procedure above _is not a dialogue retrieval model_: the fundamental difference is that adem has access to the reference response. Thus, adem can compare a model's response to a known good response, which is significantly easier than inferring response quality from solely the context.

Pre-training with VHREDWe would like an evaluation model that can make accurate predictions from few labeled examples, since these examples are expensive to obtain. We therefore employ semi-supervised learning, and use a pre-training procedure to learn the parameters of the encoder. In particular, we train the encoder as part of a neural dialogue model; we attach a third _decoder RNN_ that takes the output of the encoder as input, and train it to predict the next utterance of a dialogue conditioned on the context.

The dialogue model we employ for pre-training is the latent variable hierarchical recurrent encoder-decoder (VHRED) model Serban et al. (2016b), shown in Figure 3. The VHRED model is an extension of the original hierarchical recurrent encoder-decoder (HRED) model Serban et al. (2016a) with a turn-level stochastic latent variable. The dialogue context is encoded into a vector using our hierarchical encoder, and the VHEED then samples a Gaus

Figure 2: The adem model, which uses a hierarchical encoder to produce the context embedding \(\mathbf{c}\).

 

sian variable that is used to condition the decoder (see supplemental material for further details). After training VHRED, we use the last hidden state of the context-level encoder, when \(c\), \(r\), and \(\hat{r}\) are fed as input, as the vector representations for \(\mathbf{c},\mathbf{r}\), and \(\mathbf{\hat{r}}\), respectively. We use representations from the VHRED model as it produces more diverse and coherent responses compared to HRED.

## 5 Experiments

### Experimental Procedure

In order to reduce the effective vocabulary size, we use byte pair encoding (BPE) [14, 2], which splits each word into sub-words or characters. We also use layer normalization [3] for the hierarchical encoder, which we found worked better at the task of dialogue generation than the related recurrent batch normalization [9, 8]. To train the VHRED model, we employed several of the same techniques found in [2] and [7]: we drop words in the decoder with a fixed rate of 25%, and we anneal the KL-divergence term linearly from 0 to 1 over the first 60,000 batches. We use Adam as our optimizer [10].

When training adem, we also employ a subsampling procedure based on the model response length. In particular, we divide the training examples into bins based on the number of words in a response and the score of that response. We then over-sample from bins across the same score to ensure that adem does not use response length to predict the score. This is because humans have a tendency to give a higher rating to shorter responses than to longer responses [2], as shorter responses are often more generic and thus are more likely to be suitable to the context. Indeed, the test set Pearson correlation between response length and human score is 0.27.

For training VHRED, we use a context embedding size of 2000. However, we found the adem model learned more effectively when this embedding size was reduced. Thus, after training VHRED, we use principal component analysis (PCA) [2] to reduce the dimensionality of the context, model response, and reference response embeddings to \(n\). We found experimentally that \(n=50\) provided the best performance.

When training our models, we conduct early stopping on a separate validation set. For the evaluation dataset, we split the train/validation/test sets such that there is no context overlap (i.e. the contexts in the test set are unseen during training).

### Results

#### Utterance-level correlations

We first present new utterance-level correlation results3 for existing

Figure 3: The VHRED model used for pre-training. The hierarchical structure of the RNN encoder is shown in the red box around the bottom half of the figure. After training using the VHRED procedure, the last hidden state of the context-level encoder is used as a vector representation of the input text.

 

word-overlap metrics, in addition to results with embedding baselines and adem, in Table 2. The baseline metrics are evaluated on the entire dataset of 4,104 responses to provide the most accurate estimate of the score. 4 We measure the correlation for adem on the validation and test sets, which constitute 616 responses each.

Footnote 4: Note that our word-overlap correlation results in Table 2 are also lower than those presented in Galley et al. (2015). This is because Galley et al. measure corpus-level correlation, i.e. correlation averaged across different subsets (of size 100) of the data, and pre-filter for high-quality reference responses.

We also conduct an analysis of the response data from Liu et al. (2016), where the pre-processing is standardized by removing '\(<\)first_speaker\(>\)' tokens at the beginning of each utterance. The results are detailed in the supplemental material. We can observe from both this data, and the new data in Table 2, that the correlations for the word-overlap metrics are even lower than estimated in previous studies Liu et al. (2016); Galley et al. (2015). In particular, this is the case for BLEU-4, which has frequently been used for dialogue response evaluation Ritter et al. (2011); Sordoni et al. (2015b); Li et al. (2015); Galley et al. (2015); Li et al. (2016a).

We can see from Table 2 that adem correlates far better with human judgement than the word-overlap baselines. This is further illustrated by the scatterplots in Figure 4. We also compare with adem using tweet2vec embeddings Dhingra et al. (2016). In this case, instead of using the VHRED pre-training method presented in Section 4, we use off-the-shelf embeddings for \({\bf c},{\bf r}\), and \(\hat{\bf r}\), and fine-tune \(M\) and \(N\) on our dataset. These tweet2vec embeddings are computed at the character-level with a bidirectional GRU on a Twitter dataset for hashtag prediction Dhingra et al. (2016). We find that they obtain reasonable but inferior performance compared to using VHRED embeddings.

Figure 4: Scatter plot showing model against human scores, for BLEU-2 and ROUGE on the full dataset, and adem on the test set. We add Gaussian noise drawn from \(\mathcal{N}(0,0.3)\) to the integer human scores to better visualize the density of points, at the expense of appearing less correlated.

 

System-level correlationsWe show the system-level correlations for various metrics in Table 3, and present it visually in Figure 5. Each point in the scatterplots represents a dialogue model; humans give low scores to TFIDF and DE responses, higher scores to HRED and the highest scores to other human responses. It is clear that existing word-overlap metrics are incapable of capturing this relationship for even 4 models. This renders them completely deficient for dialogue evaluation. However, adem produces almost the same model ranking as humans, achieving a significant Pearson correlation of 0.954.5 Thus, adem correlates well with humans both at the response and system level.

Footnote 5: For comparison, BLEU achieves a system-level correlation of 0.99 on 5 models in the translation domain Papineni et al. (2002).

Generalization to previously unseen modelsWhen adem is used in practice, it will take as input responses from a new model that it has not seen during training. Thus, it is crucial that adem correlates with human judgements for new models. We test adem's generalization ability by performing a leave-one-out evaluation. For each dialogue model that was the source of response data for training adem (TF-IDF, Dual Encoder, HRED, humans), we conduct an experiment where we train on all model responses _except_ those from the chosen model, and test _only_ on the model that was unseen during training.

The results are given in Table 4. We observe that the adem model is able to generalize for all models except the Dual Encoder. This is particularly surprising for the HRED model; in this case, adem was trained only on responses that were written by humans (from retrieval models or human-generated), but is able to generalize to responses produced by a generative neural network model. When testing on the entire test set, the model achieves comparable correlations to the adem model that was trained on 25% less data selected at random.

Qualitative AnalysisTo illustrate some strengths and weaknesses of adem, we show human and adem scores for each of the responses to various contexts in Table 5. There are several instances where adem predicts accurately: in particular, adem is often very good at assigning low scores to poor responses. This seen in the first two contexts, where most of the responses given a score of 1 from humans are given scores less than 2 by adem. The single exception in response (4) for the second context seems somewhat appropriate and should perhaps have been scored higher by the human evaluator. There are also several instances where the model assigns high scores to suitable responses, as in the first two contexts.

One drawback we observed is that adem tends to be too conservative when predicting response scores. This is the case in the third context, where the model assigns low scores to most of the responses that a human rated highly. This behaviour is likely due to the squared error loss used to train adem; since the model receives a large penalty for incorrectly predicting an extreme value, it learns to predict scores closer to the average human score. We provide many more experiments, including investigation of evaluation speed, learning curves, data efficiency, a failure analysis, and the primary source of improvement over word-overlap metrics

Figure 5: Scatterplots depicting the system-level correlation results for adem, BLEU-2, BLEU-4,and ROUGE on the test set. Each point represents the average scores for the responses from a dialogue model (TFIDF, DE, HRED, human). Human scores are shown on the horizontal axis, with normalized metric scores on the vertical axis. The ideal metric has a perfectly linear relationship.

\begin{table}
\begin{tabular}{l c} \hline \hline
**Metric** & **Pearson** \\ \hline BLEU-1 & -0.079 (0.921) \\ BLEU-2 & 0.308 (0.692) \\ BLEU-3 & -0.537 (0.463) \\ BLEU-4 & -0.536 (0.464) \\ ROUGE & 0.268 (0.732) \\ \hline ADEM & **0.954** (0.046) \\ \hline \hline \end{tabular}
\end{table}
Table 3: System-level correlation, with the p-value in brackets.

 

in the supplemental material.

## 6 Related Work

Related to our approach is the literature on novel methods for the evaluation of machine translation systems, especially through the WMT evaluation task Callison-Burch et al. (2011); Machacek and Bojar (2014); Stanojevic et al. (2015). In particular, Albrecht and Hwa (2007); Gupta et al. (2015) have proposed to evaluate machine translation systems using Regression and Tree-LSTMs respectively. Their approach differs from ours as, in the dialogue domain, we must additionally condition our score on the context of the conversation, which is not necessary in translation.

There has also been related work on estimating the quality of responses in chat-oriented dialogue systems. DeVault et al. (2011) train an automatic dialogue policy evaluation metric from 19 structured role-playing sessions, enriched with paraphrases and external referee annotations. Gandhe and Traum (2016) propose a semi-automatic evaluation metric for dialogue coherence, similar to BLEU and ROUGE, based on 'wizard of Oz' type data.6 Xiang et al. (2014) propose a framework to predict utterance-level problematic situations in a dataset of Chinese dialogues using intent and sentiment factors. Finally, Higashinaka et al. (2014) train a classifier to distinguish user utterances from system-generated utterances using various dialogue features, such as dialogue acts, question types, and predicate-argument structures.

Footnote 6: In ‘wizard of Öz’ scenarios, humans play the role of the dialogue system, usually unbeknown to the interlocutors.

Several recent approaches use hand-crafted reward features to train dialogue models using reinforcement learning (RL). For example, Li et al. (2016b) use features related to ease of answering and information flow, and Yu et al. (2016) use metrics related to turn-level appropriateness and conversational depth. These metrics are based on hand-crafted features, which only capture a small set of relevant aspects; this inevitably leads to sub-optimal performance, and it is unclear whether such objectives are preferable over retrieval-based cross-entropy or word-level maximum log-likelihood objectives. Furthermore, many of these metrics are computed at the conversation-level, and are not available for evaluating single dialogue responses.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & **Test on full dataset** & \multicolumn{3}{c}{**Test on removed model responses**} \\ \hline Data Removed & **Spearman** & **Pearson** & **Spearman** & **Pearson** \\ \hline TF-IDF & 0.406 (\(<\)0.001) & 0.409 (\(<\)0.001) & 0.186 (0.021) & 0.196 (0.015) \\ Dual Encoder & 0.364 (\(<\)0.001) & 0.373 (\(<\)0.001) & 0.026 (0.749) & 0.027 (0.736) \\ HRED & 0.393 (\(<\)0.001) & 0.396 (\(<\)0.001) & 0.151 (0.060) & 0.148 (\(<\)0.070) \\ Human & 0.292 (\(<\)0.001) & 0.298 (\(<\)0.001) & 0.216 (\(<\)0.010) & 0.148 (\(<\)0.070) \\
**Average** & **0.364** & **0.369** & **0.145** & **0.130** \\ \hline
25% at random & 0.378 (\(<\)0.001) & 0.384 (\(<\)0.001) & — & — \\ \hline \hline \end{tabular}
\end{table}
Table 4: Correlation for adem when various model responses are removed from the training set. The left two columns show performance on the entire test set, and the right two columns show performance on responses only from the dialogue model not seen during training. The last row (25% at random) corresponds to the adem model trained on all model responses, but with the same amount of training data as the model above (i.e. 25% less data than the full training set).

\begin{table}
\begin{tabular}{l l r|r} \hline \hline
* hele \(<\)url? \(\rightarrow\) -a  & diad forgotten about 2 & you heard the prospower productions remix of le search & 1 & 1.513 \\ i really was you! thought put were rec: organizing someone who looked like you! & get ways \begin{tabular}{} \end{tabular} & party ’s ”is”ingly upset” album? & 1 & 1.744 \\ \cline{2-3} \multirow{3}{*}{\begin{tabular}{} \end{tabular} } & \multirow{3}{*}{\begin{tabular}{} \end{tabular} } & \multirow{3}{*}{\begin{tabular}{} \end{tabular} } & \multirow{3}{*}{
\begin{tabular}{} \end{tabular} } \\ \cline{1-1} \cline{4-4}  & & & & 3) yes won’t chag \& rarely reach \\ \cline{2-3} \cline{1-1} \cline{4-4}  & & & & 3) \\ \cline{2-3} \cline{1-1} \cline{4-4}  & & & & 3) they were? \\ \cline{2-3} \cline{1-1} \cline{4-4}  & & & & 3) if no person gets a new phone some more & 1 & 1.848 \\ \cline{2-3} \cline{1-1} \cline{4-4}  & & & & 3) won’t jost got a free pizza coupon! yet gours & 1 & 0.921 \\ \cline{2-3} \cline{1-1} \cline{4-4}  & before there does no more! & \textless{}url? \(>\) & & & 4) if no person gets a new phone \\ \hline am i out of twitter jail yeel’ testing \(\rightarrow\) yean. i posted hul \(\rightarrow\) thanks. i am a night charter betwedons on sundays. same happened list us Sunday lid & any news on meeting & 1) i’m not sure if I’m going to be able to get it. & 3 & 1.912 \\ \cline{2-3} \cline{1-1} \cline{4-4}  & & & 4) if no person has another mac user in the leadership ranks & 4 & 1.417 \\ \cline{2-3} \cline{1-1} \cline{4-4}  & & & 3) www poor baby hope apt to feeding better soon. maybe & 2 & 1.123 \\ \cline{2-3} \cline{1-1} \cline{4-4}  & & & 4) did you tweet too much? & 5 & 2.539 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Examples of scores given by the adem model.
 

The metrics that can be computed at the response-level could be incorporated into our framework, for example by adding a term to equation 1 consisting of a dot product between these features and a vector of learned parameters.

There has been significant work on evaluation methods for task-oriented dialogue systems, which attempt to solve a user's task such as finding a restaurant. These methods include the PARADISE framework Walker et al. (1997) and MeMo Moller et al. (2006), which consider a task completion signal. PARADISE in particular is perhaps the first work on learning an automatic evaluation function for dialogue, accomplished through linear regression. However, PARADISE requires that one can measure task completion and task complexity, which are not available in our setting.

## 7 Discussion

We use the Twitter Corpus to train our models as it contains a broad range of non-task-oriented conversations and it has been used to train many state-of-the-art models. However, our model could easily be extended to other general-purpose datasets, such as Reddit, once similar pre-trained models become publicly available. Such models are necessary even for creating a test set in a new domain, which will help us determine if adem generalizes to related dialogue domains. We leave investigating the domain transfer ability of adem for future work.

The evaluation model proposed in this paper favours dialogue models that generate responses that are rated as highly appropriate by humans. It is likely that this property does not fully capture the desired end-goal of chatbot systems. For example, one issue with building models to approximate human judgements of response quality is the problem of generic responses. Since humans often provide high scores to generic responses due to their appropriateness for many given contexts Shang et al. (2016), a model trained to predict these scores will exhibit the same behaviour. An important direction for future work is modifying adem such that it is not subject to this bias. This could be done, for example, by censoring adem's representations Edwards and Storkey (2016) such that they do not contain any information about length. Alternatively, one can combine this with an _adversarial evaluation model_Kannan and Vinyals (2017); Li et al. (2017) that assigns a score based on how easy it is to distinguish the dialogue model responses from human responses. In this case, a model that generates generic responses will easily be distinguishable and obtain a low score.

An important direction of future research is building models that can evaluate the capability of a dialogue system to have an engaging and meaningful interaction with a human. Compared to evaluating a single response, this evaluation is arguably closer to the end-goal of chatbots. However, such an evaluation is extremely challenging to do in a completely automatic way. We view the evaluation procedure presented in this paper as an important step towards this goal; current dialogue systems are incapable of generating responses that are rated as highly appropriate by humans, and we believe our evaluation model will be useful for measuring and facilitating progress in this direction.

## References

* Albrecht and Hwa (2007) Joshua Albrecht and Rebecca Hwa. 2007. Regression for sentence-level mt evaluation with pseudo references. In _ACL_.
* Artstein et al. (2009) Ron Artstein, Sudeep Gandhe, Jillian Gerten, Anton Leuski, and David Traum. 2009. Semi-formal evaluation of conversational characters. In _Languages: From Formal to Natural_, Springer, pages 22-35.
* Ba et al. (2016) Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. _arXiv preprint arXiv:1607.06450_ .
* Bengio et al. (1994) Yoshua Bengio, Patrice Simard, and Paolo Frasconi. 1994. Learning long-term dependencies with gradient descent is difficult. _IEEE transactions on neural networks_ 5(2):157-166.
* Bowman et al. (2016) Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. _COLING_ .
* Callison-Burch et al. (2011) Chris Callison-Burch, Philipp Koehn, Christof Monz, and Omar F Zaidan. 2011. Findings of the 2011 workshop on statistical machine translation. In _Proceedings of the Sixth Workshop on Statistical Machine Translation_. Association for Computational Linguistics, pages 22-64.
* Cooijmans et al. (2016) Tim Cooijmans, Nicolas Ballas, Cesar Laurent, and Aaron Courville. 2016. Recurrent batch normalization. _arXiv preprint arXiv:1603.09025_ .
* DeVault et al. (2011) David DeVault, Anton Leuski, and Kenji Sagae. 2011. Toward learning and evaluation of dialogue policies with text examples. In _Proceedings of the SIGDIAL 2011 Conference_. Association for Computational Linguistics, pages 39-48.

 

Bhuwan Dhingra, Zhong Zhou, Dylan Fitzpatrick, Michael Muehl, and William W Cohen. 2016. Tweet2vec: Character-based distributed representations for social media. _arXiv preprint arXiv:1605.03481_ .
* Edwards and Storkey (2016) Harrison Edwards and Amos Storkey. 2016. Censoring representations with an adversary. _ICLR_ .
* El Hihi and Bengio (1995) Salah El Hiih and Yoshua Bengio. 1995. Hierarchical recurrent neural networks for long-term dependencies. In _NIPS_. Citeseer, volume 400, page 409.
* Gage (1994) Philip Gage. 1994. A new algorithm for data compression. _The C Users Journal_ 12(2):23-38.
* Galley et al. (2015) Michel Galley, Chris Brockett, Alessandro Sordoni, Yangfeng Ji, Michael Auli, Chris Quirk, Margaret Mitchell, Jianfeng Gao, and Bill Dolan. 2015. deltableu: A discriminative metric for generation tasks with intrinsically diverse targets. _arXiv preprint arXiv:1506.06863_ .
* Gandhe and Traum (2016) Sudeep Gandhe and David Traum. 2016. A semi-automated evaluation metric for dialogue model coherence. In _Situated Dialog in Speech-Based Human-Computer Interaction_, Springer, pages 217-225.
* Gupta et al. (2015) Rohit Gupta, Constantin Orasan, and Josef van Genabith. 2015. Reval: A simple and effective machine translation evaluation metric based on recurrent neural networks. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
* Higashinaka et al. (2014) Ryuichiro Higashinaka, Toyomi Meguro, Kenji Imamura, Hiroaki Sugiyama, Toshiro Makino, and Yoshihiro Matsuo. 2014. Evaluating coherence in open domain conversational systems. In _INTERSPEECH_. pages 130-134.
* Hochreiter (1991) Sepp Hochreiter. 1991. Untersuchungen zu dynamischen neuronalen netzen. _Diploma, Technische Universitat Munchen_ page 91.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_ 9(8):1735-1780.
* Ioffe and Szegedy (2015) Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. _arXiv preprint arXiv:1502.03167_ .
* Kannan et al. (2016) Anjuli Kannan, Karol Kurach, Sujith Ravi, Tobias Kaufmann, Andrew Tomkins, Balint Miklos, Greg Corrado, Laszlo Lukacs, Marina Ganea, Peter Young, et al. 2016. Smart reply: Automated response suggestion for email. In _Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD)_. volume 36, pages 495-503.
* Kannan and Vinyals (2017) Anjuli Kannan and Oriol Vinyals. 2017. Adversarial evaluation of dialogue models. _arXiv preprint arXiv:1701.08198_ .
* Kingma and Ba (2014) Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6980_ .
* Li et al. (2015) Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2015. A diversity-promoting objective function for neural conversation models. _arXiv preprint arXiv:1510.03055_ .
* Li et al. (2016a) Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a. A persona-based neural conversation model. _arXiv preprint arXiv:1603.06155_ .
* Li et al. (2017) Jiwei Li, Will Monroe, and Dan Jurafsky. 2017. Learning to decode for future success. _arXiv preprint arXiv:1701.06549_ .
* Li et al. (2016b) Jiwei Li, Will Monroe, Alan Ritter, and Dan Jurafsky. 2016b. Deep reinforcement learning for dialogue generation. _arXiv preprint arXiv:1606.01541_ .
* Liu et al. (2016) Chia-Wei Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. _arXiv preprint arXiv:1603.08023_ .
* Lowe et al. (2015) Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015. The ubunt dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. _arXiv preprint arXiv:1506.08909_ .
* Machacek and Bojar (2014) Matous Machacek and Ondrey Bojar. 2014. Results of the wmt14 metrics shared task. In _Proceedings of the Ninth Workshop on Statistical Machine Translation_. Citeseer, pages 293-301.
* Markoff and Mozur (2015) J. Markoff and P. Mozur. 2015. For sympathetic ear, more chinese turn to smartphone program. _NY Times_ .
* Moller et al. (2006) Sebastian Moller, Roman Englert, Klaus-Peter Engelbrecht, Verena Vanessa Hafner, Anthony Jameson, Antti Oulasvirta, Alexander Raake, and Norbert Reithinger. 2006. Memo: towards automatic usability evaluation of spoken dialogue services by user error simulations. In _INTERSPEECH_.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th annual meeting on association for computational linguistics_. Association for Computational Linguistics, pages 311-318.
* Pearson (1901) Karl Pearson. 1901. Principal components analysis. _The London, Edinburgh and Dublin Philosophical Magazine and Journal_ 6(2):566.
* Ritter et al. (2011) Alan Ritter, Colin Cherry, and William B Dolan. 2011. Data-driven response generation in social media. In _Proceedings of the conference on empirical methods in natural language processing_. Association for Computational Linguistics, pages 583-593 

Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. _arXiv preprint arXiv:1508.07909_ .
* Serban et al. (2016a) Iulian Vlad Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016a. Building end-to-end dialogue systems using generative hierarchical neural network models. In _AAAI_. pages 3776-3784.
* Serban et al. (2016b) Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2016b. A hierarchical latent variable encoder-decoder model for generating dialogues. _arXiv preprint arXiv:1605.06069_ .
* Shang et al. (2015) Lifeng Shang, Zhengdong Lu, and Hang Li. 2015. Neural responding machine for short-text conversation. _arXiv preprint arXiv:1503.02364_ .
* Shang et al. (2016) Lifeng Shang, Tetsuya Sakai, Zhengdong Lu, Hang Li, Ryuichiro Higashinaka, and Yusuke Miyao. 2016. Overview of the ntcir-12 short text conversation task. _Proceedings of NTCIR-12_pages 473-484.
* Sordoni et al. (2015a) Alessandro Sordoni, Yoshua Bengio, Hossein Vahabi, Christina Lioma, Jakob Grue Simonsen, and Jian-Yun Nie. 2015a. A hierarchical recurrent encoder-decoder for generative context-aware query suggestion. In _Proceedings of the 24th ACM International on Conference on Information and Knowledge Management_. ACM, pages 553-562.
* Sordoni et al. (2015b) Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015b. A neural network approach to context-sensitive generation of conversational responses. _arXiv preprint arXiv:1506.06714_ .
* Stanojevic et al. (2015) Milos Stanojevic, Amir Kamran, Philipp Koehn, and Ondrej Bojar. 2015. Results of the wmt15 metrics shared task. In _Proceedings of the Tenth Workshop on Statistical Machine Translation_. pages 256-273.
* Turing (1950) Alan M Turing. 1950. Computing machinery and intelligence. _Mind__59_(236):433-460.
* Vinyals and Le (2015) Oriol Vinyals and Quoc Le. 2015. A neural conversational model. _arXiv preprint arXiv:1506.05869_ .
* Walker et al. (1997) Marilyn A Walker, Diane J Litman, Candace A Kamm, and Alicia Abella. 1997. Paradise: A framework for evaluating spoken dialogue agents. In _Proceedings of the eighth conference on European chapter of the Association for Computational Linguistics_. Association for Computational Linguistics, pages 271-280.
* Weizenbaum (1966) J. Weizenbaum. 1966. ELIZAa computer program for the study of natural language communication between man and machine. _Communications of the ACM_ 9(1):36-45.
* Xiang et al. (2014) Yang Xiang, Yaoyun Zhang, Xiaoqiang Zhou, Xiaolog Wang, and Yang Qin. 2014. Problematic situation analysis and automatic recognition for chi-nese online conversational system. _Proc. CLP_ pages 43-51.
* Yu et al. (2016) Zhou Yu, Ziyu Xu, Alan W Black, and Alex I Rudnicky. 2016. Strategy and policy learning for non-task-oriented conversational systems. In _17th Annual Meeting of the Special Interest Group on Discourse and Dialogue_. page 404.