SlowGuess/ACL_2020_OCR huggingface dataset:
JinYeong Bak
School of Computing KAIST
jy.bak@kaist.ac.kr
Alice Oh
alice.oh@kaist.edu
Automatic evaluation of open-domain dialogue response generation is very challenging because there are many appropriate responses for a given context. Existing evaluation models merely compare the generated response with the ground truth response and rate many of the appropriate responses as inappropriate if they deviate from the ground truth. One approach to resolve this problem is to consider the similarity of the generated response with the conversational context. In this paper, we propose an automatic evaluation model based on that idea and learn the model parameters from an unlabeled conversation corpus. Our approach considers the speakers in defining the different levels of similar context. We use a Twitter conversation corpus that contains many speakers and conversations to test our evaluation model. Experiments show that our model outperforms the other existing evaluation metrics in terms of high correlation with human annotation scores. We also show that our model trained on Twitter can be applied to movie dialogues without any additional training. We provide our code and the learned parameters so that they can be used for automatic evaluation of dialogue response generation models.
Evaluating the system generated responses for open-domain dialogue is a difficult task. There are many possible appropriate responses given a dialogue context, and automatic metrics such as BLEU (Papineni et al., 2002) or ROUGE (Lin, 2004) rate the responses that deviate from the ground truth as inappropriate. Still, it is important to develop and use an automatic metric because human annotation is very costly. In addition to BLEU and ROUGE, there is a widely-used evaluation metric based on the distributed word representation (Liu et al., 2016), but this metric shows low correlations with human judgments.
One reason for the difficulty in developing an automatic metric that correlates well with human judgements is that the range of appropriate responses for a given context is very wide. Table 1 shows an example of a conversation between Speaker A and B. While there is a ground truth response "Yeah let's go to the theater," A could have also said "That sounds good! Have you seen Thor?" or "Good. What movie?" Note that based on word overlap with the ground truth, these two responses would receive low scores. Responses labeled N#, such as "The weather is no good for walking" are not appropriate. As the Table shows, the existing metrics from BLEU to RUBER are not able to tell apart these appropriate A# responses from the inappropriate N# responses.
Some recent metrics such as ADEM (Lowe et al., 2017) and RUBER (Tao et al., 2018) compute the similarity between a context and a generated response. However, ADEM requires human-annotated scores to train and thus cannot be applied to new datasets and domains. RUBER overcomes this limitation by using the idea that a random response should be used as a "negative sample", but it is not able to distinguish the responses in the example in Table 1, because it uses only one random sample which does not provide sufficient information about appropriate and inappropriate responses.
In this paper, we propose Speaker Sensitive Responses Evaluation Model (SSREM) that analyzes the appropriateness of the responses. We use speaker sensitive responses that are generated by one speaker to train the model. We test SSREM in comparison with other evaluation metrics. First, we make annotated human scores for responses in Twitter conversation data. The evaluation scores of SSREM shows a higher correlation with human scores than other evaluation metrics. And SSREM outperforms other metrics in terms of identifying the ground truth responses given a context. We
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6376-6385 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Context Ground truth response	A: What do you want to do tonight? B: Why don't we go see a movie? A: Yeah Let's go to the theater
Utterance	BLEU	ROUGE	EMB	RUBER	SSREM	Human
A1	That sounds good! Have you seen Thor?	0.00 (3)	0.00 (3)	0.95 (2)	0.59 (2)	0.64 (1)	5.00 (1)
A2	Good, What movie?	0.00 (3)	0.00 (3)	0.92 (4)	0.55 (4)	0.62 (2)	5.00 (1)
A3	Or hang out in city	0.00 (3)	0.00 (3)	0.89 (6)	0.48 (5)	0.49 (3)	3.80 (3)
N1	The weather is no good for walking	0.32 (1)	0.15 (2)	0.94 (3)	0.47 (6)	0.44 (4)	2.60 (4)
N2	The sight is extra beautiful here	0.32 (1)	0.17 (1)	0.97 (1)	0.64 (1)	0.38 (5)	1.00 (5)
N3	Enjoy your concert	0.00 (3)	0.00 (3)	0.91 (5)	0.57 (3)	0.33 (6)	1.00 (5)
Table 1: Example of appropriate responses (A1 - A3) and non-appropriate responses (N1 - N3) for a given context and ground truth response, and the responses' scores by evaluation metrics. Emb is embedding average and Human is average scores from five people. Ranks are shown in brackets. SSREM has positive correlation with human scores.
show the additional advantage of SSREM: it can be applied to evaluate a new corpus in a different domain. We train SSREM on Twitter corpus and test it on a corpus of movie reviews, and we show that SSREM outperforms other metrics in terms of the correlation with human scores and the task of identifying the ground truth response.
Our contributions in this paper include the following.
- We present SSREM, a new response evaluation model trained with speaker sensitive negative samples (Sec 3).
- We conduct experiments on a Twitter conversation corpus and show that SSREM outperforms the others (Sec 5 and 6). We further show the applicability of SSREM with Movie dialogue corpus that are not using in the training (Sec 7).
- We provide our code and the learned parameters of SSREM which can be used for evaluation of generated responses<sup>1</sup>.
In this section, we describe existing automatic evaluation metrics for dialogue response generation and discuss their limitations.
For task-oriented dialogue models such as airline travel information system (Tur et al., 2010), completing the given task is most important, and the evaluation metrics reflect that (Hastie, 2012; Bordes et al., 2017). But open-domain conversation models do not have specific assigned tasks; the main goal of an open-domain conversation model
is generating appropriate responses given a conversation about any topic.
Existing automatic evaluation metrics compare a generated response and the ground truth response. The most widely-used metric are BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004) based on the overlap of words between the two responses. A limitation of these word overlap-based metrics is that they cannot identify the synonyms, and to overcome this limitation, the embedding-based metrics use distributed word vector representations (Liu et al., 2016). However, these metrics have poor correlation with human judgments (Liu et al., 2016; Novikova et al., 2017; Gupta et al., 2019) because they still only look at the similarity between the generated response and the ground truth. SSREM is a model with the awareness that a response can be different from the ground truth response but still appropriate for the conversation context.
The responses for a casual conversation can be varied. For example, there are four appropriate responses including ground truth response for a given context in Table 1. Some previous approaches suggest considering the context together with the response such as ADEM (Lowe et al., 2017) and RUBER (Tao et al., 2018). ADEM uses pre-trained VHRED (Serban et al., 2017) to encode the texts and compute the score by mixing similarities among the context, generated response and a ground truth. One limitation of ADEM is that it requires human annotated scores to learn the model. Human labeling is cost-intensive, so it is impractical to apply to a new dataset or domain. RUBER uses negative sampling to overcome this issue, but it uses only one random negative sample against one positive sample which is not ideal (Gutmann and Hyvarinen, 2010). SSREM does not require
<https://github.com/NoSyu/SSREM>
6377
Figure 1: Example of utterance sets for speaker  $A$ .  $SC$  stands for 'same conversation',  $SP$  for 'same partner',  $SS$  for 'same speaker', and Rand for 'random'.
SC	SP	SS	Rand
.922±1e-4	.919±2e-4	.912±3e-4	.898±2e-3
Table 2: Mean similarity among utterances in  ${SC},{SP}$  , SS and Rand sets with a 95% confidence interval
human scores to learn the model and uses many speaker sensitive negative samples.
3 Speaker Sensitive Response Evaluation Model
This section describes our Speaker Sensitive Response Evaluation Model (SSREM) that trains with speaker sensitive utterance samples. SSREM looks at a given context and its ground truth response together to evaluate a generated response. We describe the motivation of SSREM with empirical observations in section 3.1. We present the structure of SSREM in section 3.2. With the motivation, we present a training method of SSREM with speaker sensitive utterance samples in section 3.3.
We are motivated by the assumption that there is a varying degree of similarity among utterances in a corpus of conversations containing many speakers and conversations.
1. If we pick a set of random utterances from the corpus, they will not be very similar.
2. If we pick a set of utterances from a single speaker conversing with multiple partners, those utterances will be more similar than the random utterances in 1.
3. If we pick a set of utterances from conversations between a single dyad, even if the conver
sations are far apart in time, those utterances would be more similar than those in 2.
4. If we pick a set of utterances in a single conversation session, they are the most similar, even more so than those in 3.
To test these assumptions, we first categorize one speaker  $A$ 's utterances into four types of sets corresponding to the assumptions above.
- Random  $(\text{Rand}_A)$ : Random utterances from speakers who are not  $A$
- Same Speaker  $(SS_A)$ : Speaker  $A$ 's utterances
- Same Partner  $(SP_A)$ : A's utterances in conversations with the same partner  $B$
- Same Conversation  $(SC_{A})$ : A's utterances in a single conversation
Figure 1 shows one example of the sets. We make three  $SC_{A}$  sets because  $A$  participates in three conversations. We make two  $SP_{A}$  sets because  $A$  has conversations with  $B$  and  $C$ .  $SS_{A}$  is all utterances from  $A$  so we create one set of utterances for  $A$ . Finally,  $\text{Rand}_{A}$  is random utterances from non- $A$ 's utterances. We create five sets for each speaker.
From these sets, we compute the similarity among utterances in a set. First, we convert an utterance into a vector by averaging the words in the utterance with GloVe Twitter 200d (Pennington et al., 2014). And we compute the similarity of the vectors by Frobenius norm. Finally, we calculate the mean similarity of each set with a  $95\%$  confidence interval. Table 2 shows the results. Rand has the lowest similarity mean value, so it supports the first assumption.  $SS$  has higher similarity mean value than Rand. It supports the second assumption. The mean similarity value of  $SP$  is higher than  $SS$ . It supports the third assumption. Finally,  $SC$  has the highest mean similarity value. It also supports the last assumption. From the observations, we assume that utterances are clustered by the speakers and addressees.
3.2 SSREM
SSREM evaluates a generated response  $\hat{\mathbf{r}}$  from a context  $\mathbf{c}$  and a ground truth response  $\mathbf{r}$ . The output of SSREM is as follows:
$$
S S R E M (\mathbf {c}, \mathbf {r}, \hat {\mathbf {r}}) = h (f (\mathbf {c}, \hat {\mathbf {r}}), g (\hat {\mathbf {r}}, \mathbf {r})) \tag {1}
$$
where  $f(\mathbf{c},\hat{\mathbf{r}}) = \tanh (V(\mathbf{c})^T\mathbf{M}V(\hat{\mathbf{r}}))$  is a parametrized function to measure the similarity between the context  $\mathbf{c}$  and the generated response  $\hat{\mathbf{r}}$ .  $V$  is a function to convert a sequence of words to a vector.  $\mathbf{M}$  is a matrix that weights of the similarity between two vectors. It is the parameter of the  $f$  function.  $g(\mathbf{r},\hat{\mathbf{r}})$  is another function to measure the ground-truth response and the generated one.  $h$  is a function to mix the values of  $f$  and  $g$  functions. To normalize each output of the  $f$  and  $g$  functions, we adopt linear scaling to unit range (Aksoy and Haralick, 2001) which rescale the value  $x$  as follows:
$$
\tilde {x} = \frac {x - l}{u - l} \tag {2}
$$
where  $u$  is an maximum and  $l$  is minimum of  $x$ .
SSREM is similar to RUBER, which computes the similarities among  $\mathbf{c}$ ,  $\mathbf{r}$  and  $\hat{\mathbf{r}}$  separately and merge it at the end. However, SSREM uses speaker sensitive samples, whereas RUBER takes one positive sample and one negative sample.
3.3 Training with Speaker Sensitive Samples
SSREM has a parametrized function  $f$  that takes context  $\mathbf{c}$  and a generated response  $\hat{\mathbf{r}}$ . To train the  $f$  function, we define a classification problem to identify the ground truth response  $\mathbf{r}$  from a set of candidate responses  $R_{\text {cand }}$ . The  $R_{\text {cand }}$  has the ground truth response and some negative samples. A classifier tries to identify the ground truth response with the negative samples. Negative samples are usually selected from the uniform distribution. But we sample the speaker sensitive utterances which described in section 3.1 for SSREM.
Formally speaking, let  $A$  be the speaker of the ground truth response  $\mathbf{r}_A$ . It means it is  $A$ 's turn to say the response for the context  $\mathbf{c}$ . The candidate response set  $R_{\text{cand}_A}$  is given by
$$
R _ {c a n d _ {A}} = \left\{\mathbf {r} _ {A}, \mathbf {s c} _ {A}, \mathbf {s p} _ {A}, \mathbf {s s} _ {A}, \mathbf {r a n d} _ {A} \right\} \tag {3}
$$
where  $\mathbf{sc}_A\in SC_A\setminus \mathbf{c},\mathbf{sp}_A\in SP_A\setminus \mathbf{c},\mathbf{ss}_A\in$ $SS_{A}\backslash \mathbf{c}$  and  $\mathbf{rand}_A\in \mathsf{Rand}_A$  are the negative samples from speaker sensitive responses. Then, the probability of a ground truth response  $\mathbf{r}_A$  given context  $\mathbf{c}$  and  $R_{cand_A}$  is as follows:
$$
p \left(\mathbf {r} _ {A} \mid \mathbf {c}, R _ {\text {c a n d} A}\right) = \frac {\exp \left(f \left(\mathbf {c} , \mathbf {r} _ {A}\right)\right)}{\sum_ {\mathbf {r} ^ {\prime} \in R _ {\text {c a n d} A}} \exp \left(f \left(\mathbf {c} , \mathbf {r} ^ {\prime}\right)\right)} \tag {4}
$$
We maximize this probability among all context-ground truth response pair. So the loss function of the classification problem is
$$
- \sum_ {\mathbf {c}} \log \frac {\exp (f (\mathbf {c} , \mathbf {r} _ {A}))}{\sum_ {\mathbf {r} ^ {\prime} \in R _ {c a n d _ {A}}} \exp (f (\mathbf {c} , \mathbf {r} ^ {\prime}))} \tag {5}
$$
This approach is similar to learning the sentence representations (Logeswaran and Lee, 2018), but we use the speaker sensitive negative samples. It is also similar to Noise Contrastive Estimation (NCE) (Gutmann and Hyvarinen, 2010; Mnih and Teh, 2012). But we set the noise distribution to speaker sensitive distribution and only take the data sample term in the objective function of the NCE.
Selecting negative samples is important for learning. When we choose the noise distribution, it would be close to the data distribution, because otherwise, the classification problem might be too easy to learn the data (Gutmann and Hyvarinen, 2010). Mnih and Teh (2012) shows that using samples from the unigram distribution outperforms using samples from a naive uniform distribution for learning a neural probabilistic language model. Likewise, we create negative samples from the speaker sensitive utterances.  $\mathbf{sc}_A$  is more similar to the  $\mathbf{r}_A$  than any other negative samples. We show the patterns by empirical observations in section 3.1 and experimental results in section 6.2. These speaker sensitive samples make the classification problem harder and lead to learning the function  $f$  better than using the naive uniform distributed random samples.
To train SSREM, we need a conversation corpus that has many conversations from one speaker. We choose the Twitter conversation corpus (Bak and Oh, 2019) as it has 770K conversations with 27K Twitter users. We split the data as 80/10/10 for training/validation/test.
4 Annotating Human Scores
To measure the correlation SSREM with human judgments, we first gather human judgments of responses given a conversation context. We use Amazon Mechanical Turk (MTurk) to annotate the scores of the responses. We select 300 conversations from a dataset of Twitter conversations. And we generate responses for annotation using three conversation models and the ground truth response for each conversation.
- Retrieval model (Pandey et al., 2018): A
6379
Human Score	1	2	3	4	5
Twitter	211	258	342	278	71
Movie	279	267	311	217	126
Table 3: Basic statistics of human scores of the responses on Twitter conversation and Movie scripts
BM25 retrieval model (Robertson et al., 2009) that uses TF-IDF vector space.
- VHCR (Park et al., 2018): A variational autoencoder model that has a global variable for a conversation.
- VHUCM (Bak and Oh, 2019): A variational autoencoder model that considers the speakers of a conversation.
Then we ask two questions to the MTurkers. (1) How appropriate is the response overall? (2) How on-topic is the response? These questions are used in (Lowe et al., 2017). The authors show that these questions have high inter-annotator agreement among workers. They suggest using the first question to annotate the human score, and so we follow the suggestion. But we ask the second question to workers to filter out workers who submit random answers. Each worker answers these questions on a five-point Likert scale.
We annotate 1,200 responses in total. One worker answers ten conversations, four responses per conversation for a total of 40 responses. Each response is tagged by five workers for a total of 287 workers of which we retain the responses from 150 workers who passed all the tests. We tag the most selected score as the human score for each response. The inter-annotator Fleiss' kappa (Fleiss, 1971) is  $\kappa = 0.61$  which is consistent with the results in (Lowe et al., 2017). Table 3 shows the basic statistics of the annotations.
5 Experiment 1 - Comparing with Human Scores
This section describes the experiment that looks at the correlation between the model scores and the human scores for given contexts and responses.
We use a Twitter conversation corpus (Bak and Oh, 2019) to train and validate SSREM and other baseline models. For the test, we remove the ground truth responses in human-annotated corpus since
it always produces the maximum score on BLEU and ROUGE.
We compare SSREM with the following response evaluation methods:
- BLEU (Papineni et al., 2002): We compute the sentence-level BLEU score with the smoothing seven technique (Chen and Cherry, 2014).
- ROUGE (Lin, 2004): We compute the F score of ROUGE-L.
- EMB (Liu et al., 2016): We compute the average cosine similarity between ground truth response and test response in a word embedding $^{2}$ . We use pre-trained Google news word embedding (Mikolov et al., 2013) to avoid the dependency between the training data and embedding.
- RUBER (Tao et al., 2018): We train with a random negative sample to train unreferenced metric in RUBER. And we use arithmetic averaging to hybrid the referenced and unreferenced metrics.
- RSREM: We use the same structure of SSREM, but train with uniformly random negative samples, not speaker sensitive samples.
We choose functions in SSREM for the experiment. For  $V$  function, We use the word averaging technique that averages the vectors of words in the sequence. We can use advanced methods such as RNN or sentence embeddings (Reimers and Gurevych, 2019). But for the fair comparisons with RUBER, we select a similar approach. We use GloVe Twitter 200d word embedding (Pennington et al., 2014). For  $g$  function, we use sentence mover's similarity that is the state of the art evaluating reference-candidate pair of sentences by using word and sentence embeddings (Clark et al., 2019). To avoid dependency between the training data and embedding, we use Elmo embedding (Peters et al., 2018). For  $h$  function, we use arithmetic averaging that shows good results in (Tao et al., 2018).
Table 4 shows the Spearman and Pearson correlations between human scores and models scores. First, BLEU, ROUGE, and EMB are not correlated
2We experimented with the greedy and extreme embedding for comparison, but these methods were not better than the average embedding.
6380
(a) BLEU,coeff:0.005
(d) RUBER, coeff: 0.006
Figure 2: Scatter plots that show model scores against human scores. We add Gaussian noise drawn from  $N(0,0.3)$  to the human scores to better visualize the density of points (Lowe et al., 2017). The red line is a linear regression line, and the coeff is the coefficient of the line. SSREM shows a higher positive correlation with human judgment than the other models.
(b) ROUGE, coeff: 0.005
(e) RSREM, coeff: 0.013
(c) EMB, coeff: 0.003
(f) SSREM, coeff: 0.058
Metric	Spearman	Pearson
BLEU	0.024 (0.472)	0.041 (0.227)
ROUGE	0.024 (0.471)	0.052 (0.124)
EMB	0.006 (0.861)	0.012 (0.720)
RUBER	0.044 (0.192)	0.046 (0.177)
RSREM	0.088 (< 0.01)	0.101 (< 0.01)
SSREM	0.392 (< 0.001)	0.376 (< 0.001)
Table 4: Correlation between human and model scores. We compute Spearman and Pearson correlation coefficients.  $p$ -values are shown in brackets. SSREM shows higher correlation with human judgement than the other models.
with human scores. It means evaluating responses with ground truth only is not useful. These results are the same in previous research (Liu et al., 2016; Lowe et al., 2017; Tao et al., 2018). RUBER shows a higher correlation with human scores than other baselines but has a high  $p$ -value that means low statistically significant. RSREM performs better than RUBER and other baselines. It shows using multiple negative samples improves the performance of learning the model. Finally, SSREM outperforms all other methods for two correlations with low  $p$ -values. It shows the effectiveness of using speaker sensitive negative samples.
Figure 2 shows scatterplots of the human and
model scores. A dot is one response, and a red line is a linear regression line. The x-axis is the human score, and the y-axis is each automatic evaluation metric. To visualize the dots better, we adopt the technique from (Lowe et al., 2017) that adds random number  $(N(0,0.3))$  to x-axis value. But, we train the linear regression with original scores. First, BLEU and ROUGE have many zero values since there are few overlapped words between the generated response and the ground-truth response. The dots in EMB that uses word embedding to overcome the limitation are more distributed. But there are few relationships with human scores, and the linear regression coefficient is flattened. RUBER is better than BLEU, ROUGE, and EMB. RSREM that uses more negative samples shows better than RUBER. Finally, SSREM shows a higher positive correlation with human scores than other baselines.
6 Experiment 2 - Identifying True and False Responses
The second experiment presents the performance of  $f$  function in SSREM by comparing it with baselines. RUBER, RSREM, and SSREM compute the score from the context of the conversation and generated responses. To investigate the performance of the score, we set up the task that identifies the true and false responses for a given context. The
6381
Figure 3: Difference of scores on various responses in Twitter conversation corpus. The range of the vertical error bar is a  $95\%$  confidence interval of the values among the responses. SSREM outperforms the other models for identifying true and false responses.
true responses are ground-truth responses, and false ones are four negative samples that are described in section 3.3.