υBLEU: Uncertainty-Aware Automatic Evaluation
Method for Open-Domain Dialogue Systems
Yuma Tsuta †, Naoki Yoshinaga ‡, Masashi Toyoda ‡† University of Tokyo‡ Institution of Industrial Science, University of Tokyo

Slide/code/dataset:
https://bit.ly/3hDwTj8

Open-domain dialogue systems
become popular
Dialogue agents are used in daily life

Cited from:https://techcrunch.com/2019/02/05/report-smart-speaker-adoption-in-u-s-reaches-66m-units-with-amazon-leading/• Dialogue agents are expected to reply to any userutterance (open-domain dialogues)

1How to develop
open-domain dialogue systems ?
• Large-scale human-human conversations on SNS
helps to develop open-domain dialogue systems[Wu+2016]

I watched movie yesterday
What movie did you see ?

• Automatic evaluation metrics are needed
to develop dialogue systems efficiently• Existing reference-based evaluation metrics such as BLEU
do not perform well on open-domain dialogue [Liu+2016]

2

Challenge in evaluating
open-domain dialogue systems

3

Difficult to consider all possible responses• Diverse replies can be allowed• Only one reference response is available
when real conversation data is used for evaluation

Input utterance
I watched movie yesterday

Reference response
Let me know how it was

BLEU: 0.548

Generated response 1

Let me know your impression

BLEU: 0Generated response 2

What movie did you see ?

• Evaluation with one reference response is unstable

4

Related work: ΔBLEU[Galley+2015]
ΔBLEU compute weighted BLEU using additional responses
retrieved from Twitter and manual validation to those replies
Step 1. Retrieve reference responses
from dialogue logs
Dialogue logs
(Twitter)

Retrieved dialogues
U2
R2

Test example
U1
R1

Step 2. Rate reference responses
by human annotator

Query0.8-0.6
0.3

Annotator

Dialogue system

Step 3. Compute weighted BLEU with human annotated test samples

STEP 1 on ΔBLEU:
Retrieve dialogues as reference responses
Step 1. Retrieve reference responses
from dialogue logs
Dialogue logs
(Twitter)

U2
R2

Test example
U1

Retrieved dialogues

Query5

Step 2. Rate reference responses
by human annotator

0.8-0.6
0.3

Annotator

R1

dialogue system

Retrieve dialogues of which・utterance is similar to input utterance (of test example), and・response is similar to reference response (of test example)
based on BM25 as similarity function
Test example

Dialogue logs

Step 3. movie
ComputeΔBLEU with human judged scores I went theater to see movie
I watchedyesterdayHigh BM25 score
Let me know how it was
Let me know your impression

STEP 2 on ΔBLEU:
Rate reference responses by hand
Step 1. Retrieve reference responses
from dialogue logs
Dialogue logs
(Twitter)U1

Step 2. Rate reference responses
by human annotator

Retrieved dialogues

0.8U2
R2

Test example

6-0.6

Query

0.3

Annotator

R1

dialogue system

Human annotator rates retrieved responses in terms of appropriatenessto a given utterance
Step 3. Compute ΔBLEU with human judged scores

Utterance of test example
Step 3. movie yesterday
I watched

Annotator

Retrieved responses

Let me know how it was
Let me know your impression

0.7

0.8

STEP 3 on ΔBLEU:
Evaluate generated responses
Step 1. Retrieve reference responses
from dialogue logs
Dialogue logs
(Twitter)

U2
R2

Test example
U1

Retrieved dialogues

Query7

Step 2. Rate reference responses
by human annotator

0.8-0.60.3

Annotator

R1

Dialogue system

Step 3.
Compute weighted BLEU with human annotated test samples

Generated responses is• rewarded when it matches positively rated reference response• penalized when it matches negatively rated reference response

8Issues on ΔBLEU
Step 1. Retrieve reference responses
from dialogue logs
Dialogue logs
(Twitter)

Retrieved dialogues
U2
R2

Test example
U1

Query

R1

Dialogue system

Low semantical diversity of responses•Step 2. Rate reference responses
by human annotator
0.8-0.60.3Annotator

Construction cost of test data

• For the test of open-domain
Retrievalmethod
based
onwiththe similarity
ComputeΔBLEU
human judged scores
Step 3.
dialogue systems, test data onof responses using BM25 (word overlap
several domain is needed
similarity function)

Test example

Step 3. movie yesterday
I watched
Let me know how it was

Dialogue logs (unlikely to be retrieved)
I went theater to see movie

What movie did you see ?

Proposed method:
Automatic evaluation method υBLEU
Proposed method υBLEU deals with the issues on ΔBLEU by• Collecting more diverse reference responses, and• Rating reference responses automatically
Step 1. Retrieve diverse
reference responses
Dialogue logs
(Twitter)

Retrieved dialogues
U2
R2Test example
U1

Step 2. Rate responses automaticallyby neural network (NN)-rater

Query

0.8

-0.6
0.3

R1

Dialogue system

Step 3. Compute weighted BLEU with automatically rated test samples

NN-rater

9

STEP 1 on υBLEU:
Diverse responses retrieval
Step 1. Retrieve diverse
reference responses
Dialogue logs
(Twitter)U1

Step 2. Rate reference responses
by neural network (NN)-rater
Retrieved dialogues
U2
R2

Test example
Query100.8-0.60.3

NN-rater

R1

Dialogue system

To semantically diversify retrieved responses,
Step 3. Compute ΔBLEU with automatically rated test samples
retrieve dialogues based on similarity of utterances only
• Cosine similarity of averaged Glove [Pennington+2014] vector
High vector similarity Dialogue logs
Step 3. movie
ComputeΔBLEU with human judged scores I went theater to see movie
I watchedyesterday

Test example

Let me know how it was

What movie did you see ?STEP 2 on υBLEU:
Automatic response rating
Step 1. Retrieve diverse
reference responses
Dialogue logs
(Twitter)U1

Step 2. Rate reference responses
by neural network (NN)-rater
Retrieved dialogues
U2
R2

Test example

110.8
-0.6

Query0.3

NN-rater

R1

Dialogue system• NN-rater classifies
whether the conversation is appropriate• The utteranceof text example and the response of retrieved dialogue
Compute ΔBLEU with automatically rated test samples• Rate retrieved responses by the classification probability
Test example
Retrieveddialogue

U1

U1
R1
U2R2input

NN-rater(Bi-GRU+FFNN)

R2

output𝑃𝑝𝑜𝑠 & 𝑃𝑛𝑒𝑔

STEP 2 on υBLEU:
Training data of NN-rater
Step 1. Retrieve diverse
reference responses
Dialogue logs
(Twitter)

12

Step 2. Rate reference responses
by neural network (NN)-rater
Retrieved dialogues

0.8

U2
R2

Test example

-0.6

Query

U1

NN-rater

0.3

R1system• NN-rater isDialogue
trainedwith automatically collected data

• PositiveComputesample: ΔBLEU
utterance
which has several responses
with automatically rated test samples• Negative sample: randomly sampled two conversationPositive sample
U1

Negative sample
R1

R2

U1
U2

R1

R2

Experiment 1:
Comparison of response retrieval methods
Evaluate the impact of collecting additionalresponses using the similarity of utterances
1. Compute BLEU with reference responses retrieved bychanging the target and function to compute similarity• Target• Utterance & Response• Utterance only (proposal)• Function• BM25• Cosine similarity for averaged Glove vector (proposal)

2. Compare the correlations with human judgment andBLEU using each retrieved multiple reference responses13

Experiment 1: settings• Number of retrieved reference response: 15 responses• Dialogue systems:

VHRED [Serban+2017], C-BM25 (derivation of C-TFIDF[Lowe+2015]),
human response• Test data:

100 pairs of Japanese conversations on Twitter in 2019

• Human annotation:• Five annotator rated 300 responses in terms of appropriateness inthe scale of [1, 5]• Calculate Pearson correlation between individual judgment andeach evaluation metric• Show the maximum and minimum value in five correlations

14

15Experiment 1: ResultsPearson correlation between human judgment andBLEU with multiple reference response• Max. / Min. of five correlations with individual judgements
Metrics
TargetPearson correlation
Function

Max.

Min.

0.276

0.190

BM25

0.298

0.173

Utterance only BM25

0.296

0.178

Utter. & Resp.

Cosine sim.

0.322

0.177

Utterance only Cosine sim.

0.366

0.209

Original reference only

Utter. & Resp.

Proposed method retrieves more beneficial reference
responses than the method of ΔBLEU [Galley+2015]

Experiment 2:
Comparison of evaluation metrics

16Compare the correlations between human judgment
and each evaluation metric• Methods to compare• ΔBLEU [Galley+2015]• RUBER [Tao+2018]• υBLEU• RUBER with υBLEU (*)* RUBER is constituted by• referenced-based metric and• unreferenced-based metric

we also propose automatic integrated metric (RUBER with υBLEU)
replacing its referenced-based-metric with υBLEU

Experiment 2: settings• Training data of NN-rater and RUBER

5.6M pairs of Japanese conversations on Twitter in 2017

• Dialogue systems, Test data and Human annotation
Same as experiment 1

1718Experiment 2: Results
Pearson correlation between human judgment andevaluation metric• Max. / Min. of five correlations with individual judgements
Metrics

Pearson correlation
Max.

Min.

ΔBLEU

0.360

0.294

υBLEU

0.394

0.332

RUBER

0.325

0.193

RUBER with υBLEU

0.450

0.338

• Our proposal υBLEU outperformed ΔBLEU• Integrating uBLEU into RUBER improved correlation

Summary• Proposal:

Uncertainty-Aware Automatic Evaluation Method forevaluating open-domain dialogue systems• Collect semantically diverse reference responses• Rate responses automatically with neural network classifier• Using Twitter dialogues, we experimentally confirmed• Comparable with semi-automatic evaluation metric, ΔBLEU• Improve the correlation of the state-of–the-art automatic evaluation
method RUBER by integrating with υBLEU

19

Acknowledgments

20

The research was supported by NII CRIS collaborative
research program operated by NII CRIS and LINE Corporation