DEIM Forum 2019 E1-4

Dialogue Quality Distribution Prediction
based on a Loss that Compares Adjacent Probability Bins
Sosuke KATO† and Tetsuya SAKAI†† Department of Computer Science and Engineering, Waseda University
3–4–1 Okubo, Shinjuku-ku, Tokyo 169–8555, Japan
E-mail: †sow@suou.waseda.jp, †tetsuyasakai@acm.org
Abstract The NTCIR-14 Short Text Conversation Dialogue Quality subtask requires participating systems topredict a distribution of dialogue quality scores for each given customer-helpdesk dialogue. The gold distribution
represents the views of multiple annotators, and the systems are evaluated by comparing the two distributions overordinal bins of scores. In this study, we propose a loss function that is based on comparing the probabilities inadjacent bins and demonstrate its eﬀectiveness for the task. Our proposed model outperformed the baseline modelin terms of every measure for Chinese dataset.
Keywords loss function, probability distribution prediction, short text conversation, dialogue quality

1 Introductionin Figure 1 (a). When evaluating the predicted distributions
like (b), (c) in Figure 1, the distance between (a) and (b)Aiming at building a dialogue system with artificial in-and between (a) and (c) are the same if a traditional mea-

telligence, research on dialogue systems have received muchsure such as the Jensen-Shannon divergence (JSD) is used.

attention, and some competitions [1,3,5] related to dialogues

To solve this problem, the systems are evaluated by com-have been held. Tasks related to dialogues can be classified

paring the two distributions over ordinal bins of scores using

according to the property of a dialogue, e.g., a task-oriented

the Normalised Match Distance (NMD) and the Root Sym-or non-task-oriented dialogue; a human-human or human-

metric Normalised Order-aware Divergence (RSNOD) [8] inmachine dialogue, and what systems of participants do, e.g.,

STC-3.

generation, retrieval, classification, and so on.
In the End-to-End Goal-Oriented Dialog Learning Track[5] as a retrieval task, given a dialogue, a participating system selects a machine utterance or some action of the machine which should immediately follow the dialogue from thegiven candidates. In this task which deals with task-oriented
human-machine dialogues, Precision was used as an evalua-

(a)

(b)

(c)

Figure 1 Examples of distributions

tion measure and actual utterances or actions can be usedas gold data.

In this study, we propose a loss function that is based on

In the case of tasks in which a system predicts some types

comparing the probabilities in adjacent bins in order to han-of labels, gold labels usually are made by multiple annota-

dle ordinal bins, and demonstrate its eﬀectiveness for thetors manually. However, allowing one label per item means

NTCIR-14 STC-3 Dialogue Quality subtask.

that the gold data cannot directly represent the views of multiple annotators. In contrast, in the NTCIR-14 Short Text

2 Related Work

Conversation 3 (STC-3) [3] and the Dialogue Breakdown De-

2. 1 NTCIR-14 Short Text Conversation Task

tection Challenge [1], a system must predict the gold distri-

The NTCIR-14 Short Text Conversation Task [3] has threebution which represents the views of multiple annotators. To

subtasks; Chinese Emotional Conversation Generation sub-evaluate the systems, the two distributions, i.e., the gold dis-task, Nugget Detection subtask, and Dialogue Quality (STC-

tribution and the predicted distribution, must be compared.

3 DQ) subtask. The present study concerns the STC-3 DQ

The number of types of labels is relatively low, i.e., 5 in thesubtask, which is described below.

case of STC-3, and an example of a gold distribution is shown

The STC-3 DQ subtask requires participating systems topredict a distribution of dialogue quality scores for each given
customer-helpdesk dialogue. The training dataset [11]（ *1）model and then we classify the gold and predicted distribu-astions and plot the heat maps. Finally, we compare these

the customer-helpdesk dialogues with annotations by multi-

heat maps and show some pairs of gold and predicted distri-ple annotators are available and used in STC-3. The target

butions.

languages of the STC-3 DQ are Chinese and English. The

3. 1 Training of the Baseline Model

original language of the dialogues is Chinese and a part of

We divided the training dataset into the one to train the

dialogues was translated into English. The annotations formodel and the one to plot the heat maps in the ratio of 8 : 2.

the STC-3 DQ subtask in this dataset are labeled in the

The latter was also used as validation data. Using the di-following types of quality scores （ *2）;vided data for training and validation, we train the baseline•

A-score: Task Accomplishment (Has the problem beensolved? To what extent?),•

data to plot the heat maps.

S-score: Customer Satisfaction of the dialogue (not of

the product/service or the company),•

BLSTM model and predict distributions of the validation
3. 2 Distribution Mapping for Classification
Focusing on considering ordinal bins, we map a distribu-

E-score: Dialogue Eﬀectiveness (Do the utterers inter-tion to a class considering adjacent probability bins using theact eﬀectively to solve the problem eﬃciently?).

mapping table shown in Table 1. In this mapping where adja-

These scores are on a five-point scale: [−2, −1, 0, 1, 2].

cent probability bins are compared, distributions are mapped

Given a dialogue and a gold distribution p∗ , a system pre-to 16 (= 25−1 ) types on a five-point scale. For example, thedicts a distribution p for every type of quality score and the

distribution of the top example in Table 1 is classified as‘15’ because p(−2) <= p(0), p(0) <= p(1) and= p(−1), p(−1) <NMD and the RSNOD as evaluation measrues are calculated.
One of the baseline model （ *3） of the STC-3 DQ sub-p(1) <= p(2) hold in this distribution.

task is based on a Bidirectional Long Short-term Memory

3. 3 Diﬀerences among NMD, RSNOD, and JSD(BLSTM) [2,10] which needs a loss function to train a model.

Figures 2 to 5 visualize the A-score distributions of NMD,

The loss function of the baseline BLSTM model are defined

RSNOD, and JSD for comparison, where the x-axis repre-

as follows,
Lce = −

sents the gold distribution classes and the y-axis representsB∑p∗ (i) log p(i)the classes predicted by the baseline BLSTM model. The
(1)

i=1

where B denotes the number of bins, i.e., 5 in this study.
2. 2 Loss Function

number of each cell in Figure 2 denotes the number of dialogues. The number of each cell denotes the mean NMD,RSNOD and JSD for A-score in Figures 3 to 5 respectively.
We can see from Figure 2 that most of the gold distribu-

In Eq. (1), Cross Entropy (CE) [6] is calculated. The CE istions are classified as ‘12’ and ‘13’, and the baseline BLSTM

based on Kullback-Leibler divergence which forms the basis

model tends to predict the distributions which are classifiedof JSD and therefore does not consider ordinal bins. Whileas ‘12’. Moreover, we can see over Figures 3 to 5 that dis-loss functions for classification tasks have been discussed intances are relatively lager when the class of the distribution

previous work [4], those for comparing two distributions overis predicted as ‘6’.

ordinal bins have not.

We focus on the cells where the class of the predicted dis-

In Section 4, we proposed a loss function which considers

tribution is ‘6’ and we show examples of the pairs of theordinal bins and train a model using the same neural net-

gold and predicted distribution in Figures 6 and 7 where thework as a baseline BLSTM model that does not use our loss

classes of predicted distributions are ‘6’. In contrast, thefunction.

class of gold distributions in Figure 6 is ‘12’ and the one in

3 Preliminary Experiment

Figure 7 is ‘6’. We also show the evaluation measures foreach pair in Figures 6 and 7. Comparing Figures 6 and 7, in

As a preliminary experiment, we compare the evaluation

terms of JSD, Example 1 is considered better than Example

measures for ordinal bins, i.e., RSNOD and NMD, with JSD

2, which is quite counterintuitive. In contrast, according towhich does not consider ordinal bins. In order to compare

NMD and RSNOD, Example 2 is rated higher than Example

these measures visually, we plot the heat map and show ex-

1. These examples suggest that we should use a loss function

amples of distributions. First, we train the baseline BLSTM

considering ordinal bins when NMD and RSNOD are usedas the evaluation measures.

（ *1）：https://sakai-lab.github.io/stc3-dataset/（ *2）：http://sakailab.com/wp-content/uploads/2018/06/STC3atNTCIR-14.

4 Proposed Methodpdf（ *3）：https://github.com/sakai-lab/stc3-baseline

Again, from the preliminary experiment, we should use

Table 1 Mapping table to map a distribution p to a classp(−2) ↔ p(−1) p(−1) ↔ p(0) p(0) ↔ p(1) p(1) ↔ p(2) classp(0) <= p(1)

p(−1) <
= p(0)

p(0) > p(1)p(−2) <= p(−1)p(0) <= p(1)

p(−1) > p(0)p(0) > p(1)p(0) <= p(1)

p(−1) <
= p(0)

p(0) > p(1)p(−2) > p(−1)p(0) <= p(1)

p(−1) > p(0)p(0) > p(1)p(1) <= p(2)

15

p(1) > p(2)14p(1) <= p(2)

13

p(1) > p(2)12p(1) <= p(2)

11

p(1) > p(2)10p(1) <= p(2)

9p(1) > p(2)8p(1) <= p(2)

7

p(1) > p(2)6p(1) <= p(2)

5p(1) > p(2)4p(1) <= p(2)

3p(1) > p(2)2p(1) <= p(2)

1p(1) > p(2)0exampleFigure 2 The heat map of the number of dialogues for Ascore (20% of the training dataset)

Figure 4 The heat map of Average RSNOD for A-score(20% of the training dataset)

Figure 3 The heat map of Average NMD for A-score (20%Figure 5 The heat map of Average JSD for A-score (20%of the training dataset)of the training dataset)a loss function considering ordinal bins. We apply the loss
function showed in Eq. (2) as a simple one.

NMD:
0.155
RSNOD: 0.254
JSD:
0.091

1 ∑
2
{(p(i + 1) − p(i)) − (p∗ (i + 1) − p∗ (i))}
B−1 i
B−1

Ldiﬀ =(2)
Figure 6 A Distribution Pair Example 1
We use the combination of loss functions as follows,
L = αLdiﬀ + (1 − α)Lce(3)

where α denotes the paramater to adjust the weight of Ldiﬀ .

NMD:
0.129
RSNOD: 0.252
JSD:
0.123

We use Eq. (3) and the same neural network as the baseline BLSTM model of STC-3 excluding the loss function. In
other words, when we set α to 0.0, the baseline and proposedmodel are same. We set α to 0.5 as this achieved lower (i.e.,
more eﬀective) dialogue quality scores than 0.25 and 0.75 forthe validation data.

Figure 7 A Distribution Pair Example 2

5 Experiment
5. 1 Dataset•

Our proposed model outperformed the baseline

BLSTM model in terms of NMD and RSNOD of S-score
for English dataset.

We used the training dataset of STC-3 for the Chinese andTable 2 Chinese Results (A-score)

English STC-3 DQ subtask. In the main experiment, we di-Modelvide the dataset into the training, validation and testing data

Mean NMD Mean RSNOD

in the ratio of 7 : 2 : 1.

Baseline(α = 0.0)

0.07760.11995. 2 Training

Diﬀ+(α = 0.5)

0.07620.1195

As an example, we show the training loss and RSNOD

only Diﬀ (α = 1.0)

0.08120.1253as validation loss of each type of quality score and mean
RSNOD over these types for English data in Figure 8 whose
Table 3 Chinese Results (S-score)

x-axis denotes the number of global steps of training. The

Model

training loss is scaled linearly to plot it close to the others.

Mean NMD Mean RSNOD‘•’ denotes the minimum score of mean RSNOD over types

Baseline(α = 0.0)

0.07250.1174of quality score for the validation data. We use the models

Diﬀ+(α = 0.5)

0.07130.1146

only Diﬀ (α = 1.0)

0.07650.1221

which are trained until the global step of ‘•’ to predict the
distributions of the testing data.

Table 4 Chinese Results (E-score)
Model

Mean NMD Mean RSNOD

Baseline(α = 0.0)

0.0756

0.1178Diﬀ+(α = 0.5)

0.07190.1129

only Diﬀ (α = 1.0)

0.07980.1240

Table 5 English Results (A-score)
Model

Mean NMD Mean RSNOD

Figure 8 The training and validation loss
5. 3 Result
As evaluation measures, we use NMD and RSNOD (theBaseline(α = 0.0)

0.09110.1307Diﬀ+(α = 0.5)

0.09440.1333

only Diﬀ (α = 1.0)

0.09390.1334lower the better). We show the evaluation scores for STC-3
DQ subtask in terms of A-score, S-score and E-score of each

Table 6 English Results (S-score)language in Tables 2 to 7 where “Baseline” denotes the baseModelline BLSTM model of STC-3, “Diﬀ+” denotes our proposedmodel and “only Diﬀ” denotes our model which use α of 1.0.
In Tables 2 to 7, the top scores are shown in bold. We con-

Mean NMD Mean RSNOD

Baseline(α = 0.0)

0.08530.1301Diﬀ+(α = 0.5)

0.08440.1295

only Diﬀ (α = 1.0)

0.08630.1314ducted randomised Tukey HSD tests with B = 10, 000 trials [7]. For English dataset, any models do not statisticallysignificantly outperformed another at α = 0.05, therefore weTable 7 English Results (E-score)

show p-values for only Chinese dataset in Tables 8 to 13.

Model

Eﬀect sizes (i.e., standardized mean diﬀerences) based on
one-way ANOVA (without replication) [9] are also shown.
From the results with the evaluation measures for the DQ
subtasks, it can be observed that:•

Our proposed model statistically significantly outper-

formed the baseline BLSTM model at α = 0.05 in terms of
RSNOD of E-score for Chinese dataset,•

Our proposed model outperformed the baseline

BLSTM model in terms of every evaluation measure for Chinese dataset,

Mean NMD Mean RSNOD

Baseline(α = 0.0)

0.08350.1256Diﬀ+(α = 0.5)

0.08430.1265

only Diﬀ (α = 1.0)

0.08560.1275

Table 8 Statistical significance in terms of NMD (Chinese,
A-score)“only Diﬀ”. When we select a model in terms of global stepof training, we use mean RSNOD over types of quality score

Diﬀ+

only Diﬀ

Baseline p = 0.747(0.0257) p = 0.183(−0.0632)
Diﬀ+-

p = 0.023(−0.0889)

Table 9 Statistical significance in terms of RSNOD (Chinese, A-score)model using the validation data.
In this study, we simply add two loss functions in Eq. (3).
We did not plot two training losses separately. Moreover, we
can update parameters of a model using each loss function
alternately. We have to analyze how to combine two loss

Diﬀ+

only Diﬀ

Baseline p = 1.000(0.0050) p = 0.014(−0.0791)
Diﬀ+

described in section 5. 2. We have to analyze how to select a-p = 0.000(−0.0842)

Table 10 Statistical significance in terms of NMD (Chinese,functions.

7 Conclusion and Future Work
We tackled the STC-3 DQ subtask and in this subtask,
the evaluation measures, i.e., NMD and RSNOD, which con-

S-score)Diﬀ+

only Diﬀ

sider ordinal probability bins are used, therefore, we pro-Baseline p = 0.718(0.0237) p = 0.073(−0.0793)

posed the loss function which considers adjacent probability

Diﬀ+

bins. We compared the baseline model of STC-3 DQ sub--p = 0.000(−0.1031)

task to our proposed model which utilize our proposed loss
Table 11 Statistical significance in terms of RSNOD (Chinese, S-score)model in terms of every measure for Chinese dataset. More-Diﬀ+

only Diﬀ

Baseline p = 0.213(0.0407) p = 0.016(−0.0670)
Diﬀ+

function and our proposed model outperformed the baseline-p = 0.016(−0.1077)

Table 12 Statistical significance in terms of NMD (Chinese,over, our proposed model statistically significantly outperformed the baseline model at α = 0.05 in terms of RSNOD
of E-score for Chinese dataset.
In the future, we analyze the diﬀerences between the architecture of the baseline model for Chinese and English dataset

E-score)Diﬀ+

only Diﬀ

in order to make our proposed loss function work for En-

Baseline p = 0.085(0.0785) p = 0.029(−0.0888)

glish dataset. We also do experiments using the oﬃcial test

Diﬀ+

dataset of STC-3 DQ subtask.

-

p = 0.000(−0.1673)

Table 13 Statistical significance in terms of RSNOD (Chinese, E-score)Diﬀ+

only Diﬀ

Baseline p = 0.032(0.0811) p = 0.006(−0.1043)
Diﬀ+-

p = 0.000(−0.1854)

6 Discussion
Our proposed loss function improved the baseline model
for Chinese dataset; however, it did not improve the baseline model for English dataset. The texts of English dataset
were translated manually from Chinese dataset and the samegold labels were used to calculate gold distributions of dialogues. Therefore, we think the causes are derived from thearchitecture of the baseline model. We need to know thediﬀerences between the baseline model for Chinese and English dataset. For example, the pre-trained word embedding
matrix for Chinese and English dataset diﬀer. We have toanalyze the vocabulary cover rate of each word embedding
matrix for each dataset.
In Table 5, the order of models in terms of mean NMD
and mean RSNOD diﬀer; namely, “Diﬀ+” was defeated by

References[1] Higashinaka, R., Funakoshi, K., Inaba, M., Tsunomori, Y.,
Takahashi, T. and Kaji, N.: Overview of Dialogue Breakdown Detection Challenge 3, Proceedings of Dialog System
Technology Challenge 6 (DSTC6) Workshop (2017).
[2] Hochreiter, S. and Schmidhuber, J.: Long Short-term Memory, Neural computation, Vol. 9, No. 8, pp. 1735–1780(1997).
[3] Huang, M., Zeng, Z., Kato, S. and Sakai, T.: NTCIR-14
Short Text Conversation Task (STC-3), http://sakailab.
com/ntcir14stc3/ (2018).
[4] Janocha, K. and Czarnecki, W.: On Loss Functions for Deep
Neural Networks in Classification (2017).
[5] Perez, J., Boureau, Y.-L. and Bordes, A.: Dialog System &Technology Challenge 6 Overview of Track 1 - End-to-End
Goal-Oriented Dialog learning, Proceedings of Dialog System Technology Challenge 6 (DSTC6) Workshop (2017).
[6] Rubinstein, R.: The Cross-Entropy Method for Combinatorial and Continuous Optimization, Methodology And Computing In Applied Probability, Vol. 1, No. 2, pp. 127–190(1999).
[7] Sakai, T.: Metrics, Statistics, Tests, PROMISE WinterSchool 2013: Bridging between Information Retrieval andDatabases (LNCS 8173), pp. 116–163 (2014).
[8] Sakai, T.: Comparing Two Binned Probability Distributions for Information Access Evaluation, Proceedings of The
41st International ACM SIGIR Conference on Research& Development in Information Retrieval, pp. 1073–1076(2018).

[9] Sakai, T.: Laboratory experiments in information retrieval: Sample sizes, eﬀect sizes, and statistical power,
Springer (2018). https://link.springer.com/book/10.
1007/978-981-13-1199-4.
[10] Schuster, M. and Paliwal, K. K.: Bidirectional Recurrent
Neural Networks, IEEE Trans. Signal Processing, Vol. 45,
No. 11, pp. 2673–2681 (1997).
[11] Zeng, Z., Luo, C., Shang, L., Li, H. and Sakai, T.: Test Collections and Measures for Evaluating Customer-Helpdesk
Dialogues, Proceedings of The 8th International Workshop
on Evaluating Information Access, pp. 1–9 (2017).