NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Overview of the NTCIR-16
Dialogue Evaluation (DialEval-2) Task
Sijie Tao

Tetsuya Sakai

Waseda University, Japantsjmailbox@ruri.waseda.jp

Waseda University, Japantetsuyasakai@acm.org
Table 2: The Statistics of Participant Runs in Each Subtask.

ABSTRACTThis paper provides an overview of the NTCIR-16 Dialogue Evaluation (DialEval-2) task. DialEval-2 is the successor of the NTCIR-15
DialEval-1 task and the NTCIR-14 Short Text Conversation STC3 task. DialEval-2 consists of two subtasks: the Dialogue Quality(DQ) subtask and the Nugget Detection (ND) subtask. Both of thesubtasks are designed to aim automatical evaluation of customerhelpdesk dialogues. The DQ subtask requires our participants toestimate three kinds of quality score for each dialogue: task accomplishment, customer satisfaction, and dialogue effectiveness.
The ND subtask is set as a classification task, where participants
are asked to classify every turn of a dialogue to detect nugget turns.
A nugget stands for a turn being helpful for problem solving in the
dialogue. In this paper, we introduce the task definition, data collection, evaluation measures, and the official evaluation results on
the runs from the participant teams.

This paper is organised as follows. Section 2 and Section 3 describe task definition and evaluation methods, respectively. Section 4 introduces the data collection. Section 5 presents the official
evaluation results. Finally, Section 6 concludes this paper.

1

2 TASK DEFINITIONTeams

INTRODUCTION

The DialEval-1 task was lauched at NTCIR-15. The motivation of
the task is to explore ideas of automatically evaluating customerhelpdesk dialogues. Nowadays, manual evaluation by human annotators can be expensive and time assuming. If we can automatically evaluate the quality of a dialogue and find out which turns arehelpful, it will be more efficient and economical than evaluating byhuman annotators. Therefore, we continue this task at NTCIR-16.
The important dates of DialEval-2 are shown in Table 1.
DialEval-2 is the succesor of DialEval-1 at NTCIR-15 in 2020 [16].
We continue running DQ and ND subtasks. At DialEval-2, we use
the DCH-2 dataset [17]. The part of this dataset was first provided
at DialEval-1. With the translation of the Chinese collection being fully completed, the DCH-2 dataset was released as a ChineseEnglish parallel corpus. The DCH-2 dataset is used as training anddevelopment set, and a new test set is built for DialEval-2. There
are four teams participating in DialEval-2: IMNTPU [4], NKUST [1],RSLDE [5], and TUA1 [2]. The statistics of participant runs in eachsubtask are shown in Table 2.

Content

Dec 1 2021
Jan 15 2022
Feb 1 2022
March 1 2022
May 1 2022
June 14-17 2022

Test data released
Run submissions due
Evaluation Results and draft overview released
Draft participant paper submissions due
All camera-ready paper submissions due
NTCIR-16 Conference

ChineseEnglish

DQ

ND

DQ

ND

IMNTPU
NKUST
RSLDE
TUA1

1
2
3
3

1
1
2
3

0
2
3
2

1
0
2
1

1
1
3
1

Total

9

7

7

4

6

The task definition of DialEval-2 is identical to that of DialEval1. Hence this section is largely a duplicate of the task definition
section of the DialEval-1 overview paper [16].
The goal of DialEval-2 is to explore approaches to evaluatingtask-oriented, multi-round, textual helpdesk-customer dialogue systems automatically. Identical to DialEval-1, there are two subtasks:(1) Dialogue Quality (DQ) subtask, which is to assign quality scores
to each dialogue in terms of three subjective criteria: task accomplishment, customer satisfaction, and dialogue effectiveness; and(2) Nugget Detection (ND) subtask is to classify whether a customer or helpdesk turn is a nugget, where being a nugget meansthat the turn helps towards problem solving. This section details
what a customer-helpdesk dialogue is, followed by the definitionsof the two subtasks.

2.1

Customer-Helpdesk Dialogue

In DQ and ND subtasks, a customer-helpdesk dialogue is a multiround and textual dialogue that has two speakers: a Customer anda Helpdesk. The Customer usually comes with a problem and the
helpdesk should try to help the customer to solve it. An example ofa Customer-Helpdesk dialogue is shown in Figure 1: this is a tworound dialogue (i.e., there are two Customer-Helpdesk exchanges).
It can be observed that it is initiated by Customer’s report of a particular problem she is facing, which we call a trigger. This is an
example of a successful dialogue, for Helpdesk provides an actual
solution to the problem and Customer acknowledges that the problem has been solved.
We used the turn as the basis for measuring the length of a dialogue, formed by merging all consecutive posts by the same utterer. For example, if each Customer post is denoted by 𝑝𝐶 and

Table 1: Schedule of DialEval-2 at NTCIR-16
Time

Runs

51

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Conference’17, July 2017, Washington, DC, USA

Sijie Tao and Tetsuya Sakai

Customer

C: The Smartisan App Store of my mobile
phone has been disabled for nearly half a monthand the system couldn't be updated. The network
was normal. Please give me an explanation.
2016-5-22 13:45
Trigger

H: To ensure information security, we updated
the system security encryption algorithm. Please
visit the website, and download and install"System Update Service" to update your system.
For detailed operations, please visit the link
2016-5-22 13:56

Helpdesk

Solution

C: It worked properly. Thank you!
2016-5-22 23:40

Customer

Confirmation
Helpdesk

H: You are welcome
2016-5-22 23:50

Figure 1: An example of a dialogue between Customer (C) and Helpdesk (H). The left part is the translated dialogue and theright part is the screenshot of the original dialogue on Weibo [16].
each helpdesk post is denoted by 𝑝𝐻 , a dialogue of the form
[𝑝𝐶 , 𝑝𝐶 , 𝑝𝐶 , 𝑝𝐻 , 𝑝𝐻 , 𝑝𝐻 , 𝑝𝐶 , 𝑝𝐶 ]will be regarded as three turns, [𝑏𝐶 , 𝑏𝐻 , 𝑏𝐶 ], where 𝑏𝐶 is a Customer turn and 𝑏𝐻 is a Helpdesk one. This dialogue is considered
as a three-turn dialogue.

2.2

Dialogue Quality (DQ) Subtask

In Dialogue Quality (DQ) subtask, we want to obtain the subjectivescores for each dialogue automatically to quantify the quality of a
dialogue as a whole. Specifically, we introduce three quality scoresfor three different criteria:

Figure 2: Task accomplishment as state transitions, and therole of a nugget [16].

A-Score : Task Accomplishment (Has the problem been solved?
To what extent?)
S-score : Customer Satisfaction of the dialogue (not of the product/service or the company)
E-score : Dialogue Effectiveness (Do the utterers interact effectively to solve the problem efficiently?)(3) It helps Customer transition from Current State (including Initial State) towards Target State (i.e., when the problem is solved).
Compared to traditional nugget-based information access evaluation, there are two unique features in nugget-based helpdesk dialogue evaluation:

For each of them, possible options are [2, 1, 0, −1, −2]. In otherwords, participants are required to assign a score from 2 to −2 foreach of these criteria to each dialogue.

2.3• A dialogue involves two parties, Customer and Helpdesk;• Even within the same utterer, nuggets are not homogeneous,by which we mean that some nuggets may play special roles.
In particular, since the dialogues we consider are task-oriented(but not closed-domain, which makes slot filling approaches infeasible), there must be some nuggets that represent the state ofidentifying the task and those that represent the state of accomplishing it.

Nugget Detection (ND) Subtask

In Nugget Detection (ND) subtask, participants are required to identify nuggets for each dialogue, where a nugget is a turn that helps
the Customer transition from the current state (where the problem
is yet to be solved) towards the target state (where the problem has
been solved). Figure 2 reflects our view that accumulating nuggets
will eventually solve Customer’s problem. The official definition ofnuggets is (1) A nugget is a turn by either Helpdesk or Customer;(2) It can neither partially nor wholly overlap with another nugget;

Based on the above considerations, we defined the following
four mutually exclusive nugget types:

52

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Overview of the NTCIR-16
Dialogue Evaluation (DialEval-2) Task

Conference’17, July 2017, Washington, DC, USA

CNUG0

Customer’s trigger nuggets. These are nuggets thatthe dialogue [8]. We evaluate these subtasks by comparing thedefine Customer’s initial problem, which directly
probability distribution estimated by the participants with the golden
caused Customer to contact Helpdesk.
standard distribution, where the golden standard distribution is
HNUG
Helpdesk’s regular nuggets. These are nuggets in Helpdesk’s
calculated by annotators’ vote over the classes (i.e. 2 to −2 for DQ
turns that are useful from Customer’s point of view.
subtask and CNUG, HNUG, etc. for ND subtask).
CNUG
Customer’s regular nuggets. These are nuggets in
We now formalise the metrics for comparing two probability
Customer’s turns that are useful from Helpdesk’s
distributions. Let A denote a given set of classes, e.g., 𝐴 = 2, 1, 0, −1, −2
point of view.
for DQ subtask, and let 𝐿 = |𝐴|. Let 𝑝 (𝑖)(𝑖 = 1, . . . , 𝐿) denote the∑
HNUG∗
Helpdesk’s goal nuggets. These are nuggets in Helpdesk’s system estimated probability for class 𝑖, so that 𝑖 ∈𝐴 𝑝 (𝑖) = 1. Simturns which provide the Customer with a solution
ilarly, let 𝑝 ∗ (𝑖) denote the corresponding true probability, where∑∗to the problem.
𝑖 ∈𝐴 𝑝 (𝑖) = 1.
CNUG∗
Customer’s goal nuggets. These are nuggets in Customer’s turns which tell Helpdesk that Customer’s
3.1 Evaluation Metrics for Dialogue Quality
problem has been solved.
Subtask
CNAN
Customer’s not-a-nugget. It means that the current
Since the classes of DQ subtask are non-nominal, cross-bin metcustomer turn does not help towards problem solvrics are more suitable than bin-by-bin metrics. As discussed bying.
Sakai [9][12][11], bin-by-bin metrics such as Jensen-Shannon DiHNAN
Helpdesk’s not-a-nugget. It means that the current
vergence (See Section 3.2) are not adequate for this subtask as
helpdesk turn does not help towards problem solvthey do not consider the distance between classes. Thus, we utiliseing.
two cross-bin metrics: Normalised Match Distance (NMD) and RootIn the ND subtask, participants are required to predict a nugget
Symmetric Normalised Order-aware Divergence (RSNOD).
type for each turn in dialogues. Note that each nugget type may
3.1.1 Normalised Match Distance (NMD). is a normalised version
or may not be present in a dialogue, and multiple nuggets of theof Match Distance (MD), where MD is a special case of Earth Mover’s
same type may be present in a dialogue.
Distance where the probabilities add up to one and the number of∑∑bins are a given [6]. Let 𝑐𝑝 (𝑖) = 𝑖𝑘=1 𝑝 (𝑘), and 𝑐𝑝 ∗ (𝑖) = 𝑖𝑘=1 𝑝 ∗ (𝑘).
2.4 Chinese and English Subtasks
MD is just the sum of absolute errors compared from the cumulaThe dialogues are originally in Chinese, and part of them are mantive probability distributions:ually translated into English for DialEval-1. Thanks to the release∑of DCH-2 corpus, every Chinese dialogue has an English transla|𝑐𝑝 (𝑖) − 𝑐𝑝 ∗ (𝑖)|.
(1)MD(𝑝, 𝑝 ∗ ) =tion at DialEval-2. Thus, each subtask has a Chinese version and𝑖 ∈𝐴an English version, and the participants can choose the language
Then, the normalised version NMD is calculated as follows:
to build their runs.
𝑀𝐷 (𝑝, 𝑝 ∗ )NMD(𝑝, 𝑝 ∗ ) =(2)
𝐿−1
2.5 Baselines
3.1.2 Root Symmetric Normalised Order-aware Divergence (RSNOD).
is a metric that considers the distance between a pair of bins more
explicitly than NMD does [9]. First, a distance-weighted sum ofsquares (DW) is defined for each bin:∑
DW(𝑖) =|𝑖 − 𝑗 |(𝑝 ( 𝑗) − 𝑝 ∗ ( 𝑗)) 2 .
(3)

The baseline models are exactly the same as the baselines at DialEval1. There are three baseline models for each language and each subtask as follows;
BL-lstm (Baseline-run0) A baseline model1 which leverages Bidirectional Long Short-term Memory [3, 14];BL-uniform (Baseline-run1) A baseline model which always predict the uniform distribution;
BL-popularity (Baseline-run2) A baseline model which predicts
the probability of the most popular label as one, and predicts
other labels as 0. Note that the it accesses the golden truth
to find the most popular label. This baseline is to show theupper bound of a single label.

3

𝑗 ∈𝐴

Let 𝐵 ∗ = {𝑖 | 𝑝 ∗ (𝑖) > 0}, that is, the set of bins where the gold
probabilities are positive. Order-Aware Divergence (OD) is the DW
averaged over these non-empty gold bins:
1 ∑𝑂𝐷 (𝑝 ∥ 𝑝 ∗ ) = ∗𝐷𝑊 (𝑖)
(4)
|𝐵 |
∗
𝑖 ∈𝐵

Similarly, let 𝐵 = {𝑖 | 𝑝 (𝑖) > 0}. Just as the symmetric JSD isobtained from KLD, Symmetric OD can be defined by swapping
the system and gold distributions:

EVALUATION METHODSThe evaluation measures of DialEval-2 are the same as those of
DialEval-1. Hence this setion is largely a duplicate of the Evaluation Methods section of the DialEval-1 overview paper [16].
Evaluating such a customer-helpdesk dialogue is even subjective and difficult for human, and often there is no such thing as theground truth: different people may have different opinions about

OD(𝑝, 𝑝 ∗ ) + OD(𝑝 ∗, 𝑝)
2
Finally, we define the Root Symmetric Normalised OD:√
SOD(𝑝, 𝑝 ∗ )∗RSNOD(𝑝, 𝑝 ) =𝐿−1
SOD(𝑝, 𝑝 ∗ ) =1 https://github.com/DialEval-2/LSTM-baseline

53

(5)

(6)

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Conference’17, July 2017, Washington, DC, USA

Sijie Tao and Tetsuya Sakai

In the DQ subtask, we use both NMD and RSNOD as metrics
to evaluate participants’ runs. Extensive experiments on ordinalquantification measures such as NMD and RSNOD have been reported in Sakai [11–13].

3.2

to annotate the dialogues. Following the same annotation instructions as DialEval-1, each dialogue is annotated by each annotatorindependently. Besides the annotation, all the test dialogues aremanually translated into English. We hired a professional company
to complete the translation. As the translation does not change
the semantic information of the dialogues, the English dialogues
share the same annotation with the Chinese ones. Thus, we havea Chinese-English parallel test set for DialEval-2.

Evaluation Metrics for Nugget Detection
Subtask

In contrast to DQ subtask, the classes in ND subtask are nominal,
so bin-by-bin metrics are more suitable. Specifically, two metrics
are used in ND subtask: Root Normalised Sum of Squares (RNSS)
and Jensen-Shannon Divergence (JSD).

5 RESULTS
5.1 Chinese Subtasks
First, in order to identify the differences between the evaluation
metrics, for each subtask, all the runs are ranked by the two evaluation metrics respectively, and then we calculate the ranking correlation using Kendall’s 𝑡𝑎𝑢 between the two rankings, as well as
their 95% confidence intervals.2 The results are shown in In Table 8. It can be observed that the difference between different metrics are not statistically significant for both ND and DQ subtaks.
This finding is consistent with what we observed at DialEval-1 andSTC-3.
Tables 4 to 6 shows the mean evaluation scores of the DQ subtask in terms of A-score, S-score, and E-score respectively, and Table 7 shows the mean evaluation scores of the ND subtask. We
conducted randomised Tukey HSD tests using the Discpower tool
3 with 𝐵 = 5, 000 trials [7]. Tables 14 to 21 summarise the statistical significance test results, p-values, and effect sizes computedby Randomised Tukey HSD (i.e., standardised mean differences)
based on one-way ANOVA (without replication) [10]. From the result shown in Table 18, it can be observed that only TUA1-run0
statistically significantly outperforms all the baseline systems in
Chinese DQ task in terms of NMD for E-score. We also find thatin Chinese ND task, RSLDE-run0 is the only system which canoutperform Baseline-run0 in terms of both JSD and RNSS, but thedifference between them is not statistically significant.

3.2.1 Root Normalised Sum of Squares (RNSS) . is defined as follows:√∑∗2
𝑖 ∈𝐴 (𝑝 (𝑖) − 𝑝 (𝑖))RNSS =(7)2
3.2.2 Jensen-Shannon Divergence (JSD). Let 𝑝 𝑀 (𝑖) =JSD is defined as:

𝑝 (𝑖)+𝑝 ∗ (𝑖),2

KLD(𝑝 ∥ 𝑝 𝑀 ) + KLD(𝑝 𝑀 ∥ 𝑝 ∗ )2∑
𝑝 1 (𝑖)where KLD(𝑝 1 ∥ 𝑝 2 ) =𝑝 1 (𝑖) log2
𝑝 2 (𝑖)JSD(𝑝 ∥ 𝑝 ∗ ) =

(8)
(9)

𝑖 𝑠.𝑡 . 𝑝 1 (𝑖) >0

Since there are multiple turns in each dialogue and participants
are required to predict a probability distribution for each turn in
ND subtask, we need to combine the two evaluation scores intoa single one for each dialogue. Specifically, we calculate the average metric score for customer’s turns 𝑆𝐶 and helpdesk’s turns 𝑆𝐻
separately, and then a weighted sum 𝑆 𝑁 𝐷 = 𝛼𝑆𝐶 + (1 − 𝛼)𝑆𝐻 will
be used as the final evaluation score for each dialogue, where 𝛼 isa parameter that controls the relatively importance between customers’ nuggets and helpdesk’ nuggets. We let 𝛼 = 0.5 throughout
this paper.

4 DATA COLLECTION
4.1 Training and Development Data

5.2

The statistics of the DialEval-2 data collection is shown in Table 3.
We use the DCH-2 data collection [15] for training and development, as part of DCH-2 was utilised at NTCIR-15 DialEval-1 task.
The DCH-2 data collection consists of real (i.e., human-human)customer-helpdesk dialogues collected from Weibo, and there are
4,390 Chinese-English parallel dialogues. At DialEval-2, 4,090 pairsof dialogues from DCH-2, which was also partially used as training and development set at DialEval-1, are used as training set. The
other 300 pairs of dialogues which was the test set at DialEval-1,
are used as develeopment set.

4.2

English Subtasks

Tables 9 to 11 shows the mean evaluation scores of the DQ subtask
in terms of A-score, S-score, and E-score respectively, and Table 12shows the mean evaluation scores of the ND subtask.
Following the Chinese subtasks, we also conduct randomised
Tukey HSD tests for English runs, and Tables 22 to 29 summarise
the significance test results, along with p-values and effect sizes.
From the results of the DQ subtask, it can be observed that TUA1run0 is the only system that outperformed all the baseline systems on average, but the differences are not statistically significant. From the results of the ND subtask, we find that RSLDE-run0
and IMNTPU-run0 outperform Baseline-run0, but the their gains
are not statistically significant either. In Table 13, we also rank all
the participant systems according to the two evaluation metricsof each subtasks, and then compute Kendall’s 𝑡𝑎𝑢 of the rankings.

Test Data

In order to construct the test set, we sampled 65 dialogues fromthe data we have crawled from Weibo. According to topic size design [10] and the residual variances obtained from the previous
DialEval-1 task, 62 dialogues is enough to ensure a statistical powerof 80% with a minimum detectable range of 0.05 for ANOVA with 10
systems at the 5% significance level. We hired 20 Chinese students
from the Faculty of Science and Engineering at Waseda University

2 We calculate the confidence intervals using kendall.ci function of the NSM3 pack-age (https://www.rdocumentation.org/packages/NSM3/) with the following options;alpha=0.05, bootstrap=T, B=10,000.
3 http://research.nii.ac.jp/ntcir/tools/discpower-en.html

54

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Overview of the NTCIR-16
Dialogue Evaluation (DialEval-2) Task

Conference’17, July 2017, Washington, DC, USA

Table 3: Statistics of DialEval-2 Data collection. The unit of post/turn length is char for Chinese and token for English.

Training
SourceData timestamps

ChineseDev

Test

Training

English
Dev

Test

DCH-2

DCH-2

Weibo

Jan. 2013 ∼Apr. 2018

Apr. 2018 ∼Jul. 2019

Apr. 2018 ∼Jul. 2019

Jan. 2013 ∼Apr. 2018

Apr. 2018 ∼Jul. 2019

Apr. 2018 ∼Jul. 2019

4,090

300

65

4,090

300

65

19

20

20

19

20

20

#dialogues#annotators/dialogue

Translation

Quality annotation criteria

A-score, E-score, S-score (See Section 2.2)

Nugget types

CNUG0, CNUG, HNUG, CNUG∗, HNUG∗ (See Section 2.3)

Table 4: Chinese Dialogue Quality (A-score) Results

Table 6: Chinese Dialogue Quality (E-score) Results

Run

Run

Mean RSNODTUA1-run2
TUA1-run1
TUA1-run0
Baseline-run0
Baseline-run2
RSLDE-run0
RSLDE-run1
IMNTPU-run0
Baseline-run1
NKUST-run0

0.19920.20920.21540.23010.23200.24380.2446
0.24790.27670.2774Run
TUA1-run2
TUA1-run1
TUA1-run0
RSLDE-run0
RSLDE-run1
Baseline-run2
IMNTPU-run0
Baseline-run0
NKUST-run0
Baseline-run1

Mean NMD
0.13250.13690.14740.15370.15510.15770.16180.17720.24530.2500Mean RSNODTUA1-run0
TUA1-run1
RSLDE-run0
TUA1-run2
RSLDE-run1
Baseline-run0
IMNTPU-run0
NKUST-run0
Baseline-run1
Baseline-run2

0.15450.16470.16600.16710.17250.18540.18600.22530.24960.2569Run
TUA1-run0
RSLDE-run0
TUA1-run1
RSLDE-run1
TUA1-run2
IMNTPU-run0
Baseline-run0
Baseline-run2
NKUST-run0
Baseline-run1

Mean NMD
0.11360.12220.12620.12860.13100.14270.15790.17100.18970.2106

Table 5: Chinese Dialogue Quality (S-score) Results
Table 7: Chinese Nugget Detection Results
Run

Mean RSNODTUA1-run2
TUA1-run1
TUA1-run0
RSLDE-run0
RSLDE-run1
Baseline-run0
IMNTPU-run0
Baseline-run2
NKUST-run0
Baseline-run1

0.17580.18400.18840.19380.19640.19980.20320.20620.27320.2959Run
TUA1-run1
TUA1-run2
RSLDE-run1
RSLDE-run0
Baseline-run2
TUA1-run0
IMNTPU-run0
Baseline-run0
NKUST-run0
Baseline-run1

Mean NMD
Run

0.11590.11660.12290.12430.12880.13050.13150.15230.22930.2565

The results show that the measures are highly positively correlated. This is also consistent with what we observed at previous
tasks.

5.3

Mean JSDRSLDE-run0
Baseline-run0
RSLDE-run2
RSLDE-run1
NKUST-run0
TUA1-run0
Baseline-run2
Baseline-run1
NKUST-run1
TUA1-run1

0.05600.05850.06070.06340.06700.07000.18640.20420.24320.2909Run
RSLDE-run0
Baseline-run0
RSLDE-run1
RSLDE-run2
NKUST-run0
TUA1-run0
Baseline-run2
Baseline-run1
NKUST-run1
TUA1-run1

Mean RNSS
0.16040.16510.17120.17200.17610.17800.29010.33710.3774
0.39396 CONCLUSION
This paper provides an overview of the NTCIR-16 DialEval-2 task.
The overview describes the task definition, data collection, evaluation metrics, and the official evaluation results of DialEval-2. Fromthe evaluation results, we observe that only one run from TUA1
outperform the LSTM baseline significantly in Chinese DQ task
in terms of NMD for E-score. In other subtasks, none of the runs
can outperform the LSTM baseline significantly. Moreover, no substantial difference is observed between the evaluation mectrics foreach subtasks.

Top Runs

TUA1’s runs are the top runs of the DQ subtask, and RSLDE and
IMNTPU are the top runs of the ND subtask. The common feature of the top runs is that large scale pretrained language models
play important roles in their systems. TUA1 conducted their experiments with several pre-trained Transformer models, such as BERT
and RoBERTa [2]. RSLDE uses XLNet to build their top run [5], and
IMNTPU chooses XLM-RoBERTa to tackle our task [4].

55

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Conference’17, July 2017, Washington, DC, USA

Sijie Tao and Tetsuya Sakai

Table 8: Ranking Correlation between of Chinese runs
ranked by two different metrics (Kendall’s 𝑡𝑎𝑢 with 95% CIs)

Table 12: English Nugget Detection Results
Run

Dialogue Quality (A-score)
NMD vs RSNOD

0.689

RSLDE-run0
IMNTPU-run0
Baseline-run0
NKUST-run0
RSLDE-run2
RSLDE-run1
TUA1-run0
Baseline-run2
Baseline-run1[−0.189, 1.000]

Dialogue Quality (S-score)
NMD vs RSNOD

0.644[0.300, 1.000]

Dialogue Quality (E-score)
NMD vs RSNOD

0.778[0.538, 1.000]

0.956[0.706, 1.000]

Mean JSD
0.05570.0601
0.06250.0641
0.0676
0.06910.07280.18640.2042

Run

Mean RNSSIMNTPU-run0
RSLDE-run0
Baseline-run0
NKUST-run0
RSLDE-run2
TUA1-run0
RSLDE-run1
Baseline-run2
Baseline-run1

0.15740.16150.17220.17440.17780.18300.18530.29010.3371

Nugget Detection
JSD vs RNSS

Table 13: Ranking Correlation between of English runs
ranked by two different metrics (Kendall’s 𝑡𝑎𝑢 with 95% CIs)

Table 9: English Dialogue Quality (A-score) Results
Dialogue Quality (A-score)Run
TUA1-run0
Baseline-run2
Baseline-run0
IMNTPU-run0
RSLDE-run0
RSLDE-run1
Baseline-run1

Mean RSNOD
0.19670.23200.23210.25350.26150.27250.2767Run
TUA1-run0
Baseline-run2
IMNTPU-run0
Baseline-run0
RSLDE-run1
RSLDE-run0
Baseline-run1

Mean NMD

NMD vs RSNOD

0.13270.15770.16540.17800.1896
0.19570.2500TUA1-run0
Baseline-run0
IMNTPU-run0
Baseline-run2
RSLDE-run0
RSLDE-run1
Baseline-run1

Mean RSNOD
0.1855
0.19860.20200.20620.20780.21540.2959Run
TUA1-run0
Baseline-run2
IMNTPU-run0
RSLDE-run0
RSLDE-run1
Baseline-run0
Baseline-run1

NMD vs RSNODTUA1-run0
Baseline-run0
IMNTPU-run0
RSLDE-run0
RSLDE-run1
Baseline-run1
Baseline-run2

Mean RSNOD
0.17420.17450.18260.18320.18890.24960.2569Run
TUA1-run0
IMNTPU-run0
RSLDE-run0
Baseline-run0
RSLDE-run1
Baseline-run2
Baseline-run1

0.524[−0.059, 1.000]

Dialogue Quality (E-score)
NMD vs RSNOD

0.714[−0.059, 1.000]

0.889[0.613, 1.000]

Nugget Detection
JSD vs RNSS

Mean NMD[2] Fei Ding, Kang Xin, Yunong Wu, and Fuji Ren. 2022. TUA1 at the NTCIR-16
DialEval-2 Task. In Proceedings of NTCIR-16. to appear.
[3] Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-term Memory. Neural computation 9, 8 (1997), 1735–1780.
[4] Ting-Yun Hsiao, Yung-Wei Teng, Pei-Tz Chiu, Mike Tian-Jian Jiang, and MinYuh Day. 2022. IMNTPU Dialogue System Evaluation at the NTCIR-16 DialEval2 Dialogue Quality and Nugget Detection. In Proceedings of NTCIR-16. to appear.
[5] Fan Li and Tetsuya Sakai. 2022. RSLDE at the NTCIR-16 DialEval-2 Task. In
Proceedings of NTCIR-16. to appear.
[6] Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas. 2000. The earth mover’s
distance as a metric for image retrieval. International journal of computer vision40, 2 (2000), 99–121.
[7] Tetsuya Sakai. 2014. Metrics, Statistics, Tests. In PROMISE Winter School 2013:
Bridging between Information Retrieval and Databases (LNCS 8173). 116–163.
[8] Tetsuya Sakai. 2017. Towards Automatic Evaluation of Multi-Turn Dialogues: A
Task Design that Leverages Inherently Subjective Annotations. In Proceedingsof EVIA 2017.
[9] Tetsuya Sakai. 2018. Comparing Two Binned Probability Distributions for Information Access Evaluation. In The 41st International ACM SIGIR Conference on
Research & Development in Information Retrieval (Ann Arbor, MI, USA) (SIGIR’18). ACM, New York, NY, USA, 1073–1076.
[10] Tetsuya Sakai. 2018.
Laboratory experiments in information retrieval: Sample sizes, effect sizes, and statistical power.
Springer.
https://link.springer.com/book/10.1007/978-981-13-1199-4.
[11] Tetsuya Sakai. 2021.
A Closer Look at Evaluation Measures for Ordinal Quantification. In Proceedings of the CIKM 2021 Workshops.
http://ceur-ws.org/Vol-3052/paper21.pdf[12] Tetsuya Sakai. 2021. Evaluating Evaluation Measures for Ordinal Classification and Ordinal Quantification. In Proceedings of ACL-IJCNLP 2021. 2759–2769.
https://aclanthology.org/2021.acl-long.214.pdf[13] Tetsuya Sakai. 2022.
On Variants of Root Normalised Order-aware
Divergence and a Divergence based on Kendall s Tau.
(2022).
https://arxiv.org/abs/2204.07304[14] Mike Schuster and Kuldip K. Paliwal. 1997. Bidirectional Recurrent Neural Networks. IEEE Trans. Signal Processing 45, 11 (1997), 2673–2681.
[15] Zhaohao Zeng, Sosuke Kato, and Tetsuya Sakai. 2019. Overview of the NTCIR-14
Short Text Conversation Task: Dialogue Quality and Nugget Detection Subtasks.
In Proceedings of NTCIR-14. 290–315.

0.12140.12880.13120.13810.14380.14670.2565

Table 11: English Dialogue Quality (E-score) Results
Run[0.091, 1.000]

Dialogue Quality (S-score)

Table 10: English Dialogue Quality (S-score) Results
Run

0.810

Mean NMD
0.13600.14000.14290.14310.14440.17100.2106ACKNOWLEDGEMENT
We thank the DialEval-2 participants and the NTCIR chairs formaking this task happen.

REFERENCES[1] Tao-Hsing Chang and Jian-He Chen. 2022. NKUST at the NTCIR-16 DialEval-2
Task. In Proceedings of NTCIR-16. to appear.

56

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Overview of the NTCIR-16
Dialogue Evaluation (DialEval-2) Task

Conference’17, July 2017, Washington, DC, USA[16] Zhaohao Zeng, Sosuke Kato, Tetsuya Sakai, and Inho Kang. 2020. Overview of
the NTCIR-15 Dialogue Evaluation (DialEval-1) Task. In Proceedings of NTCIR15. 13–34.
[17] Zhaohao Zeng and Tetsuya Sakai. 2021. DCH-2: A Parallel Customer-Helpdesk
Dialogue Corpus with Distributions of Annotators’ Labels. CoRR abs/2104.08755(2021). arXiv:2104.08755 https://arxiv.org/abs/2104.08755

57

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Conference’17, July 2017, Washington, DC, USA

APPENDIX A

Sijie Tao and Tetsuya Sakai

STATISTICAL SIGNIFICANCE TESTSTable 14: Statistical significance in terms of NMD (Chinese DQ subtask, A-score) calculated by Randomised Tukey HSD tests
Runsignificantly better than these runs

TUA1-run2

NKUST-run0
Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.198)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.247)

TUA1-run1

NKUST-run0
Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.151)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.200)

TUA1-run0

NKUST-run0
Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.040)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.089)

RSLDE-run0

NKUST-run0
Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.973)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.023)

RSLDE-run1

NKUST-run0
Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.958)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.007)

Baseline-run2

NKUST-run0
Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.930)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.980)

IMNTPU-run0

NKUST-run0
Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.887)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.937)

Baseline-run0

NKUST-run0
Baseline-run1(𝑝 = 0.0002, 𝐸𝑆 𝐸1 = 0.724)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.773)

Table 15: Statistical significance in terms of RSNOD (Chinese DQ subtask, A-score) calculated by Randomised Tukey HSD tests
Runsignificantly better than these runs

TUA1-run2

RSLDE-run1
IMNTPU-run0
Baseline-run1
NKUST-run0(𝑝 = 0.0442, 𝐸𝑆 𝐸1 = 0.455)(𝑝 = 0.0190, 𝐸𝑆 𝐸1 = 0.488)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.777)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.783)

TUA1-run1

Baseline-run1
NKUST-run0(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.677)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.683)

TUA1-run0

Baseline-run1
NKUST-run0(𝑝 = 0.0002, 𝐸𝑆 𝐸1 = 0.615)(𝑝 = 0.0002, 𝐸𝑆 𝐸1 = 0.621)

Baseline-run0

Baseline-run1
NKUST-run0(𝑝 = 0.0326, 𝐸𝑆 𝐸1 = 0.467)(𝑝 = 0.0292, 𝐸𝑆 𝐸1 = 0.474)

Baseline-run2

NKUST-run0(𝑝 = 0.0444, 𝐸𝑆 𝐸1 = 0.454)

58

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Overview of the NTCIR-16
Dialogue Evaluation (DialEval-2) Task

Conference’17, July 2017, Washington, DC, USA

Table 16: Statistical significance in terms of NMD (Chinese DQ subtask, S-score) calculated by Randomised Tukey HSD tests
Runsignificantly better than these runs

TUA1-run1

NKUST-run0
Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.293)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.603)

TUA1-run2

NKUST-run0
Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.285)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.595)

RSLDE-run1

NKUST-run0
Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.214)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.524)

RSLDE-run0

NKUST-run0
Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.197)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.507)

Baseline-run2

NKUST-run0
Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.146)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.456)

TUA1-run0

NKUST-run0
Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.126)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.437)

IMNTPU-run0

NKUST-run0
Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.116)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.426)

Baseline-run0

NKUST-run0
Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.878)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.188)

Table 17: Statistical significance in terms of RSNOD (Chinese DQ subtask, S-score) calculated by Randomised Tukey HSD tests
Runsignificantly better than these runs

TUA1-run2

NKUST-run0
Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.963)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.188)

TUA1-run1

NKUST-run0
Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.882)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.108)

TUA1-run0

NKUST-run0
Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.839)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.064)

RSLDE-run0

NKUST-run0
Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.785)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.010)

RSLDE-run1

NKUST-run0
Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.760)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.985)

Baseline-run0

NKUST-run0
Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.725)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.950)

IMNTPU-run0

NKUST-run0
Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.692)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.917)

Baseline-run2

NKUST-run0
Baseline-run1(𝑝 = 0.0004, 𝐸𝑆 𝐸1 = 0.663)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.888)

59

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Conference’17, July 2017, Washington, DC, USA

Sijie Tao and Tetsuya Sakai

Table 18: Statistical significance in terms of NMD (Chinese DQ subtask, E-score) calculated by Randomised Tukey HSD tests
Runsignificantly better than these runs

TUA1-run0

Baseline-run0
Baseline-run2
NKUST-run0
Baseline-run1(𝑝 = 0.0148, 𝐸𝑆 𝐸1 = 0.512)(𝑝 = 0.0002, 𝐸𝑆 𝐸1 = 0.662)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.879)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.120)

RSLDE-run0

Baseline-run2
NKUST-run0
Baseline-run1(𝑝 = 0.0040, 𝐸𝑆 𝐸1 = 0.563)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.779)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.020)

TUA1-run1

Baseline-run2
NKUST-run0
Baseline-run1(𝑝 = 0.0128, 𝐸𝑆 𝐸1 = 0.517)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.733)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.974)

RSLDE-run1

Baseline-run2
NKUST-run0
Baseline-run1(𝑝 = 0.0270, 𝐸𝑆 𝐸1 = 0.489)(𝑝 = 0.0002, 𝐸𝑆 𝐸1 = 0.706)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.947)

TUA1-run2

Baseline-run2
NKUST-run0
Baseline-run1(𝑝 = 0.0484, 𝐸𝑆 𝐸1 = 0.462)(𝑝 = 0.0002, 𝐸𝑆 𝐸1 = 0.678)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.919)

IMNTPU-run0

NKUST-run0
Baseline-run1(𝑝 = 0.0064, 𝐸𝑆 𝐸1 = 0.542)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.783)

Baseline-run0

Baseline-run1(𝑝 = 0.0008, 𝐸𝑆 𝐸1 = 0.608)

Table 19: Statistical significance in terms of RSNOD (Chinese DQ subtask, E-score) calculated by Randomised Tukey HSD tests
Runsignificantly better than these runs

TUA1-run0

NKUST-run0
Baseline-run1
Baseline-run2(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.850)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.142)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.230)

TUA1-run1

NKUST-run0
Baseline-run1
Baseline-run2(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.729)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.021)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.108)

RSLDE-run0

NKUST-run0
Baseline-run1
Baseline-run2(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.713)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.004)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.092)

TUA1-run2

NKUST-run0
Baseline-run1
Baseline-run2(𝑝 = 0.0002, 𝐸𝑆 𝐸1 = 0.699)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.991)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.079)

RSLDE-run1

NKUST-run0
Baseline-run1
Baseline-run2(𝑝 = 0.0008, 𝐸𝑆 𝐸1 = 0.635)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.927)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.014)

Baseline-run0

Baseline-run1
Baseline-run2(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.772)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.859)

IMNTPU-run0

Baseline-run1
Baseline-run2(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.765)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.852)

60

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Overview of the NTCIR-16
Dialogue Evaluation (DialEval-2) Task

Conference’17, July 2017, Washington, DC, USA

Table 20: Statistical significance in terms of JSD (the Chinese ND subtask) calculated by Randomised Tukey HSD tests
Runsignificantly better than these runs

RSLDE-run0

Baseline-run2
Baseline-run1
NKUST-run1
TUA1-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.041)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.320)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.930)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 3.677)

Baseline-run0

Baseline-run2
Baseline-run1
NKUST-run1
TUA1-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.003)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.282)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.892)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 3.639)

RSLDE-run2

Baseline-run2
Baseline-run1
NKUST-run1
TUA1-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.968)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.247)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.858)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 3.605)

RSLDE-run1

Baseline-run2
Baseline-run1
NKUST-run1
TUA1-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.926)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.205)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.815)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 3.562)

NKUST-run0

Baseline-run2
Baseline-run1
NKUST-run1
TUA1-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.870)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.149)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.759)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 3.507)

TUA1-run0

Baseline-run2
Baseline-run1
NKUST-run1
TUA1-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.822)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.101)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.711)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 3.458)

Baseline-run2

TUA1-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.637)

Baseline-run1

TUA1-run1(𝑝 = 0.0002, 𝐸𝑆 𝐸1 = 1.358)

61

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Conference’17, July 2017, Washington, DC, USA

Sijie Tao and Tetsuya Sakai

Table 21: Statistical significance in terms of RNSS (the Chinese ND subtask) calculated by Randomised Tukey HSD tests
Runsignificantly better than these runs

RSLDE-run0

Baseline-run2
Baseline-run1
NKUST-run1
TUA1-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.540)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.098)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.577)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.773)

Baseline-run0

Baseline-run2
Baseline-run1
NKUST-run1
TUA1-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.484)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.042)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.521)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.717)

RSLDE-run1

Baseline-run2
Baseline-run1
NKUST-run1
TUA1-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.412)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.970)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.449)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.645)

RSLDE-run2

Baseline-run2
Baseline-run1
NKUST-run1
TUA1-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.402)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.960)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.439)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.635)

NKUST-run0

Baseline-run2
Baseline-run1
NKUST-run1
TUA1-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.354)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.912)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.391)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.587)

TUA1-run0

Baseline-run2
Baseline-run1
NKUST-run1
TUA1-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.331)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.890)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.368)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.565)

Baseline-run2

NKUST-run1
TUA1-run1(𝑝 = 0.0004, 𝐸𝑆 𝐸1 = 1.037)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.233)

62

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Overview of the NTCIR-16
Dialogue Evaluation (DialEval-2) Task

Conference’17, July 2017, Washington, DC, USA

Table 22: Statistical significance in terms of NMD (English DQ subtask, A-score) calculated by Randomised Tukey HSD tests
Runsignificantly better than these runs

TUA1-run0

RSLDE-run1
RSLDE-run0
Baseline-run1(𝑝 = 0.0042, 𝐸𝑆 𝐸1 = 0.562)(𝑝 = 0.0006, 𝐸𝑆 𝐸1 = 0.623)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.160)

Baseline-run2

Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.913)

IMNTPU-run0

Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.837)

Baseline-run0

Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.712)

RSLDE-run1

Baseline-run1(𝑝 = 0.0010, 𝐸𝑆 𝐸1 = 0.598)

RSLDE-run0

Baseline-run1(𝑝 = 0.0078, 𝐸𝑆 𝐸1 = 0.538)

Table 23: Statistical significance in terms of RSNOD (English DQ subtask, A-score) calculated by Randomised Tukey HSD tests
Runsignificantly better than these runs

TUA1-run0

IMNTPU-run0
RSLDE-run0
RSLDE-run1
Baseline-run1(𝑝 = 0.0046, 𝐸𝑆 𝐸1 = 0.535)(𝑝 = 0.0002, 𝐸𝑆 𝐸1 = 0.610)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.713)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.753)

Table 24: Statistical significance in terms of NMD (English DQ subtask, S-score) calculated by Randomised Tukey HSD tests
Runsignificantly better than these runs

TUA1-run0

Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.535)

Baseline-run2

Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.451)

IMNTPU-run0

Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.424)

RSLDE-run0

Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.346)

RSLDE-run1

Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.281)

Baseline-run0

Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.248)

Table 25: Statistical significance in terms of RSNOD (English DQ subtask, S-score) calculated by Randomised Tukey HSD tests
Runsignificantly better than these runs

TUA1-run0

Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.047)

Baseline-run0

Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.923)

IMNTPU-run0

Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.891)

Baseline-run2

Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.851)

RSLDE-run0

Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.836)

RSLDE-run1

Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.764)

63

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Conference’17, July 2017, Washington, DC, USA

Sijie Tao and Tetsuya Sakai

Table 26: Statistical significance in terms of NMD (English DQ subtask, E-score) calculated by Randomised Tukey HSD tests
Runsignificantly better than these runs

TUA1-run0

Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.872)

IMNTPU-run0

Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.825)

RSLDE-run0

Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.790)

Baseline-run0

Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.788)

RSLDE-run1

Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.773)

Baseline-run2

Baseline-run1(𝑝 = 0.0386, 𝐸𝑆 𝐸1 = 0.463)

Table 27: Statistical significance in terms of RSNOD (English DQ subtask, E-score) calculated by Randomised Tukey HSD tests
Runsignificantly better than these runs

TUA1-run0

Baseline-run1
Baseline-run2(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.910)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.998)

Baseline-run0

Baseline-run1
Baseline-run2(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.907)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.995)

IMNTPU-run0

Baseline-run1
Baseline-run2(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.808)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.896)

RSLDE-run0

Baseline-run1
Baseline-run2(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.802)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.890)

RSLDE-run1

Baseline-run1
Baseline-run2(𝑝 = 0.0004, 𝐸𝑆 𝐸1 = 0.732)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 0.820)

Table 28: Statistical significance in terms of JSD (English ND subtask) calculated by Randomised Tukey HSD tests
Runsignificantly better than these runs

RSLDE-run0

Baseline-run2
Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.659)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 3.021)

IMNTPU-run0

Baseline-run2
Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.571)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.934)

Baseline-run0

Baseline-run2
Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.522)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.885)

NKUST-run0

Baseline-run2
Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.488)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.850)

RSLDE-run2

Baseline-run2
Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.418)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.781)

RSLDE-run1

Baseline-run2
Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.387)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.750)

TUA1-run0

Baseline-run2
Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.311)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.674)

64

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Overview of the NTCIR-16
Dialogue Evaluation (DialEval-2) Task

Conference’17, July 2017, Washington, DC, USA

Table 29: Statistical significance in terms of RNSS (English ND subtask) calculated by Randomised Tukey HSD tests
Runsignificantly better than these runs

IMNTPU-run0

Baseline-run2
Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.720)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.330)

RSLDE-run0

Baseline-run2
Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.667)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.277)

Baseline-run0

Baseline-run2
Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.529)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.138)

NKUST-run0

Baseline-run2
Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.500)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.109)

RSLDE-run2

Baseline-run2
Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.456)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 2.065)

TUA1-run0

Baseline-run2
Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.388)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.997)

RSLDE-run1

Baseline-run2
Baseline-run1(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.358)(𝑝 < 0.0001, 𝐸𝑆 𝐸1 = 1.968)

Baseline-run2

Baseline-run1(𝑝 = 0.0398, 𝐸𝑆 𝐸1 = 0.609)

65