The NTCIR-16 Dialogue
Evaluation Task(DialEval-2)
dialeval2org@list.waseda.jp
Tetsuya Sakai, Sijie Tao (Waseda University)
Inho Kang (Naver Corporation)
http://sakailab.com/dialeval2/Version 20210325

Introduction• The NTCIR-16 Dialogue Evaluation Task (DialEval-2)
hosts two subtasks, Dialogue Quality (DQ) and Nugget
Detection (ND), which are exactly the same as thosefrom NTCIR-14 STC-3 and NTCIR-15 DialEval-1.
• DQ: Given a customer-helpdesk dialogue, return anestimated distribution of dialogue quality ratings forthe entire dialogue.
• ND: Given a customer-helpdesk dialogue, return anestimated distribution of labels over nugget types(similar to dialogue acts) for each turn.
• Data: Chinese and English

Task overview (same as DialEval-1)
Dialogue Quality Subtask

Nugget Detection Subtask

Chinese/English customer-helpdesk dialogue
Customer’s turnDialogueGold distribution
for Task Accomplishment,
Customer Satisfaction,
Dialogue EffectivenessscoresHelpdesk’s turn
Customer’s turnHelpdesk’s turnTurn

A nugget is a turn that helps
the customer transition
towards “Problem Solved”-202
Evaluating
distributions
over ordinalbins

Participant’s
estimateddistribution-202

Gold distribution
overRegular nugget,
Goal nugget,
Not-A-Nugget
Nugget types

Evaluating
distributions
over nominal
bins

Participant’s
estimateddistribution
Nugget types

Customer-Helpdesk dialogue: an example

Dialogue Quality Subtask (1)• Given a customer-helpdesk dialogue, return anestimated distribution of dialogue quality ratings
for the entire dialogue.
• Three types of dialogue quality ratings (Likert scale-2 to 2):
A-score: Task Accomplishment
S-score: Customer Satisfaction (about the dialogueitself, not about the product/service)
E-score: Dialogue EffectivenessDialogue Quality Subtask (2)Customer’s turnannotators

Helpdesk’s turn
Customer’s turnHelpdesk’s turnDistribution
estimated by System X
X is better than Y
-2

02

Gold
distribution

Distribution
estimated by System Y

DQ evaluation measures for comparing
gold and estimated distributions• NMD (Normalised Match Distance)• R(S)NOD (Root (Symmetric) Normalised Orderaware Divergence)• Both measures take into account the distance
between two bins, to make sure X is rated higher
than Y in the previous slide.
For more info on the evaluation measures, see
https://waseda.box.com/SIGIR2018preprint

What is a nugget?
Initial state(Customerfacing aproblem)

Turn that does notserve as a nugget

Goal state(problemSolved)

Intermediatestate

Nugget = a turn by Customer or Helpdesk that helps
Customer transition towards the goal (“problemsolved”) stateNugget Detection Subtask (1)• Given a customer-helpdesk dialogue, return anestimated distribution of labels over nugget types
for each turn.
Nugget type

Customer

Helpdesk

Trigger

CNUG0: tell the problem
to Helpdesk

Regular

CNUG

Goal

CNUG*: tell Helpdesk that HNUG*: tell Customer thethe problem has beensolution to the problem
solvedNot-a-nugget

CNaN

HNUG

HNaN

Nugget Detection Subtask (2)Gold distributions

Customer’s turnHelpdesk’s turn
Customer’s turnHelpdesk’s turnDistributions
estimated bySystem X

ND evaluation measures for comparing
gold and estimated distributions• RNSS (Root Normalised Sum of Squares)• JSD (Jensen-Shannon Divergence)• No need to use NMD or RSNOD, as the bins in theND subtask are nominal (e.g. HNUG, HNUG*,HNaN), not ordinal
For more info on the evaluation measures, see
https://waseda.box.com/SIGIR2018preprint

Why the task is important• DQ: An effective DQ system is useful for building
helpdesk systems that can generate effective
utterances for diverse users.
• ND: An effective ND system is useful for building
effective helpdesk systems that can self-diagnose at
the dialogue turn level to improve themselves.

Training data: DCH-2(Chinese-English parallel corpus with
DQ and ND labels)• Participants can obtain the data by sending anapplication form to us

Test data sample size
Topic set size design:
http://link.springer.com/content/pdf/10.1007%2Fs10791-0159273-z.pdf (open access)Based on the DQ results from DialEval-1 with RNOD and NMD,
we determined the number of test dialogues for DialEval-2.
The most unstable measure was RNOD for the DQ-A subtask,
whose residual variance was 0.00495. By plugging in this value
to http://www.f.waseda.jp/tetsuya/samplesizeANOVA2.xlsx
with m=10 systems and a minimum detectable difference
minD=0.05 under Cohen’s five-eighty convention, weobtained n=62. That is, 62 topics are enough for achieving
80% power for differences in mean RNOD of 0.05 or higher interms ANOVA for 10 systems. Based on this result, we plan toprepare about 66 test dialogues, 11 for each i, where i is thenumber of turns in a dialogue (i=2,…,7).

Important Dates (Timezone: Japan (UTC+9))

April 2021

Task registrations open: training data
given to participants

April-May 2021

Annotating the test dialogues

December 1 2021

Test dialogues released/Taskregistrations due

January 15 2022

Run submissions due

February 1, 2022

Evaluation results released

February 1, 2022

Draft task overview paper released

March 1, 2022

Draft participant paper submissions due

May 1, 2022

All camera-ready paper submissions due

June 2022

NTCIR-16 Conference in NII, Tokyo,Japan