NTCIR-14
Short Text Conversation
Task (STC-3)
Sosuke Kato

Zhaohao Zeng Tetsuya Sakai(Waseda University)
Minlie Huang(Tsinghua University)
stc3org@list.waseda.jp
http://sakailab.com/ntcir14stc3/
Twitter: @ntcirstc
Version 20180606

STC-1, -2, -3.
NTCIR-12 STC-1
22 active
participants
NTCIR-13 STC-2
27 active
participants

Japanese

ChineseTwitter,Retrieval

Weibo,Retrieval

Yahoo! News,
Retrieval+Generation

Weibo,
Retrieval+Generation

Chinese Emotional Conversation
Generation (CECG) subtask

Weibo,
Generation
for given
emotioncategories

NTCIR-14 STC-3

Dialogue Quality (DQ) and Nugget
Detection (ND) subtasks

EnglishWeibo+English translations,distribution estimation for
subjective annotations

Single-round,
Nontask-oriented

Multi-round,task-oriented(helpdesk)

STC-3 subtasks• Chinese Emotional Conversation Generation(CECG): for details, please visit
http://coai.cs.tsinghua.edu.cn/hml/challenge/• Dialogue Quality (DQ):
please read this slide deck
• Nugget Detection (ND):
please read this slide deck

Dialogue Quality (DQ)
andNugget Detection (ND)
subtasks of
STC-3@NTCIR-14
Sosuke Kato Zhaohao Zeng Tetsuya Sakai(Waseda University)

Motivation• You cannot improve what you cannot measure.
• ⇒ To build good task-oriented, multi-round, textual
dialogue systems, we need good ways to evaluatethem.

Online evaluation is important but• Costly and does not scale• Difficult to compare different systems• Not repeatable even for the same system
Evaluator

Your overall score is 20%.

Posts, Turns• PostText entered by utterer in a dialogue on Weibo, eachwith a timestamp• Turns (or utterance blocks)
Maximal consecutive posts by the same utterer in adialogue
Customer post
posts

Customer post
Helpdesk post
Customer post
Helpdesk post
Helpdesk post
Customer post
Helpdesk post

turns

NTCIR-14 STC-3 (Chinese and English)
Dialogue Quality subtask
INPUT:
a customer-helpdesk
dialogue d ∈ D

Customer post
Customer post
Helpdesk post
Customer post
Helpdesk post
Helpdesk post
Customer post
Helpdesk post
Gold distribution p* based on
N annotators reflecting
subjective views

OUTPUT: an estimated probability
distribution p of dialogue quality score

M(d): measure quantifying howp differs from p* (see later slide)ordinal bins (dialogue quality scores)

Dialogue Quality: target scores• A-score: Task Accomplishment (Has the problem
been solved? To what extent?)• S-score: Customer Satisfaction of the dialogue (notof the product/service or the company)• E-score: Dialogue Effectiveness (Do the utterers
interact effectively to solve the problemefficiently?)• Scale: -2, -1, 0, 1, 2

Nuggets• A nugget is an turn that helps the Customertransition from the current state (where theproblem is yet to be solved) towards the targetstate (where the problem has been solved).
Not-a-Nugget
turn

Customer’s
initial state(start ofdialogue,facing aproblem)

Different paths that connect
the initial and target states

Contribution of a nugget

Not-a-Nugget
turn
Customer’s
target state(problemsolved)

An intermediate state: Customer has
advanced a little owing to the information
conveyed in the nugget

Nugget types• CNUG0: Customer trigger (problem stated)• CNUG*: Customer goal (solution confirmed)• HNUG*: Helpdesk goal (solution stated)• CNUG: Customer regularContains info thatleads to solution• HNUG: Helpdesk regular• CNaN: Customer Not-a-Nugget• HNaN: Helpdesk Not-a-Nugget
Does not contain info
that leads to solution

Nugget types: an example
C: I copied a picture from my PC to my mobile phone, but it
kind of looks fuzzy on the phone. How can I solve this? P.S. I’m
no good at computers and mobile phones.

CNUG0(problemstated)

H: Please synchronise your PC and phone using iTunes first,
and then upload your picture.

HNUG*(solutionstated)

C: I’d done the synchronization but did not upload it with XXX
Mobile Assistant. I managed to do so by following your advice.
You are a real expert, thank you!

CNUG*(solutionconfirmed)

H: You are very welcome. If you have any problems using XXX
Moble Phone Software, please contact us again, or visit
XXX.com.

HNaN(Not-aNugget)NTCIR-14 STC-3 (Chinese and English)
Nugget Detection subtask
OUTPUT: estimated p’s
over helpdesk nugget typesbH

INPUT: d ∈ D
Customer post
Customer post
Helpdesk post
Customer post
Helpdesk post
Helpdesk post
Customer postHelpdesk post

OUTPUT: estimated p’s
over customer nugget typesbCM(bC)M(bH)Compares two
distributions
over nominal
bins (nuggettypes)

weighted
average
Compares two
distributions
over nominal
bins (nuggettypes)

Why nuggets?
• If nuggets can be detected automatically, they may
serve as useful features for automaticallyestimating the dialogue quality.
• Automatic nugget detection may help us diagnosea dialogue closely (why it failed, where it failed).
• Ultimately, experiences from the nugget detection
subtask may help us design Helpdesk systems thatprovide the solution to a given problem effectivelyand efficiently.

Evaluation measures (comparing system andgold distributions)• Dialogue Quality (ordinal bins):- NMD: Normalised Match Distance - a special case of EarthMover’s Distance- RSNOD: Root Symmetric Normalised Order-aware
Divergenceas M(d) for each dialogue d.
• Nugget Detection (nominal bins):- RNSS: Root Normalised Sum of Squared errors- JSD: Jensen-Shannon divergenceas M(b) for each turn b.
See: Sakai, .T:
Comparing Two Binned Probability Distributions for Information Access Evaluation
https://waseda.box.com/SIGIR2018preprint

DCH-1 Chinese dialogue test collection [Zeng+17]
http://waseda.box.com/DCH-0-1
http://ceur-ws.org/Vol-2008/paper_1.pdf• 3700 Chinese customer-helpdesk dialogues minedfrom Weibo, with annotations• English translation available for 1264 (34%) of DCH1 (more will be translated May-June)
Customer post
Customer post
Helpdesk post
Customer post
Helpdesk post
Helpdesk post
Customer post
Helpdesk post

Dialogue quality annotations(task statement, task accomplishment,
customer satisfaction, helpdesk appropriateness,customer appropriateness)

Customer trigger nugget (CNUG0)
Helpdesk regular nugget (HNUG)
Nugget annotations
at the post (sequence) level
Helpdesk goal nugget (HNUG*)
Customer goal nugget (CNUG*)

Constructing STC-3 training datafrom DCH-1 (May-Aug)• DCH-1 will be re-annotated for the Dialogue Quality(A-score, S-score, E-score) and the Nugget
Detection (CNUG0, CNUG*, HNUG*, CNUG, HNUG,
CNaN, HNaN) subtasks by 10-20 annotators per
Dialogue quality annotations
dialogue(A-score, S-score, E-score distributions)Customer post
Customer post
Helpdesk post
Customer post
Helpdesk post
Helpdesk post
Customer post
Helpdesk post

Customer trigger nugget (CNUG0)
Helpdesk regular nugget (HNUG)
Customer Not-a-Nugget (CNaN)
Helpdesk goal nugget (HNUG*)
Customer goal nugget (CNUG*)
Helpdesk Not-a-Nugget (CNaN)
Nugget annotations at the turn level

Test data
• To be crawled in April (about 300 Chinese Weibo
helpdesk/customer dialogues)• To be annotated in May-Aug• To be translated into English in May-JuneOn annotations• For both training and test data, only the Chinese
portions will be annotated. These annotations willthen be copied onto the English portions.
Original Chinese dialogue

Manually translated English dialogue
translate

Customer post
Customer post
Helpdesk post
Customer post
Helpdesk post
Helpdesk post
Customer post
Helpdesk post

DQannotations

NDannotations

copy

Customer post
Customer post
Helpdesk post
Customer post
Helpdesk post
Helpdesk post
Customer post
Helpdesk post

Schedule for DQ and ND subtasks(incl. generic NTCIR-14 schedule)
1264 out of 3700 (34%)
done

Oct-Dec, 2017 Training data translation into English
April 2018 Crawling Chinese test data from Weibo, develop an annotation
tool
May-Jun, 2018 Test data + additional training data translation into English
May-Aug, Training/test Chinese data annotation
Aug 31, 2018 STC-3 task registrations due (CECG, DQ, ND)
Sep 1, 2018 Training data with annotations released
Nov 1, 2018 Test data released
Nov 30, 2018 Run submissions due
Feb 1, 2019 Results summary and draft overview released
Mar 15, 2019 Participant paper submissions due
May 1, 2019 All camera-ready papers due
Jun 2019 NTCIR-14 Conference@NII, Tokyo