Overview of the NTCIR-16
Dialogue Evaluation(DialEval-2) Task
Sijie Tao and Tetsuya Sakai
Waseda University

Outline
1. History, definition, and motivation of DialEval
2. The new data collection for DialEval-2
3. Participants
4. Results
5. Conclusions

Outline
1. History, definition, and motivation of DialEval
2. The new data collection for DialEval-2
3. Participants
4. Results
5. Conclusions

History of the task
• NTCIR-14, Jun 2019, Short Text Conversation Task (STC-3) [Zeng+19]• DCH-1 Dataset used as training and test sets
• 3,700 + 390 for Chinese, 1,672 + 390 for English• NTCIR-15, Dec 2020, Dialogue Evaluation Task (DialEval-1) [Zeng+20]• DCH-1 used as training and development sets, new test set built• 3,700 + 390 + 300 for Chinese, 2,251 + 390 + 300 for English• NTCIR-16, Jun 2022, Dialogue Evaluation Task (DialEval-2)• DCH-2 [Zeng+21] as training and development sets, new test set built• 4,090 + 300 + 65 for both Chinese and English

Task Definition• DialEval-2 hosts two subtasks:• Dialogue Quality (DQ)
• Nugget Detection (ND)

• DQ: Given a customer-helpdesk dialogue, return an estimateddistribution of dialogue quality ratings for the entire dialogue• ND: Given a customer-helpdesk dialogue, return an estimateddistribution of labels over nugget types for each turn

An Example of a customer-helpdesk dialogue [Zeng+20]

Dialogue Quality Subtask (DQ)
• Given a customer-helpdesk dialogue, return an estimated distributionof dialogue quality ratings for the entire dialogue.
• Three types of dialogue quality ratings (Likert scale -2 to 2):• A-score: Task Accomplishment• S-score: Customer Satisfaction (about the dialogue itself, not about theproduct/service)• E-score: Dialogue EffectivenessDialogue Quality Subtask (DQ)Dialogue Quality Subtask (DQ)
• Evaluation metrics• NMD (Normalised Match Distance)• RSNOD (Root Symmetric Normalised Order-aware Divergence) [Sakai18]• Both measures take into account the distance between two bins, tomake sure X is rated higher than Y in the previous slide.

Nugget Detection Subtask (ND)
• What is a nugget?

An Example of a customer-helpdesk dialogue [Zeng+20]

Nugget Detection Subtask (ND)
• What is a nugget?

Nugget Detection Subtask (ND)
• Given a customer-helpdesk dialogue, return an estimated distributionof labels over nugget types for each turn

Nugget Detection Subtask (ND)

Nugget Detection Subtask (ND)
• Evaluation metrics• RNSS (Root Normalised Sum of Squares)• JSD (Jensen-Shannon Divergence) [Sakai18]• No need to use NMD or RSNOD, as the bins in the ND subtask arenominal (e.g. HNUG, HNUG*, HNaN), not ordinal

Motivation of the task
• Evaluate customer-helpdesk dialogues automatically• DQ: An effective DQ system is useful for building helpdesk systems
that can generate effective utterances for diverse users.
• ND: An effective ND system is useful for building effective helpdesk
systems that can self diagnose at the dialogue turn level to improve
themselves.

Outline
1. History, definition, and motivation of DialEval
2. The new data collection for DialEval-2
3. Participants
4. Results
5. Conclusions

The new data collection for DialEval-2• For DialEval-2, we use DCH-2 dataset [Zeng+21] as training anddevelopment sets
• A new test set which contains 65 dialogues is additionally built

Outline
1. History, definition, and motivation of DialEval
2. The new data collection for DialEval-2
3. Participants
4. Results
5. Conclusions

Participant teams (only four, last time we had seven)• IMNTPU (National Taipei University) [Hsiao+22]• NKUST (National Kaohsiung University of Science and Technology) [Chang+22]• RSLDE (Waseda University) [Li+22]• TUA1 (Tokushima University) [Ding+22]

Outline
1. History, definition, and motivation of DialEval
2. The new data collection for DialEval-2
3. Participants
4. Results
5. Conclusions

Results• Baselines (exactly the same as the baselines in DialEval-1) [Zeng+20]• BL-lstm (Baseline-run0): A baseline model which leverages Bidirectional Long
Short-term Memory;• BL-uniform (Baseline-run1): A baseline model which always predict theuniform distribution;• BL-popularity (Baseline-run2): A baseline model which predicts theprobability of the most popular label as one, and predicts other labels as 0.

Results(DQ, Chinese)• TUA1-run1, 2 are the top runs in terms ofRSNOD and NMD for A and S-score• Only TUA-run0 outperforms Baseline-run0
statistically significantly in terms of NMD forE-score

Results (DQ, English)• TUA1-run0 is the top run and the only run that outperforms thebaseline systems• But the differences between TUA1-run0 and the top baselines are
not statistically significant

Results (ND, Chinese)• RSLDE-run0 is the top run and the only
run that can outperform Baseline-run0in terms of both JSD and RNSS
• But the difference between them is notstatistically significant

Results (ND, English)• RSLDE-run0 and IMNTPU-run0 are
the runs can outperform Baselinerun0• But their differences between thebaseline are not statisticallysignificant

Results(Differences between metrics)• The difference between different metrics are not statisticallysignificant for both ND and DQ subtasks• Consistent with what we observed at DialEval-1 and STC-3.
[Zeng+19][Zeng+20]

Outline
1. History, definition, and motivation of DialEval
2. The new data collection for DialEval-2
3. Participants4. Results
5. Conclusions

Conclusions• Overview of DialEval-2:• Task definition• Data collection• Evaluation results• From the evaluation results, we observe that
• Only one run from TUA1 outperform the LSTM baseline significantly inChinese DQ task in terms of NMD for E-score.
• In other subtasks, none of the runs can outperform the LSTM baseline
significantly.
• No substantial difference is observed between the evaluation metrics for eachsubtasks.

References[Zeng+19] Zhaohao Zeng, Sosuke Kato, and Tetsuya Sakai. 2019. Overview of the NTCIR-14 Short Text
Conversation Task: Dialogue Quality and Nugget Detection Subtasks. In Proceedings of NTCIR-14. 290–315.
[Zeng+20] Zhaohao Zeng, Sosuke Kato, Tetsuya Sakai, and Inho Kang. 2020. Overview of the NTCIR-15 Dialogue
Evaluation (DialEval-1) Task. In Proceedings of NTCIR-15. 13–34.
[Zeng+21] Zhaohao Zeng and Tetsuya Sakai. 2021. DCH-2: A Parallel Customer-Helpdesk Dialogue Corpus withDistributions of Annotators’ Labels. CoRR abs/2104.08755 (2021). arXiv:2104.08755
https://arxiv.org/abs/2104.08755[Sakai18] Tetsuya Sakai. 2018. Comparing Two Binned Probability Distributions for Information AccessEvaluation. In The 41st International ACM SIGIR Conference on Research & Development in Information
Retrieval (Ann Arbor, MI, USA) (SIGIR’18). ACM, New York, NY, USA, 1073–1076.

References[Chang+22] Tao-Hsing Chang and Jian-He Chen. 2022. NKUST at the NTCIR-16 DialEval-2 Task. In Proceedings ofNTCIR-16. to appear.
[Ding+22] Fei Ding, Kang Xin, Yunong Wu, and Fuji Ren. 2022. TUA1 at the NTCIR-16 DialEval-2 Task. In
Proceedings of NTCIR-16. to appear.
[Hsiao+22] Ting-Yun Hsiao, Yung-Wei Teng, Pei-Tz Chiu, Mike Tian-Jian Jiang, and Min-Yuh Day. 2022. IMNTPUDialogue System Evaluation at the NTCIR-16 DialEval-2 Dialogue Quality and Nugget Detection. In Proceedingsof NTCIR-16. to appear.
[Li+22] Fan Li and Tetsuya Sakai. 2022. RSLDE at the NTCIR-16 DialEval-2 Task. In Proceedings of NTCIR-16. toappear.