A Benchmark for Models Clarifying and Requesting Information
Authors: Yujian Gan, Changling Li, Jinxia Xie, Luou Wen, Matthew Purver, Massimo Poesio


Reviewer paper summaries: This paper introduces an evaluation framework designed to assess the ability of conversational agents to ask clarification questions in task-oriented dialogues. This framework includes a bilingual English-Chinese dataset featuring 31 task types with 10 unique dialogue scenarios each, where an information seeker interacts with an information provider. This paper also introduces to use provider agent (LLMs) replicates human responses, enabling dynamic interaction for both current and future seeker agents.
Summary of strengths: 1. This paper introduces a novel approach to evaluating task-oriented dialogue systems by focusing on the agent’s ability to ask clarification questions dynamically.
2. This paper includes various dialogue scenarios (310 dialogue instances). But it would also be interesting to see how diverse each dialogue scenario can develop.
3. It's good to see this paper has addressed some concerns about its previous submission. But it would be better to address all of them.
4. it's a bilingual dataset.
Summary of weaknesses: 1. It's good to see that this paper involves two-step response generation to ensure valid responses from providers. However, it is not discussed how to judge whether response I (human providers' response cannot reply to seekers). Also, Response II cannot guarantee consistency since it's totally independent of Response I. It would be more convincing to have a verification of the quality of responses.
2. It seems L2L achieves quite low scores. However, there is no error analysis on why these language models fail. Since it is an LM-to-LM setting, is it possible that the quality of the provider also matters?
Comments, suggestions and typos: Line 1255 "adhered" --> "adhere"

In Table 3, the result for GPT4 is from the previous submission. Is it possible that different versions lead to GPT4o outperforming GPT4?
Overall assessment scores: 3.5