ReMEDIATES: Revisiting MEDIA with Tree Enhanced Semantics
Authors: Lucas Druart, Valentin Vielzeuf, Yannick Estève


Reviewer paper summaries: - This paper introduces the ReMEDIATES benchmark for evaluating spoken task-oriented dialogue systems, specifically in the hotel booking domain. 
- It enhances the French MEDIA dataset with detailed semantic tree annotations, capturing the structure and context of dialogues. 
- The benchmark addresses challenges such as producing structured outputs, maintaining dialogue context, and resolving co-references. 
- It evaluates models using cascade and end-to-end approaches, showing that end-to-end models perform better in handling context and identifying non-informative turns.
Summary of strengths: - This study introduces a new and challenging task in spoken task-oriented dialogue systems using semantic trees, which offer a detailed way to understand and parse dialogues.
- The authors have enriched the French MEDIA dataset with comprehensive semantic tree annotations, providing a valuable resource for the community.
- The benchmark evaluates multiple models (cascaded and end-to-end approaches), offering insights into their strengths and weaknesses in handling dialogue context and structure.
- The paper presents a clear and well-structured methodology for annotating and evaluating the dialogues, making it easy for others to replicate and build upon.
- This work tries to address ethical issues for data collection and usage to ensure responsible research practices.
Summary of weaknesses: - The dataset and benchmark focus on French dialogues in the hotel booking domain, which might limit its usefulness to a broader audience who work with other languages.
- This work does not explore a wide variety of models, particularly larger or more advanced models, which could provide a better understanding of the task's difficulty and model performance.
- The models struggle to distinguish informative turns from non-informative ones, especially in the early stages of the dialogue, which affects the overall performance.
- The study misses a detailed evaluation of how the proposed method impacts the performance of complete dialogue systems, making it unclear how useful the approach is in practical applications.
- The task of parsing spoken dialogues with semantic trees is complex and may require significant effort from researchers to implement, possibly limiting widespread adoption.
Comments, suggestions and typos: - The authors may improve the explanations of the methodology and results for more clarity (e.g., adding more examples to illustrate critical points).
- Including a comparison with more diverse models and architectures to provide a broader perspective on the task can help.
- The authors might offer translations or glosses of key terms and concepts for non-French speakers to make the dataset more accessible.
- Providing detailed instructions on the dataset usage and benchmark to encourage more researchers to adopt it could help.
- Discussing the potential real-world applications and impacts of proposed methods would help.
- Expanding the discussion on ethical implications and how to use the dataset responsibly could help.
- Please review the paper for minor typos or grammatical errors to improve the overall readability.
Overall assessment scores: 3