JoHANN: Benchmarking Real-World Human-AI Dialogues Across Languages and Skills
Authors: Yohan Lee, Sungho Park, Sangwoo Han, Yunsung Lee, Yongwoo Song, Adam Lee, Jiwung Hyun, Jaemin Kim, HyeJin Gong


Reviewer paper summaries: This paper introduces JoHANN, a benchmark that samples approximately 3,000 chat logs from WildChat and another Korean chat platform, defining a set of skills and a checklist for evaluating the performance of LLMs.
Summary of strengths: N/A
Summary of weaknesses: 1. Poorly organized. The paper appears to have been written in haste and is of low quality.
2. The skill set and checklist are not clearly presented in the main text. While they are mentioned in the appendix, the main text should be self-contained.
3. The paper fails to provide a clear and convincing comparison with existing benchmarks to demonstrate its unique value.
4. The discussion of related work is insufficient.

    [1] Duan H, Wei J, Wang C, et al. BotChat: Evaluating LLMs' Capabilities of Having Multi-Turn Dialogues[J]. arXiv preprint arXiv:2310.13650, 2023.

    [2] Sun Y, Liu C, Huang J, et al. Parrot: Enhancing multi-turn chat models by learning to ask questions[J]. arXiv preprint arXiv:2310.07301, 2023.

    [3] Bai G, Liu J, Bu X, et al. Mt-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues[J]. arXiv preprint arXiv:2402.14762, 2024.

5. The experimental results indicate that this benchmark is not a reliable indicator of model performance. According to Figure 7 and Table 3, the performance gap between the most advanced GPT-4 and Llama-3-70B is minimal, only 0.03.
6. In summary, this work will offer very limited potential contributions to the community.
Comments, suggestions and typos: N/A
Overall assessment scores: 2