KITE: A Benchmark for Evaluating Korean Instruction-Following Abilities in Large Language Models
Authors: Dongjun Kim, Chanjun Park, Chanhee Park, Heuiseok Lim


Reviewer paper summaries: This paper proposes two benchmarks for evaluating the Korean instruction-following ability of LLM. The first benchmark is KITE general that is translated from a English instruction-following dataset, IFEval. To further investigate the linguistic and cultural property of Korean, a new benchmark, KITE Korean, is build from scratch and focues on acrostic poem, post-position drop, honorifics and native/sino Korean number system.

The authors also conduct thorough experiments across various LLMs with different shot settings. In addtion, the authors also include human evaluation to demonstrate the correlation between human and automatic evaluation, to validate the effectiveness of the created benchmarks.
Summary of strengths: 1. The paper is well-structured, easy to follow.
2. The experiments are thorough. The inclusion of human evaluation well supports the automatic evaluation of the benchmarks, which potentially makes the benchmark more practical.
Summary of weaknesses: 1. ***Limited contributions***. The contributions are mainly from the KITE Korean benchmark, because KITE general is a translation verison of IFEval. However, KITE Korean benchmark is relatively small, containing only 100 instructions, which might cause a large variance with different prompting techniques.
Comments, suggestions and typos: NA
Overall assessment scores: 2