Find the INTENTION OF INSTRUCTION: Comprehensive Evaluation of Instruction Understanding for Large Language Models
Authors: Hyeonseok Moon, Jaehyung Seo, Seungyoon Lee, Chanjun Park, Heuiseok Lim


Reviewer paper summaries: This paper provides a new benchmark which inverts the order of instruction tuning by giving the results and finding appropriate instructions. The authors introduce how they constructed the benchmark and designed the experiments to test this ability.
Summary of strengths: - The research topic is interesting as the inverse function f'is a very intriguing area that warrants investigation.
- The selection of LMs is exhaustive, representing a large family of different LMs across the benchmarks.
Summary of weaknesses: I will detail this part from high-level discussion to low-level discussion:
 - Why is it hard for LLMs to overly follow certain instructions or not understand the nuanced complexity related to the topics to select the correct instructions to generate that answer? For me, the problem you mention is possibly more driven by over-training or a fundamental limitation of LMs' next token prediction loss.
- Given a Model M, which implements the function of input X to Y, where X is instructions in this case, is the basic formulation of the problem. But given an output Y, the mapping function is one-to-many, which means that you may have endless instructions to generate that answer. So if you use existing data as ground truth, the overall possible space for instructions is narrowed down quite well. I am kinda concerned about that.
- For Table 2, I can see GPT-4 with LLaMa3-8B, all achieving very good performances. If you test the results for larger models like LLaMa3-70B/405B or LLaMa3.1-8B/70B, you would achieve better results on your benchmarks. Based on that, I think the benchmarks may be built too easily without considering too much complexity of the real world.
Comments, suggestions and typos: - For Table 2, why GPT-4 does not report variance score? I thought it was a typo remaining in overleaf.
Overall assessment scores: 2