The Need for a Leaderboard: A Survey of LLM as a Judge in NLP
Authors: Guojun Wu


Reviewer paper summaries: This short paper surveys work on using "LLM-as-a-judge" to evaluate generated text. It notes the lack of standards for meta-evaluation, lists a few risks and limitations of using LLMs as judges, then argues that a leaderboard for evaluator models should be developed, to build common ground and overcome the lack of standards. They outline some recommendations for the development of this leaderboard, e.g. the best correlation metrics, the best benchmark datasets, and the use of challenge sets. I identified two minor flaws (see below), but overall, I think this is a decent survey and position paper.
Summary of strengths: - The methodology used for the survey is clear and thorough.
- The findings of the literature review are well summarized and structured.
- I think this paper could be a useful reference on the topic (at least for a short while). 
- The paper is clear and generally well written.
Summary of weaknesses: - The actual recommendations for leaderboard development are limited, and some of them will quickly become stale (e.g. the best benchmark datasets for this purpose).
- The language(s) covered in the papers surveyed are not mentioned. Presumably, most of the work surveyed focused on English, so the lack of evaluation on other languages would be another lacuna in the current literature.
Comments, suggestions and typos: - Line 2: "gains" should be "gained".
- Line 26: the relative ease and low cost of using LLMs as a judge is obviously another reason for their use.
- Footnote 2: "are" should be "is". 
- Appendix: please provide the paper titles, not just author+year and links.
- You should explain early on what "meta-evaluation" means.
- Section 3.2: You could also mention list-wise ranking, though I don't know if it has been used for "LLM-as-a-judge".
- Section 3.3: Please provide a reference for MQM. 
- "Branch and merge" is sometimes written with hyphens, sometimes without. Likewise for "meta-evaluation". 
- Lines 121 and 158: add space before the reference.
- Line 155: add "the" before "paper". 
- Line 181: do you mean that they *favor* long text, or that they *produce* long text? I presume the former so "are prone to" is not what you mean here.
- Line 208: put "r" in italics (or whatever font you used previously).
Overall assessment scores: 3