Kati8/ICLR_2023_2025 huggingface dataset:
authors: Jaehun Jung
authors: Faeze Brahman
authors: Yejin Choi
reviews: 
Comments: The authors introduce a principled method for evaluating language models with guaranteed human agreement. They argue that evaluation should not solely rely on model preferences in pairwise comparisons. Instead, it should assess judge models' confidence and selectively decide when to trust their judgment. This selective framework ensures that the evaluation aligns with human agreement to a specified level. They introduce Simulated Annotators, a new method for confidence estimation that enhances judge calibration, enabling comprehensive evaluation. Their Cascaded Selective Evaluation uses cost-effective models initially, escalating to stronger ones as needed, maintaining human agreement guarantees. Results show this method achieves over 80% human agreement.
Strengths: 1. Robust Human Agreement: The Cascaded Selective Evaluation framework offers a strong guarantee of alignment with human judgment, enhancing the reliability of LLM-based evaluations.

2. Improved Confidence Estimation: The introduction of Simulated Annotators enhances the confidence estimation for LLM judges, allowing for more accurate evaluations without the need for external supervision.

3. Dynamic Judge Selection: By dynamically selecting which judge model to trust, the framework reduces evaluation overhead while maintaining reliability, often surpassing the precision of relying solely on the strongest judge model.
Weaknesses: It is more like a theoretical work. Given that the experiments have proved effective in evaluating different tasks, I suggest the authors invest some effort in releasing a user-friendly toolkit based on this work, for other researchers to use.
Recommendation: 6
Reviewer confidence: 3
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: The authors propose a framework for LLM evaluation that includes a human agreement guarantee, that is called Cascaded Selective Evaluation. The evaluation task (in this case a pairwise evaluation) is given to a weak LLM judge and escalated to a stronger LLM judge, if the confidence level for the task does not exceed a certain threshold. As a confidence estimation method they propose Simulated Annotators, where multiple LLMs are used to simulate annotator preferences.
Strengths: The paper is clearly structured and well written. The approach clearly improves the evaluation process supported by LLMs, while being cost effective by only considering stronger models (such as GPT-4-turbo) as a last escalation step. The authors provide meaningful experiments and show how the method can be extended in the future.
Weaknesses: A weakness of the approach (for now), is that it has only been tested and applied in context of the pairwise preference evaluation. Nevertheless, the Simulated Annotators and Cascaded Selected Evaluation methods are promising. It will be interesting to see how the approach performs in other evaluation settings in the future. The question of when the models abstain could be answered in more detail. For now, it is only answered quantitatively, but not really qualitatively. In order to find out whether the examples the models are not sure about are the same ones about which the human judges also disagree, the examples need to be examined in more detail, i.e. more on the level of content. It would be interesting to see the ratio of examples where both humans and the models abstained from evaluation and to analyze some examples on a content level. 

Small errors:
- Line 462: the the tradeoff
Recommendation: 8
Reviewer confidence: 5
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: Current NLP paradigms have shifted to using LLMs as judges for annotations instead of humans. This paper asks how we can guarantee the reliability of using these LLMs for evaluation through human agreement. 

Given an input, how likely would humans agree with the model judgment?

For this, the approach is to use selective evaluation: the model can abstain if not confident enough to make a prediction.

The conventional way of using the predictive probability as confidence estimates doesn't show to be reliable so two methods are proposed: 
1. Simulated annotators: simulate annotator preferences through In-Context Learning and the confidence is estimated through the agreement ratio. 
2. Cascaded Selective Evaluation: Start with a cheaper model and go to stronger models when the previous model is less confident. 

This reduces the evaluation overhead with high agreement guarantees. Their abstention policy aligns with human subjectivity.
Strengths: 1. The paper tackles an important issue, we need more human subjectivity at the forefront of LLM evaluation. 
2. In essence, the Simulated Annotators technique is ensembling, so using a simple and popular technique as ensembling is an interesting way to utilize for such a task. 
4. The authors conduct many experiments to show the utility of the approach. 
3. The added ablation studies in the paper are nice. 
5. The paper is well-written and structured.
Weaknesses: 1. A clear connection to _selective prediction_ is missing. While the authors talk about _selective evaluation_, the concept is the same so it is better to make that link clear. 
2. Related Work can benefit from more work on selective prediction and human subjectivity in LLMs, e.g. 

Barbara Plank. 2022. The “Problem” of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671–10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Khurana, U., Nalisnick, E., Fokkens, A., & Swayamdipta, S. Crowd-Calibrator: Can Annotator Disagreement Inform Calibration in Subjective Tasks?. In First Conference on Language Modeling.

Bavaresco, A., Bernardi, R., Bertolazzi, L., Elliott, D., Fernández, R., Gatt, A., ... & Testoni, A. (2024). Llms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks. arXiv preprint arXiv:2406.18403.

3. Cheaper models used for Cascading are still very costly (e.g. Mistral 7B)
4. The calibration set for human preferences is based on argmax of human judgments. Using argmax gets rid of disagreement signals, which is actually important information.
Recommendation: 8
Reviewer confidence: 3
Soundness/correctness: 3
Clarity: 3