reviews: 
Comments: This paper proposes a selective evaluation framework to ensure that LLM-based evaluations align with human judgments to a defined degree of confidence. Selective Confidence Evaluation ensures that, instead of always trusting an LLM's judgment, the model evaluates its confidence in a given answer and abstains when unsure. Simulated Annotators leverage in-context learning to simulate diverse annotator perspectives, improving model calibration and reducing evaluation errors. Cascaded Selective Evaluation introduces multiple LLM judges used in a tiered manner, starting with a more cost-effective model and escalating to stronger models only if needed. The method achieves strong human alignment and reduces reliance on more costly models like GPT-4 by using smaller models when possible.
Strengths: This paper provides clear methodological contributions to improving human-LLM alignments. I especially appreciate the cascaded selective evaluation approach, which explicitly considers the costs of using stronger/more costly LLMs. Overall, this is an excellent paper with clear contributions and is well-executed.
Weaknesses: This paper has already been very carefully executed, and I have only minor suggestions. 

1.	Simulated annotators, despite their encouraging performance, will require significant processing time and resources as N and K increase. Given a substantial decrease in ECE, I believe that simulated annotators have enough value but it would be helpful for the readers if the authors can provide time and cost requirements compared to the canonical approaches. Furthermore, the authors may want to invest a bit more space in explaining the algorithm of simulated annotators including exactly how they handled in-context learning.

2.	The outperformance of cascaded selective evaluation is somewhat expected because the gist of the framework is to exclude the observations where annotator disagreement is expected. In my view, the model does a good job of identifying these hard-to-reconcile observations. A fairer comparison would be to exclude the abstained observations from the baseline models and see how their performance is affected. For example, if you maintain the same coverage for all other methods in Table 2 and Table 3, what would their empirical human agreement look like? Of course, empirically this cannot be achieved without your model – but as you show in Section 3.3, some systematic characteristics can predict abstention. One can simply devise an alternative strategy that can identify these hard-to-reconcile samples and run a multi-agentic approach on them. 

3.	I think the most striking result is in Table 7. It is surprising that the weaker cascades achieve an empirical human agreement of 80.3%. Although their coverage is 68.3%, this is indeed a remarkable result. Given this, I would try to do more with the abstained samples. Almost 30% of the sample is now abstained and running the most advanced model on those will not only increase your ultimate coverage and success rate. The relative API cost will be substantially lower than using only GPT4.

4.	Relatedly, the relative API cost of the model is largely dependent on the difficulty of your task. As seen in Tables 2, 3, and 8, the evaluator composition is heterogeneous across different tasks. When a smaller LLM can handle the majority of tasks, the overall task is likely to be cheaper. When a larger LLM needs to intervene, the overall task will be more expensive. I would caveat this (and perhaps show the range of relative API cost using some simulated evaluator compositions) in Section 3.6.

5.	It is a minor thing, but is there any reason that you chose 1-a =0.9 in Table 2, 1-a=0.85 in Table 3, and 1-a=0.8 in Table 7? I would personally prefer the three thresholds to be consistent throughout the paper.

Overall, it was an interesting read. Great paper.
Recommendation: 10
Reviewer confidence: 5
Soundness/correctness: 3
Clarity: 4