Title: On Evaluating LLM Alignment by Evaluating LLMs as Judges
Venue: NeurIPS 2025
Accepted: yes
Keywords: alignment evaluation

LLM-as-a-judge

LLM evaluation
Abstract: Alignment with human preferences is an important evaluation aspect of LLMs, requiring them to be helpful, honest, safe, and to precisely follow human instructions. Evaluating large language models' (LLMs) alignment typically involves directly assessing their open-ended responses, requiring human annotators or strong LLM judges. Conversely, LLMs themselves have also been extensively evaluated as judges for assessing alignment.  In this work, we examine the relationship between LLMs' generation and evaluation capabilities in aligning with human preferences. To this end, we first conduct a comprehensive analysis of the generation-evaluation consistency (GE-consistency) among various LLMs, revealing a strong correlation between their generation and evaluation capabilities when evaluated by a strong LLM preference oracle (GPT-4o). Utilizing this finding, we propose a benchmarking paradigm that measures LLM alignment with human preferences without directly evaluating their generated outputs, instead assessing LLMs in their role as evaluators. Our evaluation shows that our proposed benchmark, AlignEval, matches or surpasses widely used automatic LLM evaluation benchmarks, such as AlpacaEval and Arena-Hard, in capturing human preferences when ranking LLMs. Our study offers valuable insights into the connection between LLMs' generation and evaluation capabilities, and introduces a benchmark that assesses alignment without directly evaluating model outputs.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the Generation-Evaluation consistency (GE-consistency) of large language models (LLMs) and proposes a novel benchmarking paradigm, AlignEval, which evaluates LLM alignment by assessing their evaluation capabilities rather than directly judging their outputs. Experiments reveal a strong correlation between generation and evaluation rankings under specific conditions.\n\nSTRENGTHS AND WEAKNESSES:\n## Strength\n\nThis paper claims to be the first to conduct comprehensive study on GE consistency in LLMs, proposing a benchmark that bypasses direct output evaluation, offering a fresh perspective on alignment assessment. Their proposed benchmark, AlignEval, reduces evaluation costs (as no LLM-as-Judge needed) while aligning well with human preferences, making it cost-effective for real-world deployment. The paper is well-written, and I find it easy to follow and understand.\n\n## Weakness\n\n- Lack of human evaluation. I understand that reproducing the consistency analysis experiments in Section 3.2 by human annotators would be challenging and almost impossible. However, I believe it authors should at least try to validate the consistency hypothesis on a few LLMs using small-scale instructions. Since the entire paper fundamentally relies on the GE-Consistency assumption, I think additional supporting evidence would strengthen this core premise (this would be more persuasive though the current experiments across 15 LLMs are already relatively solid).\n\n- Lack of more kinds of LLMs. The authors mainly study the System-1 model. Can the proposed benchmark be used for the System-2 model (Deepseek R1, O3, Qwen3 and so on)? This needs further verification.\n\n- The author mentioned the importance of Consistency Filtering. However, this situation does not only occur with preference oracle. Did the author check how sensitive the tested LLM is to this situation? And how stable is the LLM's judgment output after multiple tests?\n\n- A big concern lies in that the correlation on alpacaeval is significantly lower than arena-hard. Does this mean that the proposed benchmark cannot adequately validate the model's performance on open domain problems?\n\n- This is related to the concern #4. Alignment is a big topic. Most of the author's experiments focus on the alignment of the preference of \"helpfulness\". Is there any similar experimental support for other preferences (safety and honesty)? I believe this is a very interesting and important question, which can make the assumptions and contributions of the article more solid.\n\nQUESTIONS:\n- Will authors open source the proposed benchmark? This is very important.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores whether LLMs exhibit consistent performance in two tasks: \"generating aligned responses\" and \"evaluating whether responses are aligned.\" The authors propose GE-consistency, which uses Spearman correlation to measure the similarity of model performance across these two tasks. Evaluations on several mainstream LLMs lead to the conclusion that good generators are often also good evaluators. Based on this finding, the paper introduces the ALIGNEVAL evaluation method, which indirectly assesses a model’s generation ability by evaluating whether it can accurately judge human preferences, thereby effectively reducing evaluation costs. Finally, the method is compared with other evaluation approaches to validate its effectiveness.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The writing is good. The authors provides a novel evaluation perspective: by assessing whether an LLM can accurately judge, it indirectly verifies its generation ability, offering a new research direction for future work.\n2. Based on the GE-consistency theory, the proposed ALIGNEVAL evaluation method can reduce computational and annotation costs.\n\nWeakness:\n1. In the GE-consistency experiments, only the Arena-Hard and AlpacaEval datasets were used. This seems insufficient, especially since the method performs well on the technical Arena-Hard dataset but poorly on the open-domain AlpacaEval dataset. More comprehensive experiments are needed to demonstrate generalizability.\n2. If the proposed consistency theory holds, then ALIGNEVAL can indeed reduce evaluation costs for subsequent work. However, as the authors mention, it may be vulnerable to adversarial adaptation, where high scores do not necessarily indicate true capability. If this issue cannot be resolved, it will still be necessary to rely on evaluating the model’s output to fully verify its generalization ability, which may not actually reduce evaluation costs.\n3. GE-consistency measures correlation, not causation. There is a potential confounding factor that \"strong models perform well on all tasks,\" meaning that high scores across the board may obscure essential differences between the two abilities (uncertain).\n\nQUESTIONS:\n1. As shown in Figure 3, small models like Gemma-2B and LLaMA-3-8B do not perform well in terms of GE-consistency. Does this mean the method is only suitable for evaluating and ranking mid- to high-end models?\n2. Since stronger models may perform better on all tasks, can the high correlation between generation and evaluation abilities be attributed to simultaneous linear improvement, rather than an intrinsic dependency?\n3. You report that consistency on AlpacaEval is significantly lower than on Arena-Hard. Does this indicate that the ALIGNEVAL method is less suitable for open-domain tasks?\n\nLIMITATIONS:\nThey discussed thelimitations of thier proposed benchmarking approach in section 5.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nCurrent alignment evaluation practices predominantly focus on open-ended response tasks, typically relying on human annotations or leveraging stronger LLMs as preference oracles. However, certain LLMs themselves possess strong evaluation capabilities. This paper systematically investigates the relationship between an LLM’s generation and evaluation abilities within alignment tasks, and observe a notable level of generation-evaluation consistency (GE-consistency). Building upon this finding, this paper proposes AlignEval, a novel alignment benchmark that enables indirect evaluation of LLMs’ alignment with human preferences without assessing their generated outputs directly. AlignEval eliminates the need for human annotations or LLM judges, providing a fully automated and cost-effective solution for alignment evaluation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. This paper addresses the consistency between the generation and evaluation capabilities of large language models (LLMs) by adopting a ranking-based consistency framework within alignment evaluation settings. \n2. This paper proposes AlignEval, a novel benchmark that enables alignment evaluation without relying on human annotations or LLM-based judges, offering a fully automated, cost-effective, and efficient solution for assessing LLM alignment with human preferences.\n\nWeaknesses:\n1. The analysis of the paper’s main conclusions lacks sufficient depth, leaving certain critical questions unaddressed. This may undermine the perceived novelty and overall impact of the paper.\n\nQUESTIONS:\n1. In the GE-consistency validation phase (Section 3), incorporating a few representative examples with manual evaluation would enhance the interpretability and persuasiveness of the observed consistency between generation and evaluation capabilities. \n2. While the current GE-consistency findings are primarily based on rank correlation, which provides valuable global insights, it raises concerns about the reliability of the absolute scores reported in subsequent experiments. \n3. In Figure 2, different models exhibit notable performance discrepancies across the two evaluation sets. It is recommended that the authors further investigate the underlying causes of this phenomenon, such as potential correlations with model scale or task generalization ability. A deeper analysis in this regard would strengthen the paper’s explanation of evaluation stability.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the relationship between large language models’ (LLMs) abilities to generate outputs aligned with human preferences and their abilities to evaluate such outputs. The authors formally define Generation–Evaluation Consistency (GE-consistency) as the Spearman correlation between LLM rankings by generation performance and by evaluation performance under a common preference oracle. They first measure GE-consistency using GPT‑4o on two benchmarks (AlpacaEval, Arena‑Hard), finding high correlations (ρ=0.839 and ρ=0.971) when filtering out inconsistent oracle judgments. They then study how GE-consistency varies with different oracles and demonstrate that stronger oracles yield higher consistency. Building on these findings, the paper proposes ALIGNEVAL, a benchmark that assesses LLM alignment by evaluating models in the judge role—using a fixed set of pairwise instances labeled by GPT‑4o or Claude-3.7-Sonnet—without requiring judges at test time. ALIGNEVAL matches or outperforms existing automatic benchmarks (e.g., AlpacaEval, Arena‑Hard) in correlating with ChatBot Arena human rankings (up to ρ=0.946), and when combined with IFEval, achieves ρ=0.94. The work highlights the importance of LLMs’ evaluation capabilities and introduces a scalable proxy for alignment assessment.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. Significance & Originality: Introduces the novel concept of GE‑consistency, connecting two facets of LLM behavior—generation and evaluation—and demonstrates its high correlation under strong oracles (Section 3) \n\n2. Benchmark Innovation: Proposes ALIGNEVAL, which repurposes judge instances to evaluate alignment without on-the-fly judging, reducing computational cost and preserving high correlation with human preferences (Section 4)\n\n3. Comprehensive Empirical Study: Evaluates 15–23 diverse LLMs across multiple oracles and instruction sets, with clear ablations on filtering and oracle strength (Figures 2, Table 1)\n\n4. Practical Impact: Demonstrates that ALIGNEVAL combined with IFEval matches or exceeds performance of established benchmarks, offering a cost‑effective alternative for both researchers and practitioners.\n\nWeaknesses\n1. Dependence on Oracle Quality: High GE‑consistency relies on very strong oracles (e.g., GPT‑4o); weaker oracles yield substantially lower consistency, limiting generalizability without access to frontier models (Figure 3) \n\n2. Oracle Self‑Bias: ALIGNEVAL exhibits self‑preference bias (models rank their own version highly), which may inflate correlations unless multiple oracles are used (Section 4.3)\n\n3. Limited Theoretical Insight: While empirical correlations are strong, the paper offers limited theoretical explanation for why GE‑consistency holds or how it evolves during training.\n\nQUESTIONS:\n1. Oracle Robustness: How does ALIGNEVAL perform if the preference labels are aggregated from multiple weaker oracles instead of a single strong one? Could ensembling or majority‑vote reduce bias and improve generality?\n\n2. Broader Alignment Aspects: Can the GE‑consistency framework be extended to other evaluation tasks, such as toxicity or factuality judgments? What challenges arise when the evaluation criterion is less subjective or more specialized?\n\n3. Theoretical Underpinning: Can the authors provide insight or hypotheses on why stronger evaluators tend to be stronger generators? For instance, is there evidence that underlying model capabilities (e.g., reasoning) drive both tasks?\n\nLIMITATIONS:\nThe authors acknowledge ALIGNEVAL is a proxy and vulnerable to adversarial fine‑tuning; combining with IFEval partially mitigates this. However, further discussion on multi‑oracle ensembling and detection of judge‑overfitting would strengthen the treatment of limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}