On Evaluating LLM Alignment by Evaluating LLMs as Judges
Authors: Yixin Liu
Pengfei Liu
Arman Cohan
Venue: NeurIPS 2025 poster ()
Keywords: alignment evaluation
LLM-as-a-judge
LLM evaluation
Primary area: deep_learning

Abstract: Alignment with human preferences is an important evaluation aspect of LLMs, requiring them to be helpful, honest, safe, and to precisely follow human instructions. Evaluating large language models' (LLMs) alignment typically involves directly assessing their open-ended responses, requiring human annotators or strong LLM judges. Conversely, LLMs themselves have also been extensively evaluated as judges for assessing alignment.  In this work, we examine the relationship between LLMs' generation and evaluation capabilities in aligning with human preferences. To this end, we first conduct a comprehensive analysis of the generation-evaluation consistency (GE-consistency) among various LLMs, revealing a strong correlation between their generation and evaluation capabilities when evaluated by a strong LLM preference oracle (GPT-4o). Utilizing this finding, we propose a benchmarking paradigm that measures LLM alignment with human preferences without directly evaluating their generated outputs, instead assessing LLMs in their role as evaluators. Our evaluation shows that our proposed benchmark, AlignEval, matches or surpasses widely used automatic LLM evaluation benchmarks, such as AlpacaEval and Arena-Hard, in capturing human preferences when ranking LLMs. Our study offers valuable insights into the connection between LLMs' generation and evaluation capabilities, and introduces a benchmark that assesses alignment without directly evaluating model outputs.

On Evaluating LLM Alignment by Evaluating LLMs as Judges
Yixin Liu!
Pengfei Liu?
Arman Cohan
Yale University
Shanghai Jiao Tong University
fyixin. liu, arman Sashann@yyll..e. edu
Abstract
Alignment with human preferences is important evaluation aspect of LLMs, quiring them to be helpful, honest, safe, and to precisely follow human instructions. Evaluating large language models' (LLMs) alignment typically involves directly assessing their open-ended responses, requiring human annotators or strong L.M judges. Conversely, LLMs themselves have also been extensively evaluated judges for assessing alignment. In this work, we examine the relationship between LLMs' generation and evaluation capabilities in aligning with human preferences. To this end, we first conduct a comprehensive analysis of the generation-evallatton consistency (GE-consistency) among various LLMs, revealing strong correla- tion between their generation and evaluation capabilities when evaluated by a strong LLM preference oracle. Utilizing this finding, we propose a benchmarking paradigm that measures LLM alignment with human preferences without directly evaluating their generated outputs, instead assessing LLMs in their role as evalua- tors. Our evaluation shows that our proposed benchmark, ALIGNEVAL, matches surpasses widely used automatic L.LM evaluation benchmarks, such as AlpacaEval and Arena-Hane Hard, in capturing human preferences when ranking LLMs. Our study offers valuable insights into the connection between LLMs' generation and eval- uation capabilities, and introduces benchmark that assesses alignment without directly evaluating model outputs
Introduction
Alignment with human preferences is key property of LLMSs, requiring them to accurately follow user instructions, generate responses that meet user needs, and reflect human values 29 Evaluating alignnen[[typicall
I.LM alignmenf [typically involves human evaluations of model outputs in response to various user queries, since it requires assessing L.LM' capabilities various open-ended tasks. However, such large-scale and reliable human evaluations are often complex, expensive, and time-consuming [46]. To scale this process, the widely used ChatBot Arena benchmark [5] relies on crowd-sourced annotations, where each instance consists of pairwise comparison between two model outputs for a given instruction. To reduce reliance on the expensive process of human evaluation, automatic alignment benchmarks have been proposed [4 22] 21 24], where LLMs judges used in place of human annotators, enabling faster evaluation while maintaining a reasonably high level of agreement with human preferences. Consequently, this LLMM-as--Juda paradigm has been commonly used for L.M evaluation in alignment and other open-ended tasks.
ALIGNEEAL available at
In this work, we use "L.LM alignment" to refer to I.LMs' general capabilities in following human instructions and providing helpful, high-quality responses, which goes beyond safety harmlessness alignment.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

GE-Consistency
Generator
Evaluator
Evaluation
LLMs
Responses
Ranking
Ranking
Judgments
LLMs
Task
Instruction
M2
Preference
Oracle
M2
M2
Figure 1:
Illustration of Generation Evaluation Consistency (GE-consistency), where LLMs' genera- tion and evaluation capability rankings are compared using a preference oracle.
Understanding and evaluating how reliable LLMs are as judges has also become an important discussed earlier, LLLMs-as-uddgss is an important paradigm for L.M evaluation. Furthermore, LLMs can also be used as judges in model training with preference optimization algorithms in place of fine-tuned reward models [35] The evaluations of LLMs as judges are typically conducted by comparing the predictions of LLMs against human annotations for the same evaluation tasks. Recent studies [47 19 have shown that frontier LLMs are strong judges for alignment evaluation. In particular, they can serve as effective out-of-the-box generative reward models (26). achieving performance competitive with fine-tuned discriminative reward models.
topic 43 19 25
We argue that it is important to study the connection between these two related capabilities of LLMs: generating responses that align with human preferences and evaluating whether responses are aligned with human preferences. Understanding this connection has significant implications for both model evaluation, by revealing the tinjconsistency between these capabilities 41 23 and model training, particularly in exploring the feasibility of self-imppovement where an LLM'ss training is supervised by its own judgments, which inherently requires model to accurately assess its own outputs [3] 42
Along this direction, prior work has examined the (in)consistency of individual LLMs when acting as both generator and validator [32 42 However, a comprehensive study is still lacking on whether performance rankings among various LLMs are consistent between the generator and evaluator roles, ie., whether models that rank higher in generation also tend to rank higher in evaluation.
Therefore, we conduct comprehensive analysis of generation-evaluue consistency) in LLM alignment ($3). We begin by formally defining GE-consistency (Figure [ measured as the correlation between the performance rankings of LLMs when evaluated as generators and as evaluators ($3.T). Next, we conduct controlled investigation operationalize this concept using& frontier LLM (GPT-4o) as the preference oracle (gold-standary evaluator) to assess the generation and evaluation capabilities of 15 LLMS ($3.2). We then extend the analysis to additional LLMs as oracles ($3.3). The results reveal that a high degree of GE-consistency, specifically Spearman's 's correlation of 0.96 between the performance rankings of LLMs as generators and as evaluators exists under specific conditions. These conditions include having strong preference oracle (GPT-4o), using challenging evaluation instances (from Frena-Hard (21), and applying an effective filtering strategy for selecting reliable task instances evaluate LLMs as evaluators.
consistency (GE-
Building on this finding, we demonstrate the utility of GE-consistency by proposing a benchmarking paradigm that assesses LLM alignment with human preferences without directly evaluating their generated outputs ) as done existing automated benchmarks like AlpacaEval (22 9 or Arena- Hard; instead, we evaluate LLMs in their roles as evaluators. This new evaluation paradigm is therefore more cost efficient since it can make use of existing preference annotations provided by either human annotators or L.LMs-as-Judges when evaluating new LLMss without requiring additional annotations. Our experiments show that the proposed benchmark, ALIGNE can match or even outperform widely used automatic evaluation benchmarks such as AlpacaEval and Arena-Hard in reflecting human preferences, using ChatBot Arena's 's LLM rankings as the gold standard. In particular, by combining ALIGNEVAL with IFEval (48,, another evaluation benchmark that does not rely on LLM judges, our benchmark achieves a Spearman's 's correlation of 0.94 with ChatBot Arena rankings across 23 LLMs.
To summarize, our contribution is two-fold: (1) we present the first comprehensive analysis of generation-evalual
consistency (GE-consistency) across multiple LLMs, demonstrating a strong correlation between their capabilities as generators and evaluators under certain evaluation conditions; and (2) propose and validate ALIGNEVAL, effective automatic benchmarking paradigm for

---

assessing LLM alignment without relying on human annotators or LLM- based evaluators, achieving performance competitive with established benchmarks that utilize -uddees-Judges while reducing the evaluation cost. Crucially, our study emphasizes the significance of L.LMs evaluation capabilities as an essential aspect of L.I. evaluation, highlighting the need for further research in this direction.
2 Related Work
Evaluating LLM Alignment and Instruction-Follovinng Alignment to human preferences is key evaluation criterion for LLMs, typically assessed by examining model responses to curated instructions spanning diverse use cases 29 4. Such evaluations can involve expert annotators [37
45 or crowd workers [8] However, due to the high cost of human evaluation, ChatBot Arena [5], crowd-annotated leaderboard, remains arguably the only benchmark offering human evaluations across a wide range of LLMs. Consequently, automatic evaluation benchmarks have been proposed, using LLMs as judges in place of human evaluators [45] 22 21 24 18). which demonstrate a high level of correlations with human evaluations. While most evaluation methods focus on free-form outputs and thus require either human LLM-based evaluators, alternative approaches assess instruction-followe
38
capabilities using rule-based, programmatic evaluation [48] 40 [.5..
MixEval 28), in contrast, reduces reliance LLMssas-Judges by benchmarking models on short- answer or multiple-choice questions that are similar to user queries mined from existing benchmarks.
Evaluating LLMs Judges. LLMs-as-judges is an important component for both model evalua- tion [22 and training in terms distillation or seff-improvement [34] 42). As a result, evaluating LLMs judges has become an important research topic (20), with human evaluations serving the
gold standard 8 43 25]. In addition to instance-level studies comparing human and LLM judgments on specific cutructtionouuputt instances, system-level evaluations have also been conducted assess whether LLMs can approximate human-derived rankings of LLMs' alignment levels 13 I1. More- over, -udgee are closely related to generative reward models (GRMs) 44
26]. and have been evaluated in reward modeling settings such as RewardBench [19. Recent work shows that frontier LLMs, when used judges or GRMs, are competitive with strong fine-tuned reward models 47 10]
Relationship Between LLMs' Generation and Evaluation Capabilities. West et al. proposes the "Generative AI Paradox" and demonstrates that LLMs can possess stronger generation capabilities than evaluation capabilities under certain circumstances. Li et al. [23], Rodriguez et al. 30 analyze gencrator-validator consistency (GV-consistency) and find that LLMs can behave inconsistently in these two roles. For example, an LLM may judge its own generated answer to a math problem as incorrect, prefer different option over its own response in multiple-choice question-answeriig task. Song et al. [22 demonstrates the opposite gap, that verification is easier than generation in certain tasks, and shows that this is critical for enabling LLM self-improvement [22 In this study, we investigate the consistency between LLMs' generation and evaluation capabilities regarding their alignment with human preferences. This measurement is related yet distinct from GV-consistency, as the evaluation task here involves assessing responses to instructions rather than validating or verifying objective correctness. We further discuss this and other differences in the next section.
3 Examining LLM Generation-Evaluatii Consistency
Our key assumption behind evaluating LLMs' alignment by assessing their correlations with human evaluators when LLMs are acting as judges is a high consistency of their inneration-evaluaatie capabilities - if an LLM is better at evaluating whether given responses align with a preference oracle (which may be human or M.M--dd,,d), its generated responses are expected to be better aligned with the same oracle well. Therefore, we begin by investigating this inneration-evalaatii consistency.
3.1 Defining Evaluuttonn Consistency
We first formally define gemeration-evaluatio consistency in LLM alignment. Given a set of LLMs M : {M,...., My ). a preference oracle J. and a set of input instructions I, we derive a ranking of the LLMs' generation capabilities by applying J to evaluate their responses to I. We denote this ranking as R(9) = (r(9)) (r.))..... r(0)). Here, ,(9) is the rank assigned to model M,, which determined by its overall performance score aggregated across I assigned by the preference oracle J.
3

---

Similarly, ranking of their evaluation capabilities is derived by assessing whether the LLMs produce evaluation results that match those of the preference oracle J when applied responses to the instruction set I, typically by comparing the predictions from pointwise scoring or pairwise comparison. We denote this as R(+) (rl ..... ...,,,(())) ) Here, r
[c) is the rank assigned to model M,,
which is determined by its overall prediction accuracy using the preference oracle's evaluations as the gold standard.
Then, the generation-evaluatie consistency c can be defined as
cM;J.I)) == C(H R))))
(I)
where C' is a certain correlation metric such as Spearman's 's ranking correlation coefficient. Below, we will denote Eq. [ as GE-consistency for brevity. We note that a related characteristic of LLMs, the
-aaliator-validaoor consistency, or GV-consistency, has been investigated by previous studies 23 30]
However, there is a key difference between GE-consistency and GV-consisteney. Specifically, the GV-consistency is defined with a single LLM regarding its inherent consistency e.g., whether an
LLM acting as a validator deems its own response to a math problem correct. On the other hand, the GE-consistency is ranking-level measure defined over set of LLMs it reflects whether an LLM that performs better at evaluation than another also performs better at generation. Therefore, although prior work has identified considerable GV-inconsistency in various LLMs (23 30), GE-consistency remains a distinct and relatively underexplored property. A high level of Conosisityy is still possible, it only requires relative alignment between generation and evaluation abilities i... better evaluators also tend to be better generators, even if the two capabilities are not individually consistent. Supporting this, there is evidence that more capable LLMs tend to be better judges or generative reward models [25 47
3.2 Measuring Conerattio-Evaluaatie Consistency using a Strong LLM as Preference Oracle
Having defined the GE-consistency, we conduct a case study to measure it. Specifically, we use strong LLM as the preference oraele to judge both the generation and evaluation capabilities of a set of LLMs. This approach has two main advantages: (1) Using an L.M as the oracle (instead of humans) allows more reproducible and scalable experiments. (2) By comparing the evaluation and generation rankings, both judged by the same strong LLM, we can assess how well evaluation performance predicts generation quality. GE-consistency is high, it suggests that LLMs' evaluation capabilities can be used to estimate their overall alignment, reducing the need for human or LLM- judge assessments on every new response. In summary, the evaluation rankings closely match the generation rankings, as judged by the strong LLM, evaluation scores can serve as a reliable proxy for generation quality evaluated by LLMssas-Judges
3.2.1 Experimental Settings
To study the GE- GE-consistency, we require: (1) set of input instructions for LLMs to respond to, (2) a collection of L.LMs to compare, and (3) a strong L.LM to serve as the preference oracle for evaluating both generation and evaluation tasks. Using these components, we measure and compare each model's ability to generate aligned responses and accurately evaluate other models' outputs.
Instruction Set. We select data sources for the instruction set (i.e., evaluation instances) I required to measure the GE-consistency: AlpacaEval 22 9 and Arena-Hard (21), with 805 and 500 instruc- tions, respectively. Both AlpacaEval and Arena-Hard are LLM evaluation benchmarks for which the instructions are carefully selected to reflect various user needs, making them suitable for our investigation.
Evaluation Oracle. We choose strong frontier LLM the time of writing, GPT-4o (g-t-40-2024- 08-06), as the evaluation oracle or judge J. following similar practices in AlpacaEval and Arena-Hard where 4PT-45iis is chosen the evaluator compare LLM outputs.
LLMS to Evaluate. We select 15 post-trained LLMs as the LLM set M to be evaluated. The detailed information of these LLMs is in Appendix A These LLMs provide succinct coverage of model sizes and families, and most are open-weight models for better reproducibility and accessibility
"gpt-4- 1144--106.- -00-vrr the default model.
'Proprietary models have the risk of version updates and deprecation. Moreover, evaluating single LLM in study requires around 80M tokens, making the full study costs prohibitive for certain proprietary models.

---

Obtaining LLM Generation Performance Ranking, To obtain the ranking of LLMs' generation capabilities, R(o), we apply the evaluation oracle J to evaluate the LLMs' outputs for the instruction set I. The evaluation is conducted in the manner of pairwise comparison, since it has been widely used in both AlpacaEval and Arena-Hard. On both AlpacaEval and Arena-Hard, we follow their settings by choosing either gpt-4-1 -oeerrewww or gpt-4-03 14 as the baseline system, respectively. The different LLMs' outputs are then compared against the outputs of the baseline system, and their performance score is derived by aggregating their win rates across the instruction set. The prompt template used for the pairwise comparison is included in Appendix [B] which directly requires the LLM to predict the better output given an output pair without explanations or reasoning process. was first introduced in Zeng et al. 43 and has demonstrated better effectiveness compared to prompts used in AlpacaEval or Arena-Hard despite its simpler format [25]. For more reliable evaluation, each output pair is evaluated twice by swapping the order of the two outputs.
Obtaining LLM Evaluation Performance Ranking. To derive the ranking of LLMs' evaluation capabilities, R(e,, we propose evaluate them using the evaluation result of the preference oracle as the ground-truth. Specifically, given an instruction x, a pair of outputs and y2, and the evaluation result of the preference oracle J. 8 E (1,2) (the index of the better output), an LLM M can then be evaluated as an evaluator by comparing its prediction 8 against the ground-truth prediction 8. Here, accuracy or inter-annotator agreement can be used as the evaluation metric. We choose to use inter-annotator agreement, specifically Cohen's 's Kappa, as the main metric instead of accuracy, since it can better reflect model performance when the label distribution unbalanced.
Instance Selection and Filtering for Evaluating LLMs' Evaluation Performance. As discussed above, evaluating LLMs' evaluation capabilities in the pairwise comparison setting requires an instruction x and a pair of outputs 92). To construct the task instances for this evaluation, we reuse the preference oracle's ' annotations, i.e., the ground-truth label indicating the better system, from its comparisons between L.M outputs and those of the baseline system originally used to assess generation capabilitiis. Given instruction set I of size I. and an L.M set M of size N, this yields total of 2LN task instances, since each pairwise comparison is conducted twice with swapped output orderings, i.e., y1 @ y2 and y/2 @ y1. We then apply a filtering process based on the (inoositeinn of the preference oracle: if the oracle produces different predictions for y1 @ y/2 and 1/2 @ y1, both instances (I, 41, 82) and (I, y/2 31) are discarded. Previous studies [36 43 25]
have found that various LLMS exhibit non-trivial level of such inconsistency. which we posit indicates uncertainty in the oracle's 's prediction. We provide further discussion in $32..2
3.2.2 Result Analysis
Figure 2 shows the correlation of the LLMs' generation and evaluation performance both AlpacaF- val and Arena-Hard. The generation performance is measured by their win rates against the baseline system (GPT-4) evaluated by the preference oracle, while the evaluation performance is measured by their agreement (Cohen's 's Kappa) with the preference oracle on the evaluation task instances after filtering as described above.
The results in Figure 2 demonstrate a 0.839 and 0.971 Spearman's 's rank correlation coefficient on AlpacaEval and Arena- Arena-Hard, respectively. It indicates relatively high level of GE-consistency among the evaluated LLMs with GPT-40 as the preference oracle, especially on Arena-Hard The difference between AlpacaEval and Arena- Hard is likely due to the differences in instruction types in the two datasets AlpacaEval contains more open-ended instructions, whereas Arena-Hard focuses more challenging, technical instructions 22 more objective and stable.
21 24
This makes evaluation the latter
In Appendix C we provide further results on WildBench (24 with the same evaluation setting. Com- pared to Arena-Hard and AlpacaEval, WildBench offers a more balanced distribution of instruction types. On WildBench, we observe a GE-consistency of 0.938 Spearman's 's correlation, suggesting that GE-consistency is a general pattern that holds across various types of instructions, including open-domain tasks.
Importance of Consistency Filtering. As discussed in $ the task instances used evaluate I.LMs' evaluation performance are filtered based on whether the preference oracle produces consistent predictions when the order of outputs in the pairwise comparison swapped. On AlpacaEval
'Appendix shows the stability of this consistency.
5

---

Generation Pislurtion Performanor AlpacaPiral (p (830)
Generationn Brsiandor Performenee Anea-Bard
docars 25440-5
Tarn 277
qure
2
Exradd bag
qpcc 72>
1156
0.3
Dealutionn Pi Leformmmee
Laze-3-75
044
Epeel 5/756
114
02 tibodqe
Tezaal L8b
0.2
Lexa-S4b
0.35
0.50
633
0:30
0:85
0:30
0.50
0.50
0.26
0.2
0.4
00
Generasion Peefocmance
Geverntion Performance
Figure 2: Generation and evaluation performance of various LLMs with preference oracle. The X-axis shows the generation performance in terms of LLMs' win rates against the baseline system (GPT-4) evaluated by the preference oracle. The Y-axis shows the evaluation rate performance in terms of LLMs' agreement rate (Cohen's 's Kappa) with the preference oracle on filtered evaluation task instances.
26 the
and Arena-Hard, 58.3% and 50.7% of instances, respectively, are filtered due inconsistent predictions from the oracle. Table [ highlights the importance of this filtering step, as the correlation between generation and evaluation rankings drops significantly without it. This is likely because the filtering process removes two types of unreliable instances: (1) cases where the outputs are too similar to allow for an objective preference, and (2) cases where the preference oracle is uncertain and effectively guessing.
Remark. The high level of GE-consistenge ob- served in this experiment, with Arena-Hard as the instruction set and GPT-40 as the prefer- ence oracle, suggests the possibility of replac- ing the LLM-as-Judge evaluation method with evaluating LLMs as judges for LLM alignment evaluation. Specifically, since an LLM's perfor- mance as an evaluator strongly correlates with its generation performance, we can construct an evaluation benchmark using fixed set of evaluation task instances annotated by the preference oracle assess LLM alignment. This allows evaluating various LLMs without requiring an LLM judge to directly assess their generated outputs. In $4 we conduct further investigatin gation along this direction.
Table 1: The impact of consistency filtering on the measurement of GE-consistency.
Spearman's Rank Correlation
AlpacaEval
Arena-Hard
w/o filtering
0.743
0.793
w/ filtering
0.839
0.971
3.3 GE-Consistency with Various LLMs as Preference Oracle
In $32.2 the GF-consistency is measured with GPT-4o as the preference oracle. We now extend this analysis to other LLMs as preference oracles, especially the ones that are less capable than GPT-4o, to have more comprchensive evaluation. To this end, use the 15 LLMs (MM) evaluated in $32. as the preference oracles, which allows us to reuse the collected data annotations. The detailed experimental settings are in Appendix D
Figure ssooss the evaluation results: (1) The GE-consistency varies with the choice of preference oracle, with larger, more capable LLMs leading to higher GE-consistency in general. For example, GE-consistency with 1lama-3-70b as the preference oracle on Arena-Hard approximately 0.9 in terms of Spearman's 's correlation. (2) The GE-consistency on Arena- is generally higher than GE-consistency on AlpacaEval, aligning with the trend observed with GPT-40 as the preference oracle in $3.2.2

---

Avpa Arem-Haad
056
GE- JB-Coosistenry
0.6
0.6
0.4
0.5
0.1
Ilanaa 2-58
gammma----0 2-20
qpen L333b
GN-3 G9-3 1uste-
qwaa-2.5-7th
mini 2t1 6-F-11
Hana & iib
talo A-do--pa-a 7itl
Baass Baans-1L2-10 2.2-78b
Bana 7ith
Bann 1-77h
gremma
Preference Oracle
Figure 3: The GÉ-consistency with different LLMs as the preference oracle. Spearman's 's correlation between the generation and evaluation capability rankings LLMs under different preference oracles is shown on the Y-axis. The preference oracles are sorted in ascending order of their corresponding co-niisssy levels on the X-aiis.
Remark. The GE-consistency observed in this analysis is generally much lower than the GE- consistency with GPT-4o as the preference oracle, especially with small models such as llama-3- 8b. This highlights the impact of the preference oracle on GE-consistency intuitivell, weaker preference oracle that produces random evaluations will yield zero GE-consistency, whereas stronger LLMs the oracle are more likely produce stable and meaningful results.
Evaluating LLM Alignment by Evaluating LLM Evaluation Capabilities
Our analysis in shows that there exists a high level of consistency among LLMs' generation and evaluation capabilities, given strong L.MM the preference oracle. Therefore, now investigate whether an evaluation benchmark that measures LLMs' evaluation capabilities can be used to evaluate the LLMs' alignment human preferences. In other words, such a benchmark aims to achieve the same goal as the automatic LLM benchmarks such as AlpacaEval and Arena-Hard, without using LLMs as judges to evaluate LLMs' outputs. To this end, we first introduce the designed benchmark. ALIGNEVAL, in 44. and the experimental settings in $4.2. then provide an in-depth analysis in $4.3
4.1 Building ALIGNEVAL
$32.22 has demonstrated that a high GE-consistency among various LLMs when GPT-4o is used as preference oracle and Arena-Hard is selected as the instruction set. We can then construct an evaluation benchmark for assessing LLMs' evaluation capabilities, where each instance consists of better
an instruction, a pair of outputs, and a label indicating the preference oracle's judgment output. For the input set, we reuse the instances from $3.2.1 consisting of pairwise comparisons between 15 LLMs and the bascline system on Arena- Hard, filtered to retain only those where GPT-4o produces consistent results. To further reduce computational cost, only one task instance is kept for each pairwise comparison by randomly selecting a single output order. This results in 267 I input instances in total. Regarding the gold-standard labels, apart from the annotations of GPT-4o, obtain the evaluation results from Claude-3.7 Claude-3.7-Sonnet (2), a recently released frontier LLM. We name the two versions of ALIGNEVAL ALIGNEVAL-GPT and ALIGNEVAL-CLAUT accordingly.
also
key advantage of ALIGNEVAL over benchmarks like AlpacaEvall or Arena-Hard is that does not require L.M judge to evaluate new models. Once constructed using preference oracle, ALIGNEVAL. instances can be reused benchmark any LLM'' evaluation performance, significantly reducing cost especially compared to setups like Arena-Hard that involve expensive API calls to
7

---

proprietary LLMs. This evaluation paradigm can also extend to human preferences, where annotations are collected once and reused for evaluating future models. We compare different automatic alignment benchmarks in Table and further discuss the experimental settings in $4.2
4.2 Experimental Settings
LLMs to Benchmark. To enable a more robust and comprchensive evaluation, apart from the 15 LLMs evaluated in $3.2.1 8 additional LLMs added to the set of LLMs to be benchmarked. These include strong models such as Gemini-2.0-Flan 2...2.000FFhhh00- and Llama-3.1 ---stt-utttt [14]. The selected LLMs span range of model families, sizes, and capabilities ensure diversity and representativeness. The detailed information of these LLMs is in Appendix A
Evaluation of Automatic Benchmarks. To evaluate and compare the effectiveness of automatic benchmarks measuring LLMs' alignment to human preferences, we use the human evaluation results in ChatBot Arena Leaderboard [5] as the gold standard ranking of LLMs' alignment level. The automatic benchmarks can then be evaluated by measuring the correlation of their produced LLM rankings with the gold-stante standadd ranking. We note that Chatbot Arena is not true "gold standard," given its opaque data collection process and potential biases [31]. Nonetheless, we adopt it due to the lack of better alternatives and to remain consistent with common practice 22 to mitigate possible biases, we use its style-controlled rankings
28 Morcover,
Below, we introduce the baseline benchmarks compared, with Table summarizing their differences.
Baselines: Benchmarks using LLMs Judges. Both AlpacaEval and Frena-Hard un- der their default configurations are included as baselines. For AlpacaEval, both the raw win rate and the length-cone contth-controlled win rate are reported. For Arena-Hard both the raw scores and the style-controlled scores are reported. In addition, we compare evaluation results using G-T-4o-as- Judge on Arena- Hard, following the same setup as in 3.2.1 This serves as the upper bound for ALIGNEVAL-GPT which aims to approximate these evaluation results.
Table 2:
Comparison of alignment evaluation benchmarks in terms of the number of instances, the need for an LLM-judge, and the estimated cost of proprietary LLM APIS for evaluating one L.M.
Benchmark
#Instances
LLM-judge
API Cost
AlpacaEval Arena-Hard GPT4o-Judge
805
$10
500
$20
500
$2
IFEval
541
X
$0
MixEval
1000
$0.1
HelpSteer3
4188
X
$0
Bench- marks. Instead of relying on the LLMs-as- Judges evaluation paradigms, benchmarks can also be constructed based on (verifiable) ground-truth. Among them, IFEval 48 evaluates LLMs on their ability follow specific instructions, using rule-based and programmatic evaluation methods, which does not require an LLM as judge. It has both loose and strict grading schemas. Mix- Eval (28] constructs task instances by matching real- world user queries with test examples from existing benchmarks. Evaluation compares model responses against gold-standard answers. The task instances include multiple-choice questions, which do not require an LLM judge, and short-answer questions, where an LLM judge is required, but only to compare outputs, making the task simpler and less computationally intensive than settings like Arena- Hard. The MixEval-Hae -ard subset is selected due to its better effectiveness.
Baselines:
Bouund-Truth-Baan
ALIGNEVAL
2671
X
$0
Baseline: LLM-Judge Evaluation using Human Preferences as Gold-Standard. ALIGNEVAL evaluated L.MSs' evaluation capabilities using strong LLMs as the preference oracle. A similar evaluation method is to use human preferences as the gold standard. To this end, we use the human annotations in HelpSteer?? 33 to construct buman--refereen judges. Specifically, each instance in HelpSteer3 contains pairwise comparison annotations two outputs for given instruction, collected from multiple human annotators. To reduce annotation noise, we retain only instances with preference strength 3, indicating that one output clearly preferred over the other based on their data collection protocol.
dataset for evaluating LLMs as
httpe://ccoodd.oogll..oo.. wow/vertee-aaee si/ generative- wwt.a/////////////////a/a/// Offahh
httpe://aarrmaaall/laaooooo
28- style-llooooooolll
The ranking snapshot was taken on April 18, 2025.
8

---

Table 3: LLM Performance on Various Benchmarks, The LLMs are ordered by their style-controlled ChatBot Arena Rankings. For AlpacaEval, both the raw and the length-controlled (LC) scores are reported. For Arena-Hard, both the raw and the style-controlled (SC) scores are reported. The highest score for each benchmark shown in bold. The top-3 scores are underlined.
ChatBot-
AlpacaEval
Areua-Hard
GPT4o-
ALIGNEVAL
Judge
MixEval
HelpSteer3
GPT
gemini-2.00ffal
10
71.7
53.9
83.0
73.3
66.7
63.3
91.5
69.5
80.8
73.4
gemini-2.0-flahhiin
66.0
48.6
80.3
69.0
64,6
51.7
90.7
65.5
72.9
70.1
gpt-to-2024-05113
68.7
71.3
claude-3.-sooncc
83.2
67.0
67.7
1lama-3.1-405b
33.6
67.2
57.5
66.0
89.3
73.1
Ilama-3.. 3.3-70b
39.7
38.6
62.9
61.0
54.6
60.9
92.2
70.1
74.1
67.2
apt 1pi-4a-mini-2024 2i4-mmii-2224-77...
38.1
49.2
74.5
70.4
62.3
47.6
84.3
66.1
74.5
67.0
claudo-3.5-baiku
38
27.7
46.4
72.1
81.8
59.2
54.8
83.8
$8.0
62.0
62.0
gemini-1.5-ftasb
41
52.3
57.0
80.2
77.5
59.6
47.4
89.3
60.2
67.6
61.4
gwen-2.5-72b
51
50.0
47.7
77.4
69.0
63.9
52.6
87.3
70.7
79.2
72.7
1lama-3.1-70b
56
32.1
35.6
56.7
56.5
55.3
62.2
87.9
69.3
71.7
64,8
2-27h
30.3
47.4
51.5
49.1
43.2
47.0
81.7
49.1
47.9
43.6
llama-3-70b
36.0
38.1
55.5
49.8
57.0
84.1
58.8
mistral-amall-24b
63.5
59.6
49.5
qwen-2-72h
67.9
3emmn-2-9h
38.6
38.5
49.3
gwen-1.555
92
24.2
34.3
36.3
44.6
40.1
45.7
61.8
46.1
51.6
47.3
Ilama-3358 3-8b
18.6
20.0
21.3
25.8
27.3
36.7
77.1
23.5
6.4
4.9
gpt-3. -0--3-5-uubb-0015
100
10.8
22.6
26.3
43.5
30.0
41.4
72.5
37.2
27.0
23.3
Ilama 3.1
103
23.9
24.4
28.7
27.7
31.5
44.8
79.6
36.5
25.8
24.4
qwen- 1.5.3h
104
22.9
27.8
27.5
35.6
33.8
39.6
56.5
43.1
26.2
24.8
gemma-2-26
110
27.1
30.0
18.9
15.5
18.1
26.2
60.8
16.1
10.7
10.5
ulu-22-pp-776
119
17.1
22.1
16.1
22.0
23.3
44.7
61.7
40.7
25.5
22.8
ALIGNEVAL++: Combining ALIGNEVAL with IFEval. Both ALIGNEVAL. and IFEval evaluate L.M alignment without relying on LLMs as judges, and they are complementary in nature ALIGNEVAL assesses an LLM's understanding of what constitutes a well-aligned output, analogous to planning step, while IFEval evaluates precise instruction-follovii We refer to this combined benchmarking approach as ALIGNEVAL+, which evaluates LLMs by averaging their rankings from ALIGNEVAL and IFEval.
ability, analogous to execution step.
4.3 Result Analysis
Table presents the evaluation results of the 23 LLMs across different benchmarks, ordered by their mlle-conttolled rankings on ChatBot Arena, offering detailed view of how each benchmark correlates with human evaluations. We detail our main findings below:
(1) We observe that models with high ChatBot Arena rankings generally perform well across benchmarks. For example, gemini-2.00flase the top-ranked LLM in our study, ranks in the top 3 on 8 out of 10 benchmarks.
(2) We observe self- ceff-erffeeene bias in ALIGNEVAL: ALIGNEVAL-GPA ranks 111-4o-002405-11 second, while ALIGNÉVAL-CLAUS ranks LLMs-as-Judges evaluation with gpt-40-20224008 200-202-----66 which ranks its earlier version first. While expected, such bias may be reduced using multiple preference oracles. Notably, both ALIGNEVAL variants consistently place gemini-2.0-flash in the top two.
highest. A similar bias appears the
Table presents the performance of different automatic alignment benchmarks regarding their correlations with the style-controlled ChatBot Arena LLM ranking, with and without being used together with IFEval(-Loose). We note two important findings:
(1) IFEval, especially with the loose grading criteria (IFEval-Loose). achieves strong performance, despite being originally developed for evaluating formatting cnaruuctionfollowiiii

---

(2) Notably, ALIGNEVAL combined with IFEval performs better or comparably to LLM-judge- based benchmarks. This demonstrates that +LIGNEVAL+ is an effective automatic evaluation benchmark for L.M alignment that does not rely on LLM judges.
Regarding the benchmark performance without 1FEval, we observe the following:
Table 4: Spearman's correlation of LLM bench- marks with ChatBot Arena rankings, with and without rank averaging with IFEval-Loose Arena- Hard-SC is the stylc-controlled score, AlpacaEval- LC the length-controlled score.
(I) ALIGNEVAL, and especially ALIGNEVAL- CLAUDE, achieves comparable or supe- rior performance to L.MMs-as-Judges-assee benchmarks.
Benchmark
w/ IFEval
w/ IFEval
(2) Compared to ground-truth-basee bench- marks, ALIINNEALL significayy outperforms MixEval. This suggests that assessing models' evaluation capabilities on instruction outputs, as done in ALIGNEVAL, is more effective than MixEval's approach of using similar benchmark examples.
IFEval(-Loosc)
0.919
0.919
IFEval-Strict
0.911
0.880
Arena-Hard
0.946
0.905
Arena-Hard-SC
0.936
0.882
A AlpacaEval
0.891
0.761
AlpacaEval-LC
0.925
0.746
GPT Judge
0.958
0.911
(3) Benchmarking LLMs as judges on Help- Steer3 is relatively less effective than on ALIGNEVAL. One likely reason is that in- struction set is not as carefully curated as those in alignment benchmarks like Arena-Hard. Prior work has proposed methods for curating high- quality instruction sets [21 (4] 28), which could improve HelpSteer3 through instance filtering, direction we leave for future work.
MixEval
0.900
0.816
HelpSteer3
0.904
0.813
ALIGNEVAL-GPT
0.946
0.856
ALIGNEVAL-CLAUUE
0.946
0.885
(4) All alignment benchmarks show lower correlations with ChatBot Arena than reported at release, especially AlpacaEval and MixEval (Appendix [E). This is likely due to the stronger LLMs evaluated here, which may make the task more challenging than in earlier studies.
Discussion and Conclusion
The results in $ demonstrate the promising effectiveness of ALIGNEVAL for evaluating LLM alignment. However, emphasize that it is proxy evaluation by design and may be vulnerable to adversarial attacks. For example, fine-tuning an L.M to act as a judge could artificially boost its ALIGNEVAL ranking without meaningfully improving its alignment. Combining ALIGNEVAL with IFEval helps mitigate this risk, as it requires the model to still be able to generate accurate responses instructions. Still, this does not fully eliminate the concern, and we leave the development of more robust evaluation settings for future work. Nonetheless, ALIGNEVAL remains a valuable benchmark for benign evaluators, such model developers, to assess and better understand LLM alignment and capabilities without relying on human or LLM judges for iterative output annotations.
We argue that more critical insight of our study is that the capability of LLMs to evaluate whether outputs align with human preferences itself an important aspect of LLM evaluation. Specifically, in $3 demonstrate strong correlation between generation and evaluation capabilities of LLMs when assessed using a strong preference oracle; in $ we leverage this finding to construct practical LLM evaluation benchmark. Moreover, the studied property, -nvalution-evaluutioo consistency, has broader implications for both training and evaluation LLMs. For example, if this consistency holds during training. it suggests the feasibility of self-improvement: stronger model could better supervise its own training process. We call for future work to further investigate and better understand GE-consistency and its implications for LLM development.
Acknowledgements
We are grateful for the TPU compute support provided by the Google TRC program and for the OpenAl API credits support provided by OpenAl's Researcher Access Program.
10

---

References
[1] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida,
J. Altenschmidt, S. Altman, S. Anadkat, et al. GPT4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[2] Anthropic. Claude 3.7 sonnet. Available via API, 2025. URL httph:///ww: http://wwww.aatrpp.....o
Com//laade/someee Released February 24, 2025. Hybrid reasoning model.
[3] J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, B. Hui, L. Ji, M. Li, J. Lin, R. Lin, D. Liu, G. Liu, C. Lu, K. Lu, J. Ma, R. Men, X. Ren, X. Ren, C. Tan,
S. Tan, J. Tu, P. Wang. S. Wang, W. Wang, S. Wu, B. Xu, J. Xu, A. Yang, H. Yang. J. Yang,
S. Yang, Y. Yao, B. Yu, H. Yuan, Z. Yuan, J. Zhang, X. Zhang, Y. Zhang, Z. Zhang, C. Zhou,
J. Zhou, X. Zhou, and T. Zhu. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
[4] Y. Bai, Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05880 2022.
[5] W.-L. Chiang, L. Zheng, Y. Sheng, A. N. Angelopoulos, Li, D. Li, B. Zhu, H. Zhang, M. Jordan, J. E. Gonzalez, and I. Stoica. Chatbot arena: An open platform for evaluating LLMs by human preference. In Forty-first International Conference on Machine Learning, 2024. URL http/porrrreeeeeeeeorrrrrr fit.orrrmnndd3-MMKK
[6] Claude. Introducing claude, Mar 2023. introducing-cles claudde
URL https: htwp:://wwwww..e
[7] A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Roziere, B. Biron,
B. Tang, B. Chern, C. Caucheteux, C. Nayak, C. Bi, C. Marra, C. McConnell, C. Keller, C. Tourct, C. Wu, C. Wong, C. C. Ferrer, C. Nikolaidis, D. Allonsius, D. Song, D. Pintz, D. Livshits. D. Esiobu, D. Choudhary, D. Mahajan, D. Garcia-Olano, D. Perino, D. Hupkes, E. Lakomkin, E. AlBadawy, E. Lobanova, E. Dinan, E. M. Smith, F. Radenovic, F. Zhang, G. Synnaeve, G. Lee, G. L. Anderson, G. Nail, G. Mialon, G. Pang, G. Cucurell, H. Nguyen, H. Korevaar, H. Xu, H. Touvron, I. Zarov, I. A. Ibarra, I. Kloumann, I. Misra, I. Evtimov, J. Copet, Lee, J. Geffert, J. Vranes, Park, J. Mahadeokar, Shah, van der Linde, J. Bil- lock, J. Hong. J. Lee, J. Fu, J. Chi, J. Huang, J. Liu, J. Wang. J. Yu, J. Bitton, J. Spisak, J. Park, J. Rocca, J. Johnstun, J. Saxe, Jia, K. V. Alwala, K. Upasani, K. Plawiak, K. Li, K. Heafield, K. Stone, K. El-Arini, K. Iyer, K. Malik, K. Chiu, K. Bhalla, L. Mantala-Yeary, L. van der Maaten, L. Chen, L. Tan, L. Jenkins, L. Martin, L. Madaan, L. Malo, L.. Blecher, L. Landzaat, L. de Oliveira, M. Muzzi, M. Pasupuleti, M. Singh, M. Paluri, M. Kardas, M. Oldham, M. Rita, M. Pavlova, M. Kambadur, M. Lewis, M. Si, M. K. Singh, M. Hassan, N. Goyal, N. Torabi, N. Bashlykov, N. Bogoychev, N. Chatterji, O. Duchenne, O. Çelebi. P. Alrassy, P. Zhang. P. Li, P. Vasic, P. Weng, P. Bhargava, P. Dubal, P. Krishnan, P. S. Koura, P. Xu, Q. He, Q. Dong, R. Srinivasan, R. Ganapathy, R. Calderer, R. S. Cabral, R. Stojnic, R. Raileanu, R. Gird- har, R. Patel, R. Sauvestre, R. Polidoro, R. Sumbaly, R. Taylor, R. Silva, R. Hou, R. Wang, S. Hosseini, S. Chennabasappa, S. Singh, S. Bell, S. S. Kim, S. Edunov, S. Nie, S. Narang, S. Raparthy, S. Shen, S. Wan, S. Bhosale, S. Zhang. S. Vandenhende, S. Batra, S. Whitman, S. Sootla, S. Collot, S. Gururangan, S. Borodinsky. T. Herman, T. Fowler, T. Sheasha, T. Geor- giou, T. Scialom, T. Speckbacher, T. Mihaylov, T. Xiao, U. Karn, V. Goswami, V. Gupta, V. Ramanathan, V. Kerkez, V. Gonguet, V. Do. V. Vogeti, V. Petrovic, W. Chu, W. Xiong, W. Fu, W. Meers, X. Martinet, X. Wang. X. E. Tan, X. Xie, X. Jia, X. Wang. Y. Goldschlag, Y. Gaur, Y. Babaei, Y. Wen, Y. Song, Y. Zhang, Y. Li, Y. Mao, Z. D. Coudert, Z. Yan, Z. Chen, Z. Papakipos, A. Singh, A. Grattafiori, A. Jain, A. Kelsey, A. Shajnfeld, A. Gangidi. A. Victoria, A. Goldstand, A. Menon, A. Sharma, A. Boesenberg, A. Vaughan, A. Baevski, A. Feinstein, A. Kallet, A. Sangani, A. Yunus, A. Lupu, A. Alvarado, A. Caples, A. Gu, A. Ho, A. Poulton, A. Ryan, A. Ramehandani, A. Franco, A. Saraf, A. Chowdhury, A. Gabriel, A. Bharambe, A. Eisenman, A. Yazdan, B. James, B. Maurer, B. Leonhardi, B. Huang, B. Loyd, B. D. Paola, B. Paranjape, B. Liu, B. Wu, B. Ni, B. Hancock, B. Wasti, B. Spence, B. Stojkovic, B. Gamido, B. Montalvo, C. Parker, C. Burton, C. Mejia, C. Wang, C. Kim, C. Zhou, C. Hu, C.H.. Chu,
11

---

C. Cai, C. Tindal, C. Feichtenhofer, D. Civin, D. Beaty, D. Kreymer, D. Li, D. Wyatt. D. Adkins, D. Xu, D. Testuggine, D. David, D. Parikh, D. Liskovich, D. Foss, D. Wang. D. Le, D. Holland, E. Dowling, E. Jamil, E. Montgomery, E. Presani, E. Hahn, E. Wood, E. Brinkman, E. Ar- caute, E. Dunbar, E. Smothers, F. Sun, F. Kreuk, F. Tian, F. Ozgenel, F. Caggioni, F. Guzmán, F. Kanayet, F. Seide, G. M. Florez, G. Schwarz, G. Badeer, G. Swee, G. Halpern, G. Thattai, G. Herman, G. Sizov, Guangyi. Zhang, G. L H. Shojanazeri. H. Zou, H. Wang. H. Zha, H. Habeeb, H. Rudolph, H. Suk, H. Aspegren, H. Goldman, I. Molybog, I. Tufanov, I.-E. Veliche, I. Gat, J. Weissman, J. Geboski, J. Kohli, J. Asher, J.-B. Gaya, J. Marcus, J. Tang, J. Chan, J. Zhen, J. Reizenstein, J. Teboul, J. Zhong, J. Jin, J. Yang. J. Cummings, J. Carvill, J. Shepard. J. McPhie, J. Torres, J. Ginsburg. J. Wang. K. Wu, K. H. U. K. Saxena, K. Prasad, K. Khandelwal. K. Zand, K. Matosich, K. Veeraraghavan, K. Michelena, K. Li, K. Huang, K. Chawla, K. Lakhotia, K. Huang, I.. Chen, I. Garg, I. A, L. Silva, L. Bell, I. Zhang, I. Guo,
L. Yu, L. Moshkovich, L. Wehrstedt, M. Khabsa, M. Avalani, M. Bhatt, M. Tsimpoukelli, M. Mankus, M. Hasson, M. Lennie, M. Reso, M. Groshev, M. Naumov, M. Lathi, M. Ke- neally, M. L. Seltzer, M. Valko, M. Restrepo, M. Patel, M. Vyatskov, M. Samvelyan, M. Clark, M. Macey, M. Wang, M. J. Hermoso, M. Metanat, M. Rastegari, M. Bansal, N. Santhanam, N. Parks, N. White, N. Bawa, N. Singhal, N. Egebo, N. Usunier, N. P. Laptev, N. Dong, N. Zhang. N. Cheng, O. Chernoguz, O. Hart, O. Salpekar, O. Kalinli, P. Kent, P. Parekh, P. Saab, Balaji. P. Rittner, Bontrager, P. Roux, Dollar, Zvyagina. Ratanchandami, P. Yuvraj. Q. Liang, R. Alao, R. Rodriguez, R. Ayub, R. Murthy, R. Nayani, R. Mitra, R. Li, R. Hogan, R. Battey, R. Wang. R. Maheswari. R. Howes, R. Rinott, S. J. Bondu, S. Datta, S. Chugh, S. Hunt, S. Dhillon, S. Sidorov, S. Pan, S. Verma, S. Yamamoto, S. Ramaswamy, S. Lindsay, Lindsay, S. Feng, S. Lin, C. Zha, S. Shankar, S. Zhang, Zhang, S. Wang, Agarwal, S. Sajuyighe, S. Chintala, S. Max, S. Chen, S. Kehoe, S. Satterfield, S. Govindaprasad, S. Gupta, S. Cho, S. Virk, S. Subramanian, S. Choudhury, S. Goldman,
Remez, T. Glaser, T. Best, T. Kohler, T. Robinson, T. Li, T. Zhang, T. Matthews, T. Chou, T. Shaked, V. Vontimitta, V. Ajayi, V. Montanez, V. Mohan, V. S. Kumar, V. Mangla, V. Ionescu, V. Poenaru,
T. Mihailescu, V. Ivanov, W. Li, W. Wang, W. Jiang, W. Bouaziz, W. Constable, X. Tang. X. Wang, X. Wu,
X. Wang, X. Xia, X. Wu, X. Gao, Y. Chen, Y. Hu, Y. Jia, Y. Qi, Y. Li, Y. Zhang, Y. Zhang, Y. Adi, Y. Nam, Yu, Wang, Y. Hao, Y. Qian, Y. He, Z. Rait, Z. Vito, Z. Rosnbrick, Z. Wen, Z. Yang, and Z. Zhao. The llaaa 3 herd of models. arXiv preprint arXiv: 2407.21783, 2024
[8] Y. Dubois, X. Li, R. Taori, T. Zhang, I. Gulrajani, J. Ba, C. Guestrin, P. Liang, and Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. In Thirty-sevnnth Conference on Neural Information Processing Systems, 2023. URL httf//forriidddhhutea
[9] Y. Dubois, B. Galambosi, P. Liang, and T. B. Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.0444 04475, 2024.
[10] E. Frick, T. Li, C. Chen, W.-L. Chiang, A. N. Angelopoulos, J. Jiao, B. Zhu, J. E. Gonzalez, and I. Stoica. How to evaluate reward models for RLHF. In The Thirteenth International Conference on Learning Representations, 2025. URL https:: //otepev//penrevieer. id=cbttLt094Q
wwwpeeerreieeeeeeefforroeeeeeeee net net/forum?
[11] M. Gao, Y. Liu, X. Hu, X. Wan, J. Bragg, and A. Cohan. Re-evaluating automatic LLM system ranking for alignment with human preference. In I.. Chiruzzo, A. Ritter, and L. Wang, editors, Findings of the Association for Computational Linguistics: NAACI. 2025, pages 4605-4629, Albuquerque. New Mexico, Apr. 2025. Association for Computational Linguistics. ISBN 16-119977 URL
org.ggg...rrrrg/ org 2025/ findings cwwtiiinnnnnaacl..mmmm
[12] T. Gemini, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu. R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
[13] A. Gera, O. Boni, Y. Perlitz, R. Bar-Haim, L. Eden, and A. Yehudai. Justrank: Benchmarking I1m judges for system ranking. arXiv preprint arXiv:2412.09566 2024.
[14] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The lama herd of models. arXiv preprint arxiv:2407.21783, 2024.
12

---

[15] Y. He. D. Jin, C. Wang, C. Bi, K. Mandyam, H. Zhang, C. Zhu, N. Li, T. Xu, H. Lv, et al. Multi-if: Benchmarking Ilms on multi-turn and multilingual instructions following, arXiv preprint ArXiv:2410. 15553, 2024.
[16] H. Ivison, Y. Wang, V. Pyatkin, N. Lambert, M. Peters, P. Dasigi, J. Jang, D. Wadden, N. A. Smith, I. Beltagy, and H. Hajishirzi. Camels in a changing climate: Enhancing Im adaptation with tulu 2, 2023.
[17] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. I. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al. Mistral 7b. arXiv preprint 13000223 2023.
[18] S. Kim, J. Suk, J. Y. Cho, S. Longpre, C. Kim, D. Yoon, G. Son, Y. Cho, S. Shafayat, J. Baek, S. H. Park, H. Hwang, J. Jo. H. Cho, H. Shin, S. Lee, H. Oh, N. Lee, N. Ho, J. Joo, M. Ko, Y. Lee, H. Chae, J. Shin, J. Jang, S. Yc, B. Y. Lin, S. Welleck, G. Neubig, M. Lee, K. Lee, and M. Seo. The BiGGen bench: A principled benchmark for fine-grained evaluation of language models with language models. In L. Chiruzzo, A. Ritter, and L. Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume I: Long Papers), pages 5877-5919, Albuquerque, New Mexico, Apr. 2025. Association for Computational Linguistics. ISBN 979-8-8911 19996-189-66 URL httppttts///caaattooooo
wrg/2025..cc... 2025 naacl- 303/
[19] N. Lambert, V. Pyatkin, J. Morrison, L. Miranda, B. Y. Lin, K. Chandu, N. Dziri, S. Kumar, T. Zick, Y. Choi, N. A. Smith, and H. Hajishirzi. RewardBench: Evaluating reward models for language modeling, arXiv preprint arXiv: 2403.1378) 2024.
[20] D. Li, B. Jiang, L. Huang, A. Beigi, C. Zhao, Z. Tan, A. Bhattacharjee, Y. Jiang, C. Chen, T. Wu, al. From generation to judgment: Opportunities and challenges of auddge. arXiv preprint N..v:24/11.65544/ 2024.
[21] Li, W.-L. Chiang, E. Frick, L. Dunlap, T. Wu, B. Zhu, J. E. Gonzalez, and I. Stoica. From crowdsourced data high-quality benchmarks: Arena-hard and benchbuilder pipeline. CoRR, abs/2406.1 1939, 2024. URL https::///// http::::/oooooo/////// 2.....850//ara.
2406. 11939
[22] X. Li, T. Zhang, Y. Dubois, R. Taori, I. Gulrajani, C. Guestrin, P. Liang, and T. B. Hashimoto. Alpacaeval: An automatic evaluator of mnstruction-following models. /tttps://githut com/
latiu-labbapppacoo eval 2023.
[23] X. L.. Li, V. Shrivastava, S. Li, T. Hashimoto, and P. Liang. Benchmarking and improving vanerator-validator consistency of language models. In The Twelfth International Conference Learning Representations, 2024. URL /tttts://ppenreee
het:fforum?idpphBBBYy wi--idd6ypppppaaa
[24] B. Lin, Y. Deng, K. Chandu, A. Ravichander, V. Pyatkin, N. Dziri, R. L. Bras, and Y. Choi Wildbench: Benchmarking LLMs with challenging tasks from real users in the wild. In The Thirleenth International Conference on Learning Representations, 2025. URL httpss /openreview. net htte:/oormiidd-MEHHCO
[25] Y. Liu, K. Shi, A. Fabbri, Y. Zhao, P. Wang, C.-S. Wu, S. Joty, and A. Cohan. RelFE: Re- evaluating instruction-follovin evaluation. In L. Chiruzzo, A. Ritter, and L. Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume Long Papers), pages 12247-12287, Albuquerque, New Mexico, Apr. 2025. Association for Computational Linguis- tics. ISBN 979-8-89176- 189-6. URL https: https::t/ccllateee http::///aclaantoooggooo//////// naccl naacl-long 610/
[26] D. Mahan, D. Van Phung, R. Rafailov, C. Blagden, N. Lile, L. Castricato, J.PP. Fränken, C. Finn, and A. Albalak. Generative reward models. arXiv preprint arXiv:2410.1283: 2024.
[27]
Meta AI. Introducing meta Ilama 3: The most capable openly available Ilm to date. https //ai met...eeaacmmmm com blonm//og//aao hwt.mmmtaala alamamma33 3/| 2024. Accessed: 2024-05-30.
[28] Ni, F. Xue, X. Yue, Y. Deng, M. Shah, K. Jain, G. Neubig, and Y. You. Mixeval: Deriving wisdom of the crowd from LLM benchmark mixtures. In The Thirly-eighth Annual Conference on Neural Information Processing Systems, 2024. URL htttss:tts:/// h/tttp:e///ppeeeeiewe wtwpeeewwwwweeeeeeeeeeeeee f i 1d=6A29LUZhfv
13

---

[29] L. Ouyang. J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin. C. Zhang, S. Agarwal, K. Slama, A. Ray, Schulman, Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 27730-27744 Curran Associates, Inc., 2022.
[30] J. D. Rodriguez, W. Ding, K. Erk, and G. Durrett. Rankalign: A ranking view of the generator- validator gap in large language models. arXiv preprint arXiv:2504. / 1381, 2025.
[31] S. Singh, Y. Nan, A. Wang, D. D' D'Suuza, S. Kapoor, A. Üstün, S. Koyejo, Y. Deng. S. Longpre, N. Smith, et al. The leaderboard illusion. arXiv preprint arXiv:2504.2087) 2025.
[32] Y. Song, H. Zhang, C. Fisenach, S. M. Kakade, D. Foster, and U. Ghai. Mind the gap: Examining the self-improvement capabilities of large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL /ttptpse///ppereveeee.... aeapeerreieev..etffrrmm net/forum? 1d=mt.JMMc33e
[33] G. Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju. L. Hussenot, T. Mesnard, B. Shahriari, A. Ramé, et al. Gemma 2: Improving open language models at practical size. arXiv preprint 2rXiv:2408.00// 18, 2024.
[34] L. Tunstall, E. Beeching, N. Lambert, N. Rajani, K. Rasul, Y. Belkada, S. Huang, L. von Werra, C. Fourrier, N. Habib, et al. Zephyr: Direct distillation of LM alignment. arXiv preprint
arXiv:2310.1699 2023.
[35] L.. Tunstall, E. F. Beeching. N. Lambert, N. Rajani, Rasul, Y. Belkada, S. Huang, L. Werra, C. Fourrier, N. Habib, N. Sarrazin, O. Sanseviero, A. M. Rush, and T. Wolf. Zephyr: Direct distillation of LM alignment. In First Conference on Language Modeling, 2024. URL https://openreviee: www.:eww.eeefrfuue dim=miiidaaaBaaaoo
[36] Wang, L.. Li, Chen, Z. Cai, D. Zhu, B. Lin, Y. Cao, L. Kong, Q. Liu, T. Liu, and Z. Sui. Large language models are not fair evaluators. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9440-9449999 9450, Bangkok, Thailand, Aug. 2024. Association for Computational Linguistics. URL htttts::/allanthollggg www.aatthoooooooog/222222222 acl- 1ong..11111
[37] Z. Wang, Y. Dong, J. Zeng, V. Adams, M. N. Sreedhar, D. Egert, O. Delalleau, J. P. Scowcroft, N. Kant, A. Swope, et al. Helpsteer: Multi-attribute helpfulness dataset for steerlm. arXiv 2023.
preprint arXiv:2311.0953
[38] Z. Wang, Y. Dong, O. Delalleau, Zeng, G. Shen, D. Egert, J. J. Zhang, M. N. Sreedhar, and O. Kuchaiev. HelpSteer2: Open-source dataset for training top-performing reward models, 2024.
[39] Z. Wang. J. Zeng, O. Delalleau, D. Egert, E. Evans, H.-C. Shin, F. Soares, Y. Dong, and O. Kuchaiev. Dedicated feedback and edit models empower inference-time scaling for open- ended general-domain tasks. arXiv preprint arXiv:2503.044 04378, 2025.
[40] B. Wen, P. Ke, X. Gu, L. Wu, H. Huang. J. Zhou, W. Li, B. Hu, W. Gao, J. Xu, Y. Liu, J. Tang, H. Wang, and M. Huang. Benchmarking complex cnrruction-followiin with multiple constraints composition. In The Thirty-eight Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL
1d=U22VNDr2GG
[41] P. West, X. Lu, N. Dziri, F. Brahman, L. Li, J. D. Hwang, L. Jiang. J. Fisher, A. Ravichander, K. Chandu, B. Newman, P. W. Koh, A. Ettinger, and Y. Choi. The generative Al paradox: "what it can create, it may not understand". In The Tweifth International Conference on Learning Representations, 2024. URL htttps://opperreeeeee www.rrvviwwneeeffrreeeee 2d=uumid=-FFHMMMSPPP
[42] W. Yuan, R. Y. Pang, K. Cho, X. Li, S. Sukhbaatar, J. Xu, and J. E. Weston. Self-rewarding language models. In Forty-first International Conference on Machine Learning, 2024.
14

---

[43] Z. Zeng, J. Yu, T. Gao, Y. Meng. T. Goyal, and D. Chen. Evaluating large language models evaluating instruction following. In The Twelfth International Conference on Learning Representations, 2024.
[44] L. Zhang, A. Hosseini, H. Bansal, M. Kazemi, A. Kumar, and R. Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arXiv:2408. 2024.
[45] L. Zheng, W.-L. Chiang, Y. Sheng. S. Zhuang. Z. Wu, Zhuang. Z. Lin, Z. Li, D. Li, E. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processsing Systems, volume 36, pages 46595-46623, Curran Associates, Inc., 2023. URL U http::tttpptcteteetiiiiiee http://prroodddinde..e..r......e mwwr cttpspcccppppppeee paper_aaprr__a files/ wttpff//pppeeee////////// faoperrr22fffllee.. 91f18a12876332221 o 196 Paper-----aar--aaattttateeae and Benchmarkss pdf
[46] L. Zheng, W.-L. Chiang, Y. Sheng. S. Zhuang. Z. Wu, Zhuang. Z. Lin, Z. Li, D. Li, Xing. et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
[47] E. Zhou, G. Zheng, B. Wang, Z. Xi, Dou, R. Bao, W. Shen, Xiong. J. Fan, Y. Mou, R. Zheng, T. Gui, Q. Zhang, and X. Huang. RMB: Comprehensively benchmarking reward models in LLM alignment. In The Thirteenth International Conference on Learning Representations, 2025. URL httpt:t///opeeeeiiee nwt/ffruum/e ktggr119rrmmg
[48] J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou. Instruction- following evaluation for large language models. arXiv preprint arXiv:2311.0991 2023.
15

---

A Detailed Information about Benchmarked LLMs
Name
Size
License
Reference
1lma---8h
8h
llamaš Community
llam--370m
70b
llama3 Community
Meta Al
27 Dubey
llama 3.10-.3.1181 1-8b
8h
Ilama3.1 Community
laama 3.100b
70b
Ilama3.1 1 Community
lama-3.3-70b
70b
llama3.3 Community
gemma-2-4 2-2b
2b
Gemma
gemma- 2-9b
Gemma
Team
gemma 2-27b
27b
Gemma
tulu-2 2-bo--
A12 ImpACT Low-risk
Ivison
gwen 1. 1.5-32b
Qianwen
(wen 1.5-72b
72b
Qianwen
Bai al.
qwen-2-72b
72b
Qianwen
qwen-2.5-72b
72h
Qianwen
gpt-3.5-turbo-0125
Proprietary
Achiam al.
301-4o-mmm-2224-07-18
Proprietary
Table 5: 15 LLMs evaluated in $32.11
Name
Size
License
Reference
llama-3. 405b
405b
llama3.3 Community
Dubey et al.
2024-05--3
Proprietary
Achiam
claude-3 3-haiuu
Proprietary
Claude 6
claude-3.3 -onnett
Proprietary
gemini 1eminni-2.0-flaa -las-tes
Proprietary
gemmmn-2.00-Hashh
Proprictary
Gemini et al.
gemmn-1.33 IlSsh
Proprietary
mistral-small
24h
Apache 2.0
Jiang et al.
Table 6: 8 additional LLMs evaluated in $4.2
Table 5 contains the 15 LLMs evaluated in 3.2.1 Table contains the additional LLMs evaluated in 4.2
B Prompt Template for Evaluating LLMs as Judges
The prompt template for evaluating LLMs as judges is shown in Figure
C GE-Consistency with GPT-4o as the Preference Oracle on WildBench
In $32.2, we examined the GE-consistency on AlpacaEval and Arena-Hard across 15 LLMs using GPT-4o as the preference oracle. The results show that the observed GE-consistency is higher Arena-Hard than on AlpacaEval, specifically, 0.971 Spearman's 's rank correlation versus 0.839
correlation. Here, we conduct an examination under the same setting WildBench 24].
Figure demonstrates 0.939 Spearman's correlation on WildBench regarding the Go-consist, Compared to Arena-Hard and AlpacaEval, WildBench offers more balanced distribution of in- struction types, for example, 14% involve creative writing, 12% involve reasoning, and 17% involve information seeking. On this more diverse instruction set, the observed correlation is still relatively strong, indicating that high GE-consistency is general pattern that holds across various types of instructions, including open-domain tasks.
16

---

Base
[System Messagel
You are helpful assistant in evaluating the quality of the outputs for given instruction. Your goal select the best output for the given instruction.
[User Message]
Select the Output Output (b) that hetter for the given instruction. The outputs are generated by two different chatbots respectively.
Here are some rules of the evaluation:
(1) You should prioritize evaluating whether the output honestlyprrecisctylllossy executes the instruction, then consider its helpfulness, accuracy, level of detail, harmlessness, etc. (2) Outputs should NOT contain more/less than what the instruction asks for, as such outputs do NOT precisely execute the instruction.
(3) You should avoid any potential bias and your judgment should be objective possible. For example, the order in which the outputs were presented should NOT affect your judgment, Output (a) and Output (b) are equally likely to the better.
Do NOT provide any explanation for your choice. Do NOT say both / neither good. You should answer using ONLY "Output (a)" "Output (b)". Do NOT output any other words.
# Instruction: [INSTRUCTION]
# Output (a): {OUTPUT_!}
Output (b): [OUTPUT_2]
# Which is better, Output (a) or Output (b)? Your response should be either "Output (a)" or "Output (b)":
Figure 4: Prompt template for evaluating LLMs as Judges.
D Detailed settings for Evaluating GE-Consisteney with Various LLMs as Preference Oracle
Here, outline the detailed settings for using various LLMMs as preference oracles in measuring GE-consistency ($3.3) When calculating the GE-consistency (Eq. T with a certain LLM as the preference oracle J. remove it from the LLM to evaluated (MM. since it will achieve perfect evaluation performance with itself as the evaluation oracle. This process might lead to a discrepancy in evaluation settings since the resulting LLM set M {M} is different for each LLM M. However, we find that the GE-consistency measured with GPT-4o as the preference oracle remains stable when any specific LLM is removed from the LLM M. which is detailed in Appendix [D] Therefore, the impact of this process on the GE-consistency should be moderate, allowing a meaningful comparison of different LLMs as a preference oracle. Figure we show that GE-consistency measured with GPT-4o in $3.2.2 is quite stable when any of the LLMs is removed from the L.LM set to be evaluated. This result ensures the reliability of the analysis the LLM set.
where one L.LM removed from
17

---

Geueration va Evalustion Perforunace on WildBeuch ( 0.938)
opeen:
0.7
Tama--33-706
gHoma-2-277:
Evalsation Pedformanee
Liiin 3.1-706
0.5-
prmman-2-00 2:9h
qwes-115-72b
0.4
tuln-22lpo-700
0.3
Arann 3.1
0.2
@ctess 2:2b
Mams-344 3-81
0.2
0.3
0.1
0.5
0.6
0.7
Generation Performance
Figure 5: Generation and evaluation performance of various LLMs with gpt-40-2024-08-22 the preference oracle. The X-axis shows the generation performance in terms LLMs'' win rates against the baseline system (GPT-4) evaluated by the preference oracle. The Y-axis shows the evaluation performance in terms of LLMs' agreement rate (Cohen's Kappa) with the preference oracle on filtered evaluation task instances.
Table 7: Comparison of different automatic evaluation benchmarks regarding their Spearman's 's correlation with non-style-controlled ChatBot Arena rankings, with and without rank averaging with IFEval-Loose. Sena--ard-SCC is the style-controlled score, AlpacaEval-Lo the conthlcon score.
Benchmark
IFEval
IFEval
[FEval(-Loose)
0.893
0.893
IFEval-Strict
0.907
0.895
Arena-Hard
0.952
0.936
Arena-Hard-SC
0.914
0.857
AlpacaEval
0.916
0,839
AlpacaEval-LC
0,950
0.815
GPT4o-Judge
0.958
0.911
MixEval
0.829
0.727
HelpSteer3
0.871
0.781
ALIGNEVAL-GPT
0.922
0.847
ALLLAEVAL-CLLAUDE
0.916
0.848
E LLM Benchmark Correlations with Non-style-controlled ChatBot Arena Rankings
In 4.33 the effectiveness of various LLMs is evaluated against the style-controlled LLM rankings from ChatBot Arena. Table 7 insttedd shows their correlations with the non-style-conton rankings. which indicates a similar trend.
18

---

GE- -B--Cosllteneey
0.7
0.4
AlpacaEval Arena Arena-Hard Hard
0.1
00
Haroe Jelb
Banc 1166
ype0-2.5-7th 72h
gie Re-toainn--aee ge--beainl-354.m 38------6---1 18
Dana 3-76
Hana 1780)
yoen 15-20
quen L.-22
Epl 796
grimm 2:26
Hamms
gramna
Preference Oracle
Figure 6: The stability of the GE-consistency with GPT-4o as the preference oracle when an arbitrary LLM is removed from the set of LLMS to be evaluated. Spearman's 's correlation between LLMS' generation and evaluation capabilities rankings is reported on the Y-axis. The X-axis shows the LLM that is removed from the LLM set to be evaluated.
19

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA]. .
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
general, answering
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: this study, we aim to analyze the correlation between LLMs' generation and evaluation capabilities. This is clearly stated in the introduction and analyzed throughout the submission.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
20

---

Justification:
In
we discussed the limitations of our proposed benchmarking approach.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof'
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[NA]
Justification:
We did not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The experiments in $ and $4 use open-source datasets. And we provide clear descriptions regarding the experimental settings.
Guidelines:
21

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
[Yes]
We will include the constructed dataset and the codebase in the supplemental material.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
gublic/guides/Codeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //niss.ccppulic//guus http://////uiied///deuuubiiiiiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
22

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
These details are included in 3 and
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: In Appendix D we provided an analysis regarding the stability of the measured GB-consistency.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The evaluation cost of our proposed benchmark, as well as the others, is compared in $4
Guidelines:
23

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htttps://neurre htt.//errppp.eeeppibeee/ciiaiaaa iacc
Answer: [Yes]
Justification: Yes, to the best of our knowledge, the research presented in this paper fully conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
There is no significant and immediate societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications,
authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

---

Answer:
[NA]
Justification:
We believe our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have provided citations to the used assets, including all the LLMs bench- marked in Appendix(A
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the bataasets.ppprsstttt paperswi theode. com/datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: we have provided clear description of our constructed benchmark in $4
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
25

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: We only used LLMs to assist writing.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our L.MM policy /bttps://neura: ce /c/////aneeeeccc//22/////11 for what should or should be described.
26


Reviews:
[{"review_id": "O9ildQsus0", "reviewer": "Reviewer_VP2b", "summary": "This paper investigates the relationship between large language models’ (LLMs) abilities to generate outputs aligned with human preferences and their abilities to evaluate such outputs. The authors formally define Generation–Evaluation Consistency (GE-consistency) as the Spearman correlation between LLM rankings by generation performance and by evaluation performance under a common preference oracle. They first measure GE-consistency using GPT‑4o on two benchmarks (AlpacaEval, Arena‑Hard), finding high correlations (ρ=0.839 and ρ=0.971) when filtering out inconsistent oracle judgments. They then study how GE-consistency varies with different oracles and demonstrate that stronger oracles yield higher consistency. Building on these findings, the paper proposes ALIGNEVAL, a benchmark that assesses LLM alignment by evaluating models in the judge role—using a fixed set of pairwise instances labeled by GPT‑4o or Claude-3.7-Sonnet—without requiring judges at test time. ALIGNEVAL matches or outperforms existing automatic benchmarks (e.g., AlpacaEval, Arena‑Hard) in correlating with ChatBot Arena human rankings (up to ρ=0.946), and when combined with IFEval, achieves ρ=0.94. The work highlights the importance of LLMs’ evaluation capabilities and introduces a scalable proxy for alignment assessment.", "questions": "1. Oracle Robustness: How does ALIGNEVAL perform if the preference labels are aggregated from multiple weaker oracles instead of a single strong one? Could ensembling or majority‑vote reduce bias and improve generality?\n\n2. Broader Alignment Aspects: Can the GE‑consistency framework be extended to other evaluation tasks, such as toxicity or factuality judgments? What challenges arise when the evaluation criterion is less subjective or more specialized?\n\n3. Theoretical Underpinning: Can the authors provide insight or hypotheses on why stronger evaluators tend to be stronger generators? For instance, is there evidence that underlying model capabilities (e.g., reasoning) drive both tasks?", "limitations": "The authors acknowledge ALIGNEVAL is a proxy and vulnerable to adversarial fine‑tuning; combining with IFEval partially mitigates this. However, further discussion on multi‑oracle ensembling and detection of judge‑overfitting would strengthen the treatment of limitations.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. Significance & Originality: Introduces the novel concept of GE‑consistency, connecting two facets of LLM behavior—generation and evaluation—and demonstrates its high correlation under strong oracles (Section 3) \n\n2. Benchmark Innovation: Proposes ALIGNEVAL, which repurposes judge instances to evaluate alignment without on-the-fly judging, reducing computational cost and preserving high correlation with human preferences (Section 4)\n\n3. Comprehensive Empirical Study: Evaluates 15–23 diverse LLMs across multiple oracles and instruction sets, with clear ablations on filtering and oracle strength (Figures 2, Table 1)\n\n4. Practical Impact: Demonstrates that ALIGNEVAL combined with IFEval matches or exceeds performance of established benchmarks, offering a cost‑effective alternative for both researchers and practitioners.\n\nWeaknesses\n1. Dependence on Oracle Quality: High GE‑consistency relies on very strong oracles (e.g., GPT‑4o); weaker oracles yield substantially lower consistency, limiting generalizability without access to frontier models (Figure 3) \n\n2. Oracle Self‑Bias: ALIGNEVAL exhibits self‑preference bias (models rank their own version highly), which may inflate correlations unless multiple oracles are used (Section 4.3)\n\n3. Limited Theoretical Insight: While empirical correlations are strong, the paper offers limited theoretical explanation for why GE‑consistency holds or how it evolves during training.", "was_revised": true, "final_justification": "The authors' response has cleared up my doubts. Since my score is already positive, I will not be changing it.", "rebuttal": "Thank you for your helpful comments. We are encouraged by your recognition of the impact and novelty of our work.\n\n---\n> “Dependence on Oracle Quality: High GE‑consistency relies on very strong oracles (e.g., GPT‑4o); weaker oracles yield substantially lower consistency, limiting generalizability without access to frontier models (Figure 3)”\n\nThank you for the comment. Our investigation indeed shows that achieving high GE-consistency requires strong preference oracles. However, **due to the design of our proposed benchmark, AlignEval, this reliance on frontier models does not pose a bottleneck for generalizability.** Specifically, AlignEval leverages existing preference annotations from strong oracles to evaluate LLMs as evaluators, so **constant access to frontier models is not required once the benchmark is constructed.** This also enables the reuse of expensive human annotations using the same methodology. We will expand on this point in the revised manuscript.\n\n---\n> “Oracle Self‑Bias: ALIGNEVAL exhibits self‑preference bias (models rank their own version highly), which may inflate correlations unless multiple oracles are used (Section 4.3)”\n\nWe note that this self-bias does not necessarily inflate the correlations. In cases where earlier versions of a model are rated poorly by human evaluators, the model’s self-preference may actually lower the correlation. Thus, rather than introducing a systematic bias, the self-bias is more likely to increase the variance of the correlations. Furthermore, as discussed in Lines 318-322, AlignEval is still able to rank gemini-2.0-flash, the top system according to human evaluation, among the top two, despite the presence of potential self-bias.\n\n---\n> “Limited Theoretical Insight: While empirical correlations are strong, the paper offers limited theoretical explanation for why GE‑consistency holds or how it evolves during training.” “Theoretical Underpinning: Can the authors provide insight or hypotheses on why stronger evaluators tend to be stronger generators? For instance, is there evidence that underlying model capabilities (e.g., reasoning) drive both tasks?”\n\nThese are indeed interesting questions. This work is motivated by the hypothesis that an LLM capable of generating outputs well aligned with human preferences should also be better at evaluating whether an output aligns with those preferences. Our investigation confirms that such a property, generation-evaluation (GE) consistency, emerges when using a strong preference oracle. To the best of our knowledge, this is the first work to formally introduce and examine GE-consistency and its potential implications. We hope future work will build on this direction, for instance, by exploring the insightful questions you have raised regarding its theoretical foundations and training dynamics.\n\n---\n> “Broader Alignment Aspects: Can the GE‑consistency framework be extended to other evaluation tasks, such as toxicity or factuality judgments? What challenges arise when the evaluation criterion is less subjective or more specialized?”\n\nThank you for these thoughtful questions.\n1. We would like to note that the preference oracle used in our study already considers multiple alignment aspects, such as toxicity and factuality. Specifically, the evaluation prompt (Figure 4 in the Appendix, Page 16) instructs the model to “prioritize evaluating whether the output honestly/precisely/closely executes the instruction, then consider its helpfulness, accuracy, level of detail, harmlessness, etc.”\n2. We agree that more targeted studies focusing specifically on aspects like toxicity or factuality are valuable. However, incorporating such studies is challenging within the scope of the current work, due to both the breadth of our focus and the lack of comprehensive human evaluation benchmarks (e.g., similar to Chatbot Arena) dedicated to these specific dimensions. That said, we recognize the importance of these directions and will include a discussion of them in the revised manuscript.\n\n---\n> “Oracle Robustness: How does ALIGNEVAL perform if the preference labels are aggregated from multiple weaker oracles instead of a single strong one? Could ensembling or majority‑vote reduce bias and improve generality?”\n\nThank you for these thoughtful questions. To explore this, we conducted a pilot study where we aggregated preference labels from two weaker oracles, Qwen3-8B and Qwen3-14B, using the AlignEval+ framework (i.e., with IFEval). The results show that this ensemble approach improves correlation with human judgments: used individually, Qwen3-8B and Qwen3-14B achieve Spearman’s correlations of 0.853 and 0.865 with Chatbot Arena, respectively. When combined via majority vote, the ensemble reaches a higher correlation of 0.876. Moreover, we believe that ensembling can help reduce potential self-bias introduced by individual oracles, as discussed above. We will report this result and expand on it in the revised manuscript."}, {"review_id": "iZj2teIjlM", "reviewer": "Reviewer_FfNP", "summary": "Current alignment evaluation practices predominantly focus on open-ended response tasks, typically relying on human annotations or leveraging stronger LLMs as preference oracles. However, certain LLMs themselves possess strong evaluation capabilities. This paper systematically investigates the relationship between an LLM’s generation and evaluation abilities within alignment tasks, and observe a notable level of generation-evaluation consistency (GE-consistency). Building upon this finding, this paper proposes AlignEval, a novel alignment benchmark that enables indirect evaluation of LLMs’ alignment with human preferences without assessing their generated outputs directly. AlignEval eliminates the need for human annotations or LLM judges, providing a fully automated and cost-effective solution for alignment evaluation.", "questions": "1. In the GE-consistency validation phase (Section 3), incorporating a few representative examples with manual evaluation would enhance the interpretability and persuasiveness of the observed consistency between generation and evaluation capabilities. \n2. While the current GE-consistency findings are primarily based on rank correlation, which provides valuable global insights, it raises concerns about the reliability of the absolute scores reported in subsequent experiments. \n3. In Figure 2, different models exhibit notable performance discrepancies across the two evaluation sets. It is recommended that the authors further investigate the underlying causes of this phenomenon, such as potential correlations with model scale or task generalization ability. A deeper analysis in this regard would strengthen the paper’s explanation of evaluation stability.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. This paper addresses the consistency between the generation and evaluation capabilities of large language models (LLMs) by adopting a ranking-based consistency framework within alignment evaluation settings. \n2. This paper proposes AlignEval, a novel benchmark that enables alignment evaluation without relying on human annotations or LLM-based judges, offering a fully automated, cost-effective, and efficient solution for assessing LLM alignment with human preferences.\n\nWeaknesses:\n1. The analysis of the paper’s main conclusions lacks sufficient depth, leaving certain critical questions unaddressed. This may undermine the perceived novelty and overall impact of the paper.", "was_revised": true, "final_justification": "The author’s response did not alter my assessment of this work; therefore, my rating remains unchanged.", "rebuttal": "We appreciate your thoughtful review and questions. We hope the following responses help clarify your concerns regarding the “certain critical questions unaddressed.”\n\n---\n> “In the GE-consistency validation phase (Section 3), incorporating a few representative examples with manual evaluation would enhance the interpretability and persuasiveness of the observed consistency between generation and evaluation capabilities.”\n\nWe agree that including representative examples with manual evaluation could enhance the interpretability of the observed GE-consistency. However, GE-consistency is defined as a rank correlation across multiple LLMs and diverse instructions, and a meaningful human evaluation would require substantial effort and time. Due to these constraints, we are unable to conduct a systematic investigation during the rebuttal period. That said, we will note this limitation and plan to provide such case studies in the revised manuscript.\n\n---\n> “While the current GE-consistency findings are primarily based on rank correlation, which provides valuable global insights, it raises concerns about the reliability of the absolute scores reported in subsequent experiments.”\n\nThank you for your comment. Our approach follows prior work (e.g., [1][2]) that constructs automatic LLM benchmarks using correlation with human evaluations as the primary metric, as it directly reflects the benchmark’s utility in ranking different LLMs. We agree that assessing the reliability of absolute score assignments could strengthen the evaluation. However, doing so would require analyzing the calibration of assigned scores, which involves specific methodologies beyond the current scope of our work. We will note this limitation in the revised manuscript.\n\nThat said, as an initial step, we re-evaluated GE-consistency in Section 3.2.2 using Pearson’s correlation instead of Spearman’s, which considers the absolute values of the scores. The results still show a high level of GE-consistency: the Pearson correlation on AlpacaEval and Arena-Hard is 0.902 and 0.966, respectively, compared to the original Spearman correlations of 0.839 and 0.971.  **This suggests that GE-consistency still holds even when considering absolute scores.**\n\nReferences\n\n[1] Dubois, Yann, et al. \"Length-controlled alpacaeval: A simple way to debias automatic evaluators.\" COLM 2024\n\n[2] Li, Tianle, et al. \"From Crowdsourced Data to High-quality Benchmarks: Arena-Hard and Benchbuilder Pipeline.\" ICML 2025.\n\n---\n> “In Figure 2, different models exhibit notable performance discrepancies across the two evaluation sets. It is recommended that the authors further investigate the underlying causes of this phenomenon, such as potential correlations with model scale or task generalization ability. A deeper analysis in this regard would strengthen the paper’s explanation of evaluation stability.”\n\nThank you for this helpful suggestion. We believe the observed discrepancy is primarily due to differences in instruction difficulty: Arena-Hard is more challenging than AlpacaEval, as it contains more technical and complex instructions. This pattern is consistent with the official leaderboard performance of the two benchmarks. We will add a note regarding this in the revised manuscript."}, {"review_id": "WuhDYqDNLG", "reviewer": "Reviewer_5TaH", "summary": "This paper explores whether LLMs exhibit consistent performance in two tasks: \"generating aligned responses\" and \"evaluating whether responses are aligned.\" The authors propose GE-consistency, which uses Spearman correlation to measure the similarity of model performance across these two tasks. Evaluations on several mainstream LLMs lead to the conclusion that good generators are often also good evaluators. Based on this finding, the paper introduces the ALIGNEVAL evaluation method, which indirectly assesses a model’s generation ability by evaluating whether it can accurately judge human preferences, thereby effectively reducing evaluation costs. Finally, the method is compared with other evaluation approaches to validate its effectiveness.", "questions": "1. As shown in Figure 3, small models like Gemma-2B and LLaMA-3-8B do not perform well in terms of GE-consistency. Does this mean the method is only suitable for evaluating and ranking mid- to high-end models?\n2. Since stronger models may perform better on all tasks, can the high correlation between generation and evaluation abilities be attributed to simultaneous linear improvement, rather than an intrinsic dependency?\n3. You report that consistency on AlpacaEval is significantly lower than on Arena-Hard. Does this indicate that the ALIGNEVAL method is less suitable for open-domain tasks?", "limitations": "They discussed thelimitations of thier proposed benchmarking approach in section 5.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The writing is good. The authors provides a novel evaluation perspective: by assessing whether an LLM can accurately judge, it indirectly verifies its generation ability, offering a new research direction for future work.\n2. Based on the GE-consistency theory, the proposed ALIGNEVAL evaluation method can reduce computational and annotation costs.\n\nWeakness:\n1. In the GE-consistency experiments, only the Arena-Hard and AlpacaEval datasets were used. This seems insufficient, especially since the method performs well on the technical Arena-Hard dataset but poorly on the open-domain AlpacaEval dataset. More comprehensive experiments are needed to demonstrate generalizability.\n2. If the proposed consistency theory holds, then ALIGNEVAL can indeed reduce evaluation costs for subsequent work. However, as the authors mention, it may be vulnerable to adversarial adaptation, where high scores do not necessarily indicate true capability. If this issue cannot be resolved, it will still be necessary to rely on evaluating the model’s output to fully verify its generalization ability, which may not actually reduce evaluation costs.\n3. GE-consistency measures correlation, not causation. There is a potential confounding factor that \"strong models perform well on all tasks,\" meaning that high scores across the board may obscure essential differences between the two abilities (uncertain).", "was_revised": true, "final_justification": "The authors have largely addressed my concerns on other points. However, their explanation regarding the distinction between correlation and causation, while partially reasonable, does not fully convince me. I have adjusted my score accordingly.", "rebuttal": "Thank you for your helpful comments and insightful questions.\n\n---\n> “In the GE-consistency experiments, only the Arena-Hard and AlpacaEval datasets were used. This seems insufficient, especially since the method performs well on the technical Arena-Hard dataset but poorly on the open-domain AlpacaEval dataset. More comprehensive experiments are needed to demonstrate generalizability.” “You report that consistency on AlpacaEval is significantly lower than on Arena-Hard. Does this indicate that the ALIGNEVAL method is less suitable for open-domain tasks?”\n\nThank you for the comment.\n\n1. We would first like to clarify that GE-consistency is not a method but a property we aim to study. As defined in Eq. 1, GE-consistency depends on both the choice of preference oracle and instruction set. In Section 3, we analyze how GE-consistency varies under different configurations of these two factors. Specifically, we use AlpacaEval and Arena-Hard as instruction sets due to their wide adoption. Our results show that the choice of instruction set indeed has a substantial impact on the level of GE-consistency.\n\n2. Utilizing the above finding, we propose AlignEval, a benchmark whose goal is to align with human evaluations, using the LLM rankings from ChatBot Arena as the gold standard. AlignEval leverages the high GE-consistency observed on the Arena-Hard instruction set to enable efficient benchmarking. Since the ChatBot Arena rankings are aggregated from human votes on responses to user instructions, many of which are open-ended, the fact that AlignEval achieves a high level of correlation with ChatBot Arena suggests that it can effectively reflect human evaluations of alignment on open-ended tasks.\n\n3. Following your suggestion, **we conducted further experiments** using the same setup as in Section 3.2 on WildBench [1], with GPT-4o as the preference oracle to evaluate 15 LLMs. Compared to Arena-Hard and AlpacaEval, WildBench offers a more balanced distribution of instruction types, for example, 14% involve creative writing, 12% involve reasoning, and 17% involve information seeking. On this more diverse instruction set, we observe a GE-consistency of 0.938 Spearman’s correlation, **suggesting that GE-consistency is a general pattern that holds across various types of instructions including open-domain tasks**.\n\n4. We believe that the lower GE-consistency observed on AlpacaEval is partly due to the inherent noise in evaluating more open-ended and easier task instances. As evidence, Table 4 shows that the original AlpacaEval benchmark achieves only around 0.75 Spearman correlation with ChatBot Arena. This suggests that the AlpacaEval instruction set may have inherent limitations for robust LLM evaluation, and thus lower GE-consistency is expected.\n\n\n\nWe will update the revised manuscript with these results.\n\nReference\n\n[1] Lin, Bill Yuchen, et al. \"WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild.\" ICLR 2025\n\n\n---\n> “If the proposed consistency theory holds, then ALIGNEVAL can indeed reduce evaluation costs for subsequent work. However, as the authors mention, it may be vulnerable to adversarial adaptation, where high scores do not necessarily indicate true capability. If this issue cannot be resolved, it will still be necessary to rely on evaluating the model’s output to fully verify its generalization ability, which may not actually reduce evaluation costs.”\n\nThank you for the comment. We agree that this is an important point, and we would like to expand our discussion in our conclusion section as follows, which we will update in the revised manuscript:\n\n1. Since AlignEval is a proxy evaluation method, there is inevitably a trade-off between evaluation cost and accuracy/robustness. We believe the evaluation protocol we propose, combining AlignEval with IFEval, can help mitigate this risk, as it still requires the model to produce accurate outputs in response to instructions. In addition, targeted evaluations focusing on adversarial responses can be incorporated. These may be more cost-effective than the original LLM-judge-based evaluation while still addressing adversarial vulnerabilities.\n\n2. AlignEval can be a valuable tool for benign model developers who do not have incentives to adversarially tune their models to exploit the metric. In such cases, AlignEval can serve as a cost-effective method for assessing model alignment during rapid development cycles.\n\n3. To the best of our knowledge, this is the first work to formally introduce and study the notion of generation-evaluation consistency and its potential implications. Therefore, we consider the most important finding of this work to be that the evaluation capability of LLMs is itself an important evaluation aspect of LLMs’ alignment. We hope future work will build on this direction, further exploring topics such as the susceptibility of this evaluation paradigm to adversarial manipulation and the development of effective safeguards.\n\n---\n> “GE-consistency measures correlation, not causation. There is a potential confounding factor that \"strong models perform well on all tasks,\" meaning that high scores across the board may obscure essential differences between the two abilities (uncertain).” “Since stronger models may perform better on all tasks, can the high correlation between generation and evaluation abilities be attributed to simultaneous linear improvement, rather than an intrinsic dependency?”\n\nThank you for this interesting and insightful comment. We conducted additional analyses to investigate this potential confounding factor and summarize below why it is not the sole reason for the high generation-evaluation consistency (GE-consistency) observed in our work:\n\nIn Section 3.2.2, Figure 2, we report a Spearman’s correlation of 0.971 between LLMs’ generation and evaluation performance rankings (i.e., GE-consistency) on the instruction set of Arena-Hard using GPT-4o as the preference oracle. Under the same setting, the Spearman’s correlation between the generation performance ranking and the performance ranking on MixEval is 0.836, which is substantially lower than the observed GE-consistency. MixEval [1] evaluates alignment using a collection of tasks drawn from commonly used benchmarks such as MMLU and GSM8K, and therefore reflects a broader range of general capabilities. **The notably lower correlation suggests that the high GE-consistency we observe cannot be solely attributed to stronger models performing well across all tasks.**\n\nThank you again for the thoughtful comment. We will incorporate these discussion points into the revised manuscript.\n\nReference\n\n[1] Ni, Jinjie, et al. \"Mixeval: Deriving wisdom of the crowd from llm benchmark mixtures.\" Advances in Neural Information Processing Systems 37 (2024): 98180-98212.\n\n---\n> “As shown in Figure 3, small models like Gemma-2B and LLaMA-3-8B do not perform well in terms of GE-consistency. Does this mean the method is only suitable for evaluating and ranking mid- to high-end models?”\n\nThank you for the question. We’d like to clarify that the evaluation in Figure 3 investigates GE-consistency when using preference oracles of varying capabilities. The results show that GE-consistency is lower when a less capable LLM is used as the oracle. This is intuitive: less capable LLMs may lack consistency in their evaluations and produce unreliable evaluation results. This does not mean that the method is only suitable for evaluating mid- to high-end models. Rather, it highlights that a reliable evaluation requires a sufficiently strong preference oracle. As shown in Section 3.2.2, when a strong LLM is used as the oracle, it can meaningfully assess and differentiate models across a broad range of capabilities, including weaker models, which tend to receive lower scores in both generation and evaluation performance."}, {"review_id": "qfhTCQBc3n", "reviewer": "Reviewer_fNcL", "summary": "This paper investigates the Generation-Evaluation consistency (GE-consistency) of large language models (LLMs) and proposes a novel benchmarking paradigm, AlignEval, which evaluates LLM alignment by assessing their evaluation capabilities rather than directly judging their outputs. Experiments reveal a strong correlation between generation and evaluation rankings under specific conditions.", "questions": "- Will authors open source the proposed benchmark? This is very important.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strength\n\nThis paper claims to be the first to conduct comprehensive study on GE consistency in LLMs, proposing a benchmark that bypasses direct output evaluation, offering a fresh perspective on alignment assessment. Their proposed benchmark, AlignEval, reduces evaluation costs (as no LLM-as-Judge needed) while aligning well with human preferences, making it cost-effective for real-world deployment. The paper is well-written, and I find it easy to follow and understand.\n\n## Weakness\n\n- Lack of human evaluation. I understand that reproducing the consistency analysis experiments in Section 3.2 by human annotators would be challenging and almost impossible. However, I believe it authors should at least try to validate the consistency hypothesis on a few LLMs using small-scale instructions. Since the entire paper fundamentally relies on the GE-Consistency assumption, I think additional supporting evidence would strengthen this core premise (this would be more persuasive though the current experiments across 15 LLMs are already relatively solid).\n\n- Lack of more kinds of LLMs. The authors mainly study the System-1 model. Can the proposed benchmark be used for the System-2 model (Deepseek R1, O3, Qwen3 and so on)? This needs further verification.\n\n- The author mentioned the importance of Consistency Filtering. However, this situation does not only occur with preference oracle. Did the author check how sensitive the tested LLM is to this situation? And how stable is the LLM's judgment output after multiple tests?\n\n- A big concern lies in that the correlation on alpacaeval is significantly lower than arena-hard. Does this mean that the proposed benchmark cannot adequately validate the model's performance on open domain problems?\n\n- This is related to the concern #4. Alignment is a big topic. Most of the author's experiments focus on the alignment of the preference of \"helpfulness\". Is there any similar experimental support for other preferences (safety and honesty)? I believe this is a very interesting and important question, which can make the assumptions and contributions of the article more solid.", "was_revised": true, "final_justification": "Most of my concerns are addressed, and I believe my current rating reflects my assessment of this paper.", "rebuttal": "Thank you for your thoughtful review. We appreciate your recognition of the distinct perspective and contribution of our work.\n\n---\n> “Lack of human evaluation. I understand that reproducing the consistency analysis experiments in Section 3.2 by human annotators would be challenging and almost impossible. However, I believe it authors should at least try to validate the consistency hypothesis on a few LLMs using small-scale instructions. Since the entire paper fundamentally relies on the GE-Consistency assumption, I think additional supporting evidence would strengthen this core premise (this would be more persuasive though the current experiments across 15 LLMs are already relatively solid).”\n\nWe appreciate the suggestion. We agree that experiments with human annotations would be valuable, though they would require a significant amount of effort. We are grateful for your understanding of this difficulty. Even conducting such evaluations on a smaller set of instructions would be challenging, as we would still need to annotate outputs from multiple LLMs to compute rank correlations. Due to these constraints, we were unable to conduct such a study during the rebuttal period. However, we will aim to incorporate such analyses in the revised version, ideally using available human annotation datasets. If that turns out to be infeasible, we will include quantitative case studies with human evaluations to provide further intuition into GE-consistency.\n\n---\n> “Lack of more kinds of LLMs. The authors mainly study the System-1 model. Can the proposed benchmark be used for the System-2 model (Deepseek R1, O3, Qwen3 and so on)? This needs further verification.”\n\nThank you for the comment. We agree that extending evaluations to System-2/reasoning models is a valuable direction. However, several factors currently limit our ability to conduct a broader study on these models:  \n(1) There are still relatively few reasoning-focused models available on the ChatBot Arena, and some checkpoints (e.g., earlier versions of Gemini-2.5-thinking) have become obsolete or inaccessible.  \n(2) Reasoning models often require different evaluation configurations due to behavioral differences. For instance, generation temperature is either non-configurable or should not be set to zero.  \n(3) Querying these models via API incurs substantial cost, especially when multiple generations are needed for stable estimates.\n\nDespite these limitations, we conducted a small-scale experiment using three reasoning models, o3-mini, o4-mini, and r1, as additional evaluated models in AlignEval with IFEval (AlignEval+). **The resulting correlations with ChatBot Arena rankings remained stable, suggesting that AlignEval can generalize to reasoning models.** Using the updated ChatBot Arena rankings that include these models, we obtained:\n\n|                          | AlignEval-GPT | AlignEval-Claude |\n|--------------------------|----------------|------------------|\n| w/o models | 0.932          | 0.940            |\n| w/ reasoning models    | 0.940          | 0.934            |\n\nWe will expand on these results and include them in the revised manuscript.\n\n---\n> “The author mentioned the importance of Consistency Filtering. However, this situation does not only occur with preference oracle. Did the author check how sensitive the tested LLM is to this situation? And how stable is the LLM's judgment output after multiple tests?”\n\nThank you for the comment.\n\n1. The self-inconsistency issue indeed occurs in models beyond the preference oracle. Below are the self-consistency rates (i.e., the proportion of consistent outputs across pairwise comparisons in two orders) for the 15 LLMs evaluated in Section 3.3. The results suggest that more capable LLMs tend to be more self-consistent.\n\n| Model                   | Score  |\n|-------------------------|--------|\n| qwen-2-72b              | 0.730  |\n| qwen-2.5-72b            | 0.720  |\n| gpt-4o-mini-2024-07-18  | 0.709  |\n| gemma-2-27b             | 0.642  |\n| gemma-2-9b              | 0.612  |\n| llama-3.1-70b           | 0.573  |\n| llama-3.3-70b           | 0.541  |\n| llama-3-70b             | 0.419  |\n| qwen-1.5-72b            | 0.493  |\n| qwen-1.5-32b            | 0.343  |\n| gpt-3.5-turbo-0125      | 0.307  |\n| tulu-2-dpo-70b          | 0.259  |\n| llama-3.1-8b            | 0.242  |\n| gemma-2-2b              | 0.225  |\n| llama-3-8b              | 0.093  |\n\n2. We note that the self-inconsistency of the LLMs under evaluation does not compromise the reliability of AlignEval. Once examples are filtered based on the preference oracle’s self-consistency, any inconsistencies observed in the evaluated LLMs reflect their own limitations in producing reliable judgments.\n\n3. Regarding the stability of the LLMs' judgment outputs across multiple runs: we set the temperature to 0 when evaluating LLMs as judges. Due to the limited time during the rebuttal period, we were unable to run repeated generations to assess output stability, but we plan to include such results in the revised version.\n\nThank you for these helpful questions. We will update the manuscript accordingly.\n\n---\n> “A big concern lies in that the correlation on alpacaeval is significantly lower than arena-hard. Does this mean that the proposed benchmark cannot adequately validate the model's performance on open domain problems?”\n\n1. We would first like to clarify that GE-consistency is itself not a benchmark but a property we aim to study. As defined in Eq. 1, GE-consistency depends on both the choice of preference oracle and instruction set. In Section 3, we analyze how GE-consistency varies under different configurations of these two factors. Specifically, we use AlpacaEval and Arena-Hard as instruction sets due to their wide adoption. Our results show that the choice of instruction set indeed has a substantial impact on the level of GE-consistency.\n\n2. Utilizing the above finding, we propose AlignEval, a benchmark whose goal is to align with human evaluations, using the LLM rankings from ChatBot Arena as the gold standard. AlignEval leverages the high GE-consistency observed on the Arena-Hard instruction set to enable efficient benchmarking. Since the ChatBot Arena rankings are aggregated from human votes on responses to user instructions, many of which are open-ended, the fact that AlignEval achieves a high level of correlation with ChatBot Arena suggests that it can effectively reflect human evaluations of alignment on open-ended tasks.\n\n3.  **We have conducted further experiments** using the same setup as in Section 3.2 on WildBench [1], with GPT-4o as the preference oracle to evaluate 15 LLMs. Compared to Arena-Hard and AlpacaEval, WildBench offers a more balanced distribution of instruction types, for example, 14% involve creative writing, 12% involve reasoning, and 17% involve information seeking. On this more diverse instruction set, we observe a GE-consistency of 0.938 Spearman’s correlation, **suggesting that GE-consistency is a general pattern that holds across various types of instructions including open-domain tasks**.\n\n4. We believe that the lower GE-consistency observed on AlpacaEval is partly due to the inherent noise in evaluating more open-ended and easier task instances. As evidence, Table 4 shows that the original AlpacaEval benchmark achieves only around 0.75 Spearman correlation with ChatBot Arena. This suggests that the AlpacaEval instruction set may have inherent limitations for robust LLM evaluation, and thus lower GE-consistency is expected.\n\nThank you for the question. We will add a related discussion in the revised version to clarify this point.\n\nReference\n\n[1] Lin, Bill Yuchen, et al. \"WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild.\" ICLR 2025\n\n---\n> “... Most of the author's experiments focus on the alignment of the preference of \"helpfulness\". Is there any similar experimental support for other preferences (safety and honesty)? I believe this is a very interesting and important question, which can make the assumptions and contributions of the article more solid.”\n\nThank you for the suggestion! \n1. We would like to note that the preference oracle we used already takes into account various alignment aspects, such as safety and honesty. Specifically, the evaluation prompt (Figure 4 in the Appendix, Page 16) instructs: “You should prioritize evaluating whether the output honestly/precisely/closely executes the instruction, then consider its helpfulness, accuracy, level of detail, harmlessness, etc.”\n2. We agree that more focused studies on safety or honesty would be interesting and valuable. However, incorporating them into the current submission is challenging, as it falls outside the scope of our present study and is limited by the lack of comprehensive human evaluation benchmarks (e.g., similar to Chatbot Arena) for these specific aspects, which we rely on as external ground truth. That said, we acknowledge the importance of these directions and will add related discussion in the revised manuscript.\n\n---\n> “Will authors open source the proposed benchmark? This is very important.”\n\nYes, we will open-source the proposed benchmark upon publication. We have already included the benchmark and the codebase for using it to evaluate models in the supplementary material. Thank you for the question. We will make this point clearer in the revised manuscript."}]

Author rebuttal:

Meta-review: # Summary
This paper explores whether LLMs exhibit consistent performance in two tasks: "generating aligned responses" and "evaluating whether responses are aligned." The authors propose GE-consistency, which uses Spearman correlation to measure the similarity of model performance across these two tasks. Evaluations on several mainstream LLMs lead to the conclusion that good generators are often also good evaluators. Based on this finding, the paper introduces the ALIGNEVAL evaluation method, which indirectly assesses a model’s generation ability by evaluating whether it can accurately judge human preferences, thereby effectively reducing evaluation costs. Finally, the method is compared with other evaluation approaches to validate its effectiveness.

# Strengths
* Significance & Originality: Introduces the novel concept of GE‑consistency, connecting two facets of LLM behavior—generation and evaluation—and demonstrates its high correlation under strong oracles
* Benchmark Innovation: Proposes ALIGNEVAL, which repurposes judge instances to evaluate alignment without on-the-fly judging, reducing computational cost and preserving high correlation with human preferences
* Comprehensive Empirical Study: Evaluates 15–23 diverse LLMs across multiple oracles and instruction sets, with clear ablations on filtering and oracle strength

# Weaknesses
* Dependence on Oracle Quality: High GE‑consistency relies on very strong oracles (e.g., GPT‑4o); weaker oracles yield substantially lower consistency, limiting generalizability without access to frontier models
* Oracle Self‑Bias: ALIGNEVAL exhibits self‑preference bias (models rank their own version highly), which may inflate correlations unless multiple oracles are used
* Limited Theoretical Insight: While empirical correlations are strong, the paper offers limited theoretical explanation for why GE‑consistency holds or how it evolves during training.

# Concerns addressed during Author Response Period
* Correlation on alpacaeval is significantly lower than arena-hard. Does this mean that the proposed benchmark cannot adequately validate the model's performance on open domain problems?
* The authors run additional experiments which suggest that GE-consistency is a general pattern that holds across various types of instructions including open-domain tasks.
The preference oracle in the paper already takes into account various alignment aspects, such as safety and honesty.

# Overall
The paper proposes ALIGNEVAL, reducing the computational cost of evaluating the alignment capabilities of LLMs. The approach is novel and interesting. There are a few remaining concerns as the GE-consistency relies on a strong oracle quality and self-biases.
Decision: Accept (poster)