Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarking

Authors: Benjamin Feuer, Micah Goldblum, Teresa Datta, Sanjana Nambiar, Raz Besaleli, Samuel Dooley, Max Cembalest, John P Dickerson
Primary area: foundation or frontier models, including LLMs
Keywords: LLM, large language model, alignment, post-training, benchmarking, evaluation

Abstract:
The release of ChatGPT in November 2022 sparked an explosion of interest in post-training and an avalanche of new preference optimization (PO) methods. These methods claim superior alignment by virtue of better correspondence with human pairwise preferences, often measured by LLM-judges. In this work, we attempt to answer the following question -- do LLM-judge preferences translate to progress on other, more concrete metrics for alignment, and if not, why not? We define a concrete metric for alignment, and introduce SOS-Bench (Substance Outweighs Style Benchmark), the largest standardized, reproducible LLM meta-benchmark to date. We find that (1) LLM-judge preferences do not correlate with concrete measures of safety, world knowledge, and instruction following; (2) LLM-judges have powerful implicit biases, prioritizing style over factuality and safety; and (3) the supervised fine-tuning (SFT) stage of post-training has a large impact on alignment, with data scaling and prompt diversity as the driving factors.

Decision: accept (poster)
Reviewer scores: 8, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730430209135

thread_type:
official_review

<thread_object>
creator: reviewer_ngwc

<content>
summary:
This paper investigates the reliability of benchmarks that employ LLM-as-a-judge evaluation of model alignment. Specifically, it aims to investigate:
1. Correlation of LLM-judge preferences with more traditional (input, output) benchmarks, where the ground truth is known
2. Implicit biases of LLM-as-a-judge preferences when evaluating pluralistic objectives, that the authors explicitly disentangle into [completeness, correctness, safety, conciseness, style]

The paper also presents SOS-Bench, an amalgamation of existing traditional benchmarks, which is partitioned in a way to evaluate the HHH principles.

strengths:
The research question the authors tackle is timely and relevant, especially given the increase in recent works that evaluate alignment largely using LLM-as-a-judge. 

The work investigates questions that are in the community's consciousness and provides experimental evidence and analyses.

The paper reads well and is interesting. The presentation interleaves empirical analysis with intuition and justifications.

weaknesses:
One of the main focus of this work is empirical analysis of the reliability of LLM-as-a-judge benchmarks. As such, the main questions on my mind were whether the empirical findings were **robust** and **systematic**, and adequately controlled for potential confounding factors.

Below are my **major concerns**:
1. **LLM judges [S4]**: The authors evaluate results on one judge model (either gpt4 or gpt4o-mini) and one judge benchmark (Arena-Hard-Auto). I understand the effort and costs involved in evaluating multiple judges and benchmarks, but some consistency would strengthen the findings here. Nonetheless, I found this to be the most convincing set of results in the paper.
2. **Novelty of SOSBench [S5]**: Based on my understanding, SOS-Bench is an amalgamation of 19 existing benchmarks, binned together to evaluate three aspects of world knowledge, instruction following, and safety. I am not confident that this can be claimed as a novel contribution, especially since reporting evaluations on multiple benchmarks is common practice in alignment papers/ML research in general. If the author's contributions here are in collating and integrating them, I would consider it more as valuable engineering work rather than a research contribution.
3. **Implicit bias from LLM vs implicit bias from benchmark selection [S5]**: The authors could consider being more nuanced and mindful when discussing different approaches to evaluating alignment. Specifically, the principles of alignment are often pluralistic and ill-defined, e.g. 'safety' or 'helpfulness'. Distilling such principles into an explicitly specified metric or evaluating them indirectly (through traditional benchmarks) is also susceptible to its pathologies. For example, the authors mentioned `BLEU`, which has several well-documented biases. Additionally, the authors have introduced their own biases when using the performance of *world knowledge* as a surrogate for honesty, *instruction following* for helpfulness, and *safety* for harmlessness. While these might be reasonable surrogates, the point is, that the authors have injected their own implicit/explicit biases into the selection of benchmarks and thus evaluation metrics.
4. **Robustness of findings [S6]**: I have concerns about the robustness of the findings and the systematicity of experimental design. I will list my questions in detail below:
* `Data scaling improves alignment in the SFT stage of post-training.` The authors have controlled for differences in pretraining by starting from the same seed model. Is this paragraph (and Fig 2) only comparing models after SFT or after SFT + preference alignment? Analyzed by itself, the results appear to indicate that models are better aligned when SFT is performed with more data. This alone is not surprising. The authors also mention that SFT scale is more important than **curation** and **other important factors**. This relative impacts (of these other important factors) are not analyzed/reported, so it is unclear whether scale or a latent confounder has a higher impact on alignment post SFT.
* `Specialist post-training methods underperform generalist methods.` The authors mention 'diversity of prompts'---this would benefit from an explicit definition. Is it the narrow/broadness of the SFT dataset? Here, the comparison does not control for the size of the dataset, so it is not clear whether the observed differences in performance on specialist domains is due to 'scale' or 'prompt diversity'. Additionally, the authors have not investigated the quality of the datasets compared. I tried to have a look and couldn't find some of these datasets (e.g. Code LLAMA 3) online (the implication here is that they might be lower quality). In short, the results are not strongly convincing that 'diversity of prompts' is the primary factor that could explain the difference in downstream performance.
* `Preference optimization trades world knowledge for improved safety and instruction following.` This is a well-documented phenomenon colloquially referred to as the 'alignment tax'. Additionally, while the authors acknowledge that this result could be confounded by the choice of preference dataset, I feel that this as a crucial factor that must be controlled for any empirical insights to be made.
5. **SFT vs PO**: In the abstract, the authors claim `supervised finetuning stage, and not the PO stage, has the greatest impact on alignment'. Perhaps I missed it, but where this substantiated or investigated?

IMO, this work is ambitious (which I applaud), attempting to tackle many questions but unfortunatley compromising on thoroughness and depth. Specifically, the authors try to approach two tasks in one paper: (1) investigating pathologies of LLM-as-a-judge evaluations, and (2) proposing their own benchmark. I can see that the two are related. But my feeling is the impact would be significantly improved if the paper focused more in-depth on analyzing either of the two.

questions:
I also have some minor clarifying questions and suggestions:

1. The paper is missing many in-text references (e.g. for the datasets in Tables 4 and 5), making it slightly challenging for the reader to form context when analyzing the presented results.
2. How is `Loss` defined in Table 3?
3. In Table 5, is Arena score based on an 'overall judgement'. If so, it would be interesting to also compare against fine-grained evaluations of [completeness, conciseness, style, safety, correctness] to see if there are any correlations there with WK, IF, S.
4. The authors should reference pluralistic alignment (which concerns a similar effect as those analyzed) in S4.1 and Table 2.
5. The authors mention the Bradley-Terry model somewhat ad hoc. Did the authors intend to imply anything about the BT model through the presented results/analyses?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730478424211

thread_type:
official_review

<thread_object>
creator: reviewer_8qfx

<content>
summary:
This paper studies the drawbacks of the LLM-as-a-judge framework and proposes SOS-Bench, a ground-truth meta-benchmark, to circumvent those drawbacks.

strengths:
The paper proposes studying the drawbacks of the LLM-as-a-judge framework, a critical topic.
- The most interesting findings, in my view, are in Table 3;
- The approach to studying biases reported in Table 3 seems sound.

weaknesses:
The paper's drawbacks are greater than its contributions. In the following, I list what I think the be the biggest weaknesses of the paper and offer some suggestions on some of these points:

1) The paper does not provide convincing evidence of biases of the LLM-as-a-judge framework except from what is presented in Table 3. I believe the results in Table 2 are misinterpreted as they do not provide enough evidence of such biases (e.g., style over content preferences). I have two main reasons to support my claim:
     (i) The rank correlation of content and overall score in Table 2 seems to be very high. The fact that the correctness scores are "compressed" does not mean anything by itself. For example, that fact could happen if the benchmark has a good number of very easy or hard questions;
     (ii) "Completeness" seems to be more related to content/correctness than to style.

     Suggestion 1: for Table 2, it would be informative to present the rank (and Pearson) correlations between overall scores and specific scores.

     Suggestion 2: you could try regressing the overall score against the fine-grained scores and statistically test for the coefficients of this regression model. One limitation is that your covariates can be highly correlated to each other, inflating the estimate variances; you might need to collect more data. A second idea is running factor analysis in the full set of scores and then analyzing the low-dimensional factors.

     Suggestion 3: it would be more interesting to focus more on controlled experiments as those presented in Table 3.

2) I do not see a big value in introducing another meta-benchmark like SOS-Bench. We already have HELM and Open LLM Leaderboard, which are already very comprehensive. Moreover, size should not matter that much in the quality of benchmarks as emphasized by the authors; the accuracy of scores does not change much after a certain sample size (think about concentration inequalities, the law of large numbers, etc). Bigger sizes only make evaluation harder.

     Suggestion: instead of just combining many benchmarks by running LLMs in all of them, the authors could try to propose a method to derive a small and comprehensive "summary" (meta) benchmark. This summary benchmark could be much smaller (perhaps even ~100x smaller) than the current version of SOS-Bench and contain key/relevant examples/questions from each one of the three criteria in HHH. That would make the benchmark more usable and the contribution would be more expressive.

Minor points:
(i) Table 1 could be a correlation matrix.

questions:
NA
</content>
</thread_object>

</thread>
<thread>
creation date:
1730625775427

thread_type:
official_review

<thread_object>
creator: reviewer_j78n

<content>
summary:
This work exposes the issue of significant implicit bias in benchmarks that employ LLM judges to evaluate the alignment with human preferences. They evaluate LLMs with concrete metrics and suggest that the post-training and evaluation for the alignment of LLMs should be further designed to focus on more specific desirable factors.

strengths:
1. This work tested the bias of LLM-judge between and within criteria and provided convincing results.
2. This work analyzed the performance of the latest LLMs on concrete evaluation metrics and summarized the relationship between performance and post-training settings.

weaknesses:
1. The SOS bench proposed in this paper is basically a combination of existing static evaluation datasets regarding instruction following, world knowledge, safety, etc. What specific limitations of existing static datasets does SOS-Bench address? Are there any novel data processing, aggregation, or evaluation techniques used in creating SOS-Bench?

questions:
Please respond to the concerns in the "Weaknesses" part.

Q1. What specific role do you envision for static benchmarks with ground truth answers versus LLM-judge benchmarks in evaluating alignment? Are you suggesting completely replacing LLM-judge benchmarks or proposing a hybrid approach? How do you see these different evaluation methods complementing each other?

Q2. Could you provide more details on the process of converting LLM-judge preference results to the absolute percentage scores shown in Table 2? Specifically:

- 2.1 What exact steps were used to convert preference judgments to percentage scores?
- 2.2 Were any normalization or scaling steps applied?
- 2.3 How did you account for or mitigate potential biases in this conversion process?
- 2.4 Did you perform any sensitivity analysis on different scoring methods?"
</content>
</thread_object>

</thread>
<thread>
creation date:
1730707515569

thread_type:
official_review

<thread_object>
creator: reviewer_newj

<content>
summary:
This work investigates when LLM-judge preferences translate to progress on other metrics for alignment, and in cases of failure, why not. In particular, this work finds that LLM-judges have implicit biases, and prioritize stylistic preferences over eg factuality and safety. This work also introduces SOS-bench, a new alignment benchmark with ground truth and with the goal of gauging progress on helpful, honest, and harmless principles. Experiments show that data scaling in SFT and improving prompt diversity are important predictors of improved alignment.

strengths:
1. The paper is well motivated and definitely investigates an important question. It is valuable to know the failure modes of LLM-as-judges.
2. Paper is well written and logically organized.

weaknesses:
1. Since SOS-bench is a collection of existing benchmarks, I’m not sure whether this is a novel contribution (but rather a thoughtful choice for what benchmarks to use).
2. In table 2: Why not advocate for using correctness for the score (instead of using the raw score)?

questions:
Figure 2 claims that “in the SFT stage of post-training, the size of the dataset, rather than the method used to curate the data, is the strongest predictor of alignment.” Does this assume that all the data is deduplicated? Also, prior works seem to indicate that higher quality data (and data from different domains) can be more important than size. Can the authors comment on this?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732301707451

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank all the reviewers for their time and for providing constructive comments for enhancing the paper. We appreciate that reviewers recognized:

* Our work is well-motivated and well organized
* We provide convincing evidence of certain failure modes of LLM judges
* We provide important analysis on the performance of the latest LLMs using concrete evaluation metrics

We have conducted extensive additional experiments and added several new discussion points, which we hope will address some reviewer concerns.
We would be very happy to keep the discussion going, addressing any points that remain unclear, or any new suggestions. Thanks again for your suggestions!

## Revised Draft

Following helpful suggestions from our reviewers, we have added several new experimental results to the paper.
1. **(Improved presentation of main results.)** We replaced the table with a figure which is easier to analyze at a glance (we will still include the tables in our repository artifacts). In the figure, we now report the Pearson correlation for each factor w.r.t. overall score. We see that style score and overall score are perfectly correlated.
2. **(Ablating the choice of LLM judge.)** At a substantial cost of over $500 USD, we have followed the request of several reviewers by testing three additional judge LLMs; Claude Sonnet, GPT 4o, and GPT-3.5. We will include the raw results from these judges in our repository artifacts, and we report the aggregate correlations (Pearson and Spearman) in our main paper, Table 2. Remarkably, we find that across all four models, the average correlation between style score and overall score is still perfect.
3. **(Ablating the LLM Judge Pipeline.)** Some reviewers had questions about how much impact other potential confounds in the pipeline could be expected to have. We add 6 ablation studies, covering a wide range of variations, such as changing the semantics of the judge template, eliminating order-independent judging, changing the baseline model, and instructing the LLM judge not to give its own answer before judging; see Appendix I of our revised manuscript.
4. **(Ablating the choice of questions.)** In order to address concerns about how much the choice of questions in Arena-Hard-Auto impacts judgments, we run the same experiment again, substituting 500 questions from popular subreddits such as AskHistorians for the Arena-Hard-Auto questions. We find that, even after changing all of the questions, the Pearson’s correlation with the original judgments is .94, which strongly suggests that the content of the questions plays little role in the judgments.
5. **(Factor analysis.)** Following the requests of reviewers, we have added a factor analysis of Arena-Hard judgments (for a single judge) to Appendix K. We observe three factors which account for roughly 64% of the variance, which we term quality, accuracy and brevity. Troublingly, safety has very low loadings on all factors and a very low communality (0.04), suggesting it's measuring something almost entirely independent of the other metrics.

## Common Questions

**CQ1.** Why is SOS-Bench an important contribution, considering that HELM and Open LLM Leaderboard already exist?

**CA1.** While existing premier static benchmarks such as HELM Core (itself a meta-benchmark) and LiveBench are very important, and inspired aspects of our work, they are not designed as alignment benchmarks in the way that Arena-Hard-Auto is. The Open LLM Leaderboard has issues with saturation on some of its tasks, and only measures world knowledge. LiveBench has an implicit dependence on instruction following because all of its questions are open-ended, making it difficult to precisely assess the performance of smaller and older LLMs which tend to be used in academic papers. The HELM family is complex, with many non-intersecting metrics, models and datasets; important factors in model alignment, such as safety (bias and toxicity in HELM) are not covered for all models, and instruction following is mostly evaluated implicitly. Finally, it is not obvious how one would compare alignment methods (E.G., DPO vs PPO) using HELM. Therefore, we believe that SOS-Bench has the potential to be high-impact; it combines existing benchmarks in a novel way that makes it easy to measure progress on salient factors in alignment. We provide code to reproduce our benchmark exactly (and we will provide all relevant model weights once we are de-anonymized), which will make it easy to compare future work to our baselines. We also note that meta-benchmarks have served as a key contribution for multiple recent publications: https://arxiv.org/abs/2308.11838 (ICLR24), https://arxiv.org/pdf/2306.09468 (ICLR24).
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper investigates a well motivated problem: do LLM-judge preferences translate to progress on other, more concrete metrics for alignment, and if not, why not? In particular, the work finds that LLM-judges have implicit biases, and prioritize stylistic preferences over other more important substances like factuality and safety. They then present a new alignment benchmark, SOS-bench, with the goal of evaluating truthfully the progress on helpful, honest, and harmless principles. Experiments show that data scaling in SFT and improving prompt diversity are important predictors of improved alignment. 

All reviewers rated positively towards the work. More importantly, through the helpful and productive discussion period, we think the work is improved substantially with the amount of ablation studies added.