Style over Substance: Failure Modes of LLM Judges in Alignment
  Benchmarking
Authors: name: Benjamin Feuer, name: Micah Goldblum, name: Teresa Datta, name: Sanjana Nambiar, name: Raz Besaleli, name: Samuel Dooley, name: Max Cembalest, name: John P. Dickerson

The release of ChatGPT in November 2022 sparked an explosion of interest in
post-training and an avalanche of new preference optimization (PO) methods.
These methods claim superior alignment by virtue of better correspondence with
human pairwise preferences, often measured by LLM judges. In this work, we
attempt to answer the following question -- do LLM-judge preferences translate
to progress on other, more concrete metrics for alignment, and if not, why not?
We define a concrete metric for alignment, and introduce SOS-Bench, the largest
standardized, reproducible LLM meta-benchmark to date. We find that (1)
LLM-judgments do not correlate with concrete measures of safety, world
knowledge, and instruction following; (2) LLM judges have powerful implicit
biases, prioritizing style over factuality and safety; and (3) the supervised
fine-tuning (SFT) stage of post-training, and not the PO stage, has the
greatest impact on alignment, with data scaling and prompt diversity as the
driving factors. Our codebase and complete results can be found at
https://github.com/penfever/sos-bench.