JuStRank: Benchmarking LLM Judges for System Ranking
Authors: name: Ariel Gera, name: Odellia Boni, name: Yotam Perlitz, name: Roy Bar-Haim, name: Lilach Eden, name: Asaf Yehudai

Given the rapid progress of generative AI, there is a pressing need to
systematically compare and choose between the numerous models and
configurations available. The scale and versatility of such evaluations make
the use of LLM-based judges a compelling solution for this challenge.
Crucially, this approach requires first to validate the quality of the LLM
judge itself. Previous work has focused on instance-based assessment of LLM
judges, where a judge is evaluated over a set of responses, or response pairs,
while being agnostic to their source systems. We argue that this setting
overlooks critical factors affecting system-level ranking, such as a judge's
positive or negative bias towards certain systems. To address this gap, we
conduct the first large-scale study of LLM judges as system rankers. System
scores are generated by aggregating judgment scores over multiple system
outputs, and the judge's quality is assessed by comparing the resulting system
ranking to a human-based ranking. Beyond overall judge assessment, our analysis
provides a fine-grained characterization of judge behavior, including their
decisiveness and bias.