Quantitative LLM Judges
Authors: name: Aishwarya Sahoo; name: Jeevana Kruthi Karnuthala; name: Tushar Parmanand Budhwani; name: Pranchal Agarwal; name: Sankaran Vaidyanathan; name: Alexa Siu; name: Franck Dernoncourt; name: Jennifer Healey; name: Nedim Lipka; name: Ryan Rossi; name: Uttaran Bhattacharya; name: Branislav Kveton
Published: 2025-06-03 14:44:23

Abstract: LLM-as-a-judge is a framework in which a large language model (LLM)
automatically evaluates the output of another LLM. We propose quantitative LLM
judges, which align evaluation scores of existing LLM judges to human scores in
a given domain using regression models. The models are trained to improve the
score of the original judge by using the judge's textual evaluation and score.
We present four quantitative judges for different types of absolute and
relative feedback, which showcases the generality and versatility of our
framework. Our framework is more computationally efficient than supervised
fine-tuning and can be more statistically efficient when human feedback is
limited, which is expected in most applications of our work. We validate these
claims empirically on four datasets using two base judges. Our experiments show
that quantitative judges can effectively improve the predictive power of
existing judges through post-hoc modeling.