J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning
Authors: name: Chenxi Whitehouse; name: Tianlu Wang; name: Ping Yu; name: Xian Li; name: Jason Weston; name: Ilia Kulikov; name: Swarnadeep Saha
Published: 2025-05-15 14:05:15

Abstract: The progress of AI is bottlenecked by the quality of evaluation, and powerful
LLM-as-a-Judge models have proved to be a core solution. Improved judgment
ability is enabled by stronger chain-of-thought reasoning, motivating the need
to find the best recipes for training such models to think. In this work we
introduce J1, a reinforcement learning approach to training such models. Our
method converts both verifiable and non-verifiable prompts to judgment tasks
with verifiable rewards that incentivize thinking and mitigate judgment bias.
In particular, our approach outperforms all other existing 8B or 70B models
when trained at those sizes, including models distilled from DeepSeek-R1. J1
also outperforms o1-mini, and even R1 on some benchmarks, despite training a
smaller model. We provide analysis and ablations comparing Pairwise-J1 vs
Pointwise-J1 models, offline vs online training recipes, reward strategies,
seed prompts, and variations in thought length and content. We find that our
models make better judgments by learning to outline evaluation criteria,
comparing against self-generated reference answers, and re-evaluating the
correctness of model responses.