Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for
  LLM-as-a-Judge
Authors: name: Qiyuan Zhang; name: Yufei Wang; name: Yuxin Jiang; name: Liangyou Li; name: Chuhan Wu; name: Yasheng Wang; name: Xin Jiang; name: Lifeng Shang; name: Ruiming Tang; name: Fuyuan Lyu; name: Chen Ma
Published: 2025-02-18 03:31:06

Abstract: LLM-as-a-Judge, which generates chain-of-thought (CoT) judgments, has become
a widely adopted auto-evaluation method. However, its reliability is
compromised by the CoT reasoning's inability to capture comprehensive and
deeper details, often leading to incomplete outcomes. Existing methods mainly
rely on majority voting or criteria expansion, which is insufficient to address
the limitation in CoT. We propose Crowd-based Comparative Evaluation, which
introduces additional crowd responses to compare with the candidate responses,
thereby exposing deeper and more comprehensive details within the candidate
responses. This process effectively guides LLM-as-a-Judge to provide a more
detailed CoT judgment. Extensive experiments demonstrate that our approach
enhances evaluation reliability, achieving an average accuracy gain of 6.7%
across five benchmarks. Moreover, our method produces higher-quality CoTs that
facilitate judge distillation and exhibit superior performance in rejection
sampling for supervised fine-tuning (SFT), referred to as crowd rejection
sampling, thereby enabling more efficient SFT. Our analysis confirms that CoTs
generated by ours are more comprehensive and of higher quality, and evaluation
accuracy improves as inference scales.