Judging the Judges: A Collection of LLM-Generated Relevance Judgements
Authors: name: Hossein A. Rahmani; name: Clemencia Siro; name: Mohammad Aliannejadi; name: Nick Craswell; name: Charles L. A. Clarke; name: Guglielmo Faggioli; name: Bhaskar Mitra; name: Paul Thomas; name: Emine Yilmaz
Published: 2025-02-19 17:40:32

Abstract: Using Large Language Models (LLMs) for relevance assessments offers promising
opportunities to improve Information Retrieval (IR), Natural Language
Processing (NLP), and related fields. Indeed, LLMs hold the promise of allowing
IR experimenters to build evaluation collections with a fraction of the manual
human labor currently required. This could help with fresh topics on which
there is still limited knowledge and could mitigate the challenges of
evaluating ranking systems in low-resource scenarios, where it is challenging
to find human annotators. Given the fast-paced recent developments in the
domain, many questions concerning LLMs as assessors are yet to be answered.
Among the aspects that require further investigation, we can list the impact of
various components in a relevance judgment generation pipeline, such as the
prompt used or the LLM chosen.
  This paper benchmarks and reports on the results of a large-scale automatic
relevance judgment evaluation, the LLMJudge challenge at SIGIR 2024, where
different relevance assessment approaches were proposed. In detail, we release
and benchmark 42 LLM-generated labels of the TREC 2023 Deep Learning track
relevance judgments produced by eight international teams who participated in
the challenge. Given their diverse nature, these automatically generated
relevance judgments can help the community not only investigate systematic
biases caused by LLMs but also explore the effectiveness of ensemble models,
analyze the trade-offs between different models and human assessors, and
advance methodologies for improving automated evaluation techniques. The
released resource is available at the following link:
https://llm4eval.github.io/LLMJudge-benchmark/