# Deeper Than the Pool: Are Language Models Really Suited for Completion of Unlabeled Query-Document Judgments ?

Saber Zahhar, Nazim Kazar, Alexandre Dahan

<figure><img src="image_1.png" /></figure>

## To cite this version:

Saber Zahhar, Nazim Kazar, Alexandre Dahan. Deeper Than the Pool: Are Language Models Really Suited for Completion of Unlabeled Query-Document Judgments ?. LLM4Eval @ SIGIR 2025, Clemencia Siro; Hossein A. Rahmani; Mohammad Aliannejadi; Nick Craswell; Charles L. A. Clarke; Guglielmo Faggioli; Bhaskar Mitra; Paul Thomas; Emine Yilmaz, Jul 2025, Padua, Italy. hal-05121639

HAL Id: hal-05121639

https://hal.science/hal-05121639v1

Submitted on 1 Jul 2025

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Deeper Than the Pool: Are Language Models Really Suited for Completion of Unlabeled Query-Document Judgments ?

Saber ZAHHAR

saber.zahhar@devoteam.com

Devoteam

Levallois-Perret, France

Nazim KAZAR

nazimkazar@yahoo.fr

Devoteam

Paris, France

Alexandre DAHAN

alexandre.dahan@devoteam.com

Devoteam

Levallois-Perret, France

## Abstract

Automatic relevance assessment using large language and vision-language models promises to alleviate the time, cost and bias inherent to traditional query-document relevance judgment tasks. In this work, we conduct a large-scale evaluation of 84 open-source and proprietary models on the conQA benchmark, consisting of descriptive and conceptual queries. We evaluate models that have been fed text-only, image-only and combined modalities under the quadratic Cohen's κ and Spearman's ρ against both majority-vote and average-vote human aggregations. Our results show that models, especially GPT-4o, aligns closely with human consensus on descriptive queries and direct visual input, even surpassing the 95% bootstrapped upper bound of individual human agreement in the image-only setting. However, all models experience notable performance degradation on abstract conceptual queries and when relying solely on caption text. We further observe diminishing returns in performance, cost and latency as a model's size increases. We release both human and machine votes¹ to spur future work on bridging the performance gap in multimodal relevance judgment.

## CCS Concepts

* Information systems → Retrieval models and ranking; Reliability assessment; Multimedia and multimodal retrieval.

## Keywords

Information Retrieval, Large Language Models, Vision-Language Models, LLM as a Judge

## ACM Reference Format:

Saber ZAHHAR, Nazim KAZAR, and Alexandre DAHAN. 2025. Deeper Than the Pool: Are Language Models Really Suited for Completion of Unlabeled Query-Document Judgments?. In *Proceedings of SIGIR 2025 (LLM4Eval @ SIGIR 2025)*. ACM, New York, NY, USA, 11 pages. https://doi.org/XXXXXXX. XXXXXXX

¹https://drive.google.com/drive/folders/17vU5Pa-Q1vt40KI4L6vSnQ-IK8T2eOip

## 1 Introduction

Most natural language processing tasks are evaluated through a comparison of a model's output against a gold standard. The production of said gold standard traditionally always involved humans. Information Retrieval (IR) is no exception to this constraint. IR systems are given a set of documents to rank against a query, i.e. systems must provide a relevance assessment between a query and all considered documents. This is the classical paradigm, named Cranfield [9], upon which most IR systems are designed.

As previously mentioned, evaluating an IR system's output, i.e. a set of query-document pairs tagged with a relevance grade, requires that said pairs are already assessed. These assessments, or *relevance judgments*, represent the *ground truth*. This standard procedure introduces a web of constraints:

* **Annotation costs:** Recruiting, training, and retaining skilled annotators, especially domain experts, can easily rack up a significant amount of budget. For example, the popular Text Retrieval Conference (TREC) usually calls for the training and monitoring of six contractors. There is also a need for software development to support the recording of relevance judgments correctly and efficiently [11].

* **Time-consuming processes:** Manual relevance judgments typically take days or weeks to collect and curate. TREC's annotation process usually takes two to four weeks [11].

* **Limited scalability:** As modern IR benchmarks grow to millions of query-document pairs, exhaustive coverage becomes infeasible. A compromise, known as *pooling*, selects the top results from one or more IR systems which are then manually assessed for relevance. However, this may introduce inadvertently artifacts into which documents are judged and which ones are not. For example, if pooling was done through the popular probabilistic framework Best Matching 25 [21], chances are that this same model may score higher in the test collection [5, 6].

* **Subjectivity and inconsistency:** For all its intent and purposes, gold standards based on human annotations are destined to be biased. Obviously, humans that participate in the relevance judgments can never produce a perfect ideal, if such a thing exists. However, ensuring a sufficient sample size for each query-document pair is critical to reduce the effect of noisy votes and to converge towards a faithful representation of human behavior.

* **Crowdsourcing quality trade-offs:** While non-expert crowd workers can reduce costs, they introduce quality control challenges, necessitating redundant annotations, complex aggregation strategies, and still often failing to match expert-level judgments [12, 20].

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.

LLM4Eval @ SIGIR 2025, Padua, Italy

© 2025 Copyright held by the owner/author(s). Publication rights licensed to ACM.

ACM ISBN 978-1-4503-XXXX-X/2018/06

https://doi.org/XXXXXXX.XXXXXXX
With the democratization of language models, a growing trend emerged where researchers investigated the capacity of neural networks to mimic or even replace altogether humans from the annotation process. This field, usually referred to as *LLM as a Judge*, has been met with stark controversies from both ends of the spectrum.

Indeed, advocates such as [25] made statements regarding the total replacement of humans in the annotation task, citing their inherent limitation in judging the entire set of query-document pairs. They showed that large language models (LLMs) agree significantly with original human annotations and could potentially be used to *patch* missing judgments. Similar claims have been done in *product search* [18], *legal case retrieval* [17] and *low-resource languages* [10].

However, [1] challenge these optimistic claims by showing that, when an LLM is only used to *patch* missing relevance judgments, the resulting system-ranking correlations drop dramatically. Thus, mixing human and LLM annotations yields substantially lower agreement rates than if one only used LLM judgments. Still, using only LLM judgments can also lead to the pooling bias aforementioned. The authors reveal that, depending on which model is employed, IR runs can be either unfairly boosted or penalized, this bias grows almost linearly with the proportion of judgments done by LLMs. This bias as been observed in several other instances as well [16, 19, 26]. Likewise, [3] demonstrates how easily LLMs can be fooled by artificially injecting query terms into the documents, tricking the LLMs with Search Engine Optimization tactics.

While an unpopular opinion, other researchers call for a complete embargo against the use of LLMs. [23] and [8] raise several paradoxes in replacing humans by LLMs in the relevance judgment. Notably, [23] points out that when an LLM is used to generate the *ground-truth* relevance judgments, we have in effect declared the model's own predictions to be the gold standard. In that sense, [8] arguments that any new IR system evaluated against judgments produced by an earlier, weaker LLM will necessarily appear of less or equal performance. This self-imposed ceiling can in practice hinder real advances and disincentivize progress, trapping the field in a bottomless pit of its own making.

Still, most researchers have accepted and adapted to the ever-growing presence of LLMs in IR. [24, 28] introduce effective methods to include LLMs in the annotation process while mitigating their inherent biases. [4] provides a benchmark of several LLMs on 20 evaluation tasks. They reveal that LLM performance is far from systematic and greatly varies across a wide range of configurations (human expertise, machine or human generated text, grading scale etc.). Thus, they strongly advise to first validate LLM performance against task-specific human judgments before any deployment of automated LLM-as-a-Judge evaluation pipelines.

Taken together, the foregoing review highlights that conventional IR evaluation, which has been grounded in costly and time-consuming annotation process as well as limited in scale and potentially biased through pooling, has driven researchers to explore LLMs as automated judges, with mixed and sometimes paradoxical results. While some work reports high agreement between LLMs and humans and even suggests full replacement of human annotators, others demonstrate that mixing or relying solely on LLM-generated judgments can introduce systematic biases, degrade

correlation with true system rankings, and culminate in a out-of-control and black-box like evaluation task.

In light of this context, we propose to study conQA [22], a multi-modal IR dataset of images paired with text captions sampled from MS-COCO [15]. conQA puts at our disposal 80 queries divided into 30 descriptive and 50 conceptual queries. A conceptual query "*does not mention specific objects or actions, but refers to a generic abstraction, e.g., working class life.*" The original authors of conQA reveal that state-of-the-art IR vision-language models (VLMs) suffer up to a x4 drop in performance on conceptual queries compared with descriptive queries, revealing how difficult this innovative set of queries can be. Furthermore, its reliance on pooled, crowd-sourced judgments may further compound annotation bias. By examining LLMs and VLMs agreement with human annotators on varying configurations, our work aims to disentangle the effects of modality and query complexity on both judgment reliability. Our results indicate that (V/L)LMs align well with human judgments on descriptive queries or image input. However, their performance is considerably impacted when considering conceptual queries and text input. Some proprietary models like GPT-4o align better to humans than humans themselves. Recent open-source models like Gemma-3-27b-it hold promise for the future of this field.

# 2 Experimental Setup

## 2.1 Dataset: conQA

<figure><img src="image_3.png" /></figure>

A giraffe standing in a grassy savanna with a blue sky and lake in the background.

<caption>**Figure 1: Example of a MS-COCO document annotated with a caption.**</caption>

Introduced by [22], conQA is a benchmark dataset designed to evaluate text-to-image retrieval models beyond traditional caption-based queries. It comprises 80 queries divided into 50 conceptual and 30 descriptive queries. Descriptive queries explicitly mention objects present in the image, such as "people chopping vegetables," while conceptual queries refer to abstract concepts like "working class life" without directly naming specific objects. Each class of query share the same distribution with regards to its length in characters and words, with an average of 3.4 words per query. The dataset sampled over 43,000 images from MS-COCO [15] of everyday life scenery. [22] reveals that state-of-the-art IR VLMs suffer up to a x4 drop in performance on conceptual queries compared with descriptive queries, revealing how difficult this innovative set of queries can be. In their annotation process, each query was evaluated against a 100 pooled images, resulting in 6,941 unique images having at least 3 crowdsourced annotations. This means
that only 16% of the dataset has some sort of projection against the queries.

## Relabeling conQA

The original relevance judgments of conQA suffer from significant drawbacks:

* Though non-binary, conQA's annotation guidelines do not provide grades of relevancy but rather gives the possibility to answer "Unsure" rather than "Relevant" or "Irrelevant";

* Despite the validation controls, some query-document pairs are not correct. For example, the query "planting flowers pot roots" has several non-relevant images labeled as "Relevant" in the top-5 ranking;

* Even so, the small sample size per query-document pair can be difficult to aggregate, e.g. 3 annotators each voting for one of the unique labels "Irrelevant", "Unsure" and "Relevant" does not provide any useful information.

Thus, we decided to expand the annotation by:

* Introducing a 4-graded scale from "0 - Not Relevant" to "3 - Perfectly Relevant", passing by "1 - Somewhat Relevant" and "2 - Highly Relevant". This actively encourages the user to make a decision.

* Improving validation controls by filtering based on times-tamps and **extreme** disagreement with other annotators, i.e. distribution similar to a uniform law.

* Sampling 10-15 images per query by stratification (using the original conQA's relevance labels), this resulted in 1,052 unique query-document pairs to be annotated by around 30 work colleagues from various fields, e.g. "Data Scientists", "Network Engineers", "Project Managers" etc.

Though covering less distinct pairs, we acquire more confidence in each pair as per [24]. This allows us to conduct a more robust and reliable comparison between model judgments and aggregated human votes, ensuring that each annotation reflects a clearer consensus and reduces noise from ambiguous or low-agreement labels.

## 2.2 Model Selection

Using the APIs of NVIDIA<fnref n="2" /> and OpenAI<fnref n="3" />, we implemented a diverse collection of models: closed-source systems such as OpenAI's GPT-4, open-weight models like Google's Gemma-3-27b-it, multimodal architectures including Microsoft's Phi-3/3.5/4-vision series, and text-only like Meta's LLaMA 3.1/3.2/3.3 series.

However, more than half of NVIDIA's advertised models proved out-of-scope for our task due to region restrictions, constrained output, non-English support, and similar constraints. Appendix A provides a comprehensive summary of the excluded models.

## 2.3 Prompt Design

There is no consensus regarding the ideal design for human annotation nor machine-readable prompt. [4] revealed that no prompting strategy improves systematically the annotation process for LMs. Still, we took inspiration of [25, 28]'s prompt design as it provides both a machine-readable and human-digestible prompt for our task.

When considering the text modality, we concatenated the available MS-COCO captions. When considering the image modality, we decoded the image's content in base64 as per the API's guidelines. Figure 2 showcases the prompt we used depending on the current modality of the document considered.

### Initial Query-Document Prompt

A query and a document are presented to you. You must assess how relevant the document is to the query. The relevance grades are one of the following:

* **Not Relevant:** There is no relevance between the document and the query.

* **Somewhat Relevant:** Debatable match; I can't say it's completely unrelated.

* **Highly Relevant:** Good match, though not perfect.

* **Perfectly Relevant:** A perfect match between the query and the document; I would expect this document to appear as the top result in a search engine.

**Query:** {query}

**Document:**

+ (if "captions" in mode)

' .join(docCaptions)

+ (if "image" in mode)

"<img src='data:image/png;base64,{img64}'/>"

+

DO NOT EXPLAIN YOUR ANSWER.
JUST OUTPUT ONLY ONE OF THE FOLLOWING:
'Not Relevant', 'Somewhat Relevant', 'Highly Relevant', 'Perfectly Relevant'

<caption>Figure 2: Polished version of the prompt layout for query-document relevance assessment</caption>

## 2.4 Evaluation

Given the ordinal nature of the relevance judgments. We evaluate our systems both in terms of Cohen's Weighted Kappa κ and Spearman's Rank correlation ρ between two models. Since we want to further penalize two annotators that say "Not Relevant" and "Perfectly Relevant" rather than "Somewhat Relevant" and "Highly Relevant", we select the quadratic weighting for κ.

Since we have more than one individual human annotator. We follow [4] and aggregate these votes using two statistical operations; the mode and mean, which will be referred from now on as "Majority voter" and "Average voter" respectively.

<footnote n="2">²https://build.nvidia.com/</footnote>

<footnote n="3">³https://platform.openai.com/docs/</footnote>
<table><caption>**Table 1:** Example of a model response in our experiments, showing metadata such as the query, document identifier, modality considered, model used, response time, token counts and relevance vote.</caption><thead><tr><th>Key</th><th>Value</th></tr></thead><tbody><tr><td>Query</td><td>people chopping vegetables (query_00)</td></tr><tr><td>Document</td><td>2349267.jpg</td></tr><tr><td>Model</td><td>google/gemma-3-27b-it</td></tr><tr><td>Mode</td><td>image+captions</td></tr><tr><td>Time (ms)</td><td>15763</td></tr><tr><td>Input Tokens</td><td>486</td></tr><tr><td>Output Tokens</td><td>2</td></tr><tr><td>Vote</td><td>Not Relevant</td></tr></tbody></table>

# 3 Results

## 3.1 Human assessments

Table 2 indicates a broad range of inter-annotator agreement across individual raters. When comparing each rater to the Majority voter, $κ$ scores span from 0.47 up to 0.82 with a mean of 0.67, while agreement against the Average voter ranges from 0.44 to 0.83 with a mean of 0.68. Spearman correlations follow a similar pattern, averaging 0.66 versus the Majority and 0.68 versus the Average.

The Cohen's $κ$ values place overall agreement solidly in the “substantial” range [13], but the wide span highlights nevertheless the considerable heterogeneity among raters. Spearman also corroborates that rank-order judgments are similarly consistent, but again with enough variation that some annotators disagree markedly on relative quality or severity. In practice, this level of dispersion warns against relying on any single annotator and underscores the value of aggregating multiple perspectives.

**Does an annotator's professional background condition its agreement with internal and external voters?** There is no clear evidence of stronger consensus within any single discipline: for instance, Developers achieve an intra-cluster mean Kappa of 0.562 versus 0.558 with raters outside their group, and DevOps practitioners display comparable intra-/extra-cluster agreement. An outlier is the Data cohort, whose intra-cluster Kappa is only 0.276 but whose extra-cluster agreement rises to 0.573—a discrepancy likely driven by the relatively small number of directly comparable annotations among Data Engineers.

**Does the original labeling schema reliably reflect consensus relevance across annotator judgments per query?** Figure 4 illustrates clear misalignments between the original conQA labels and observed annotator votes. For example, the conceptual query “terrors of the unknown” received no “Perfectly Relevant” votes despite three sampled items being sourced from documents labeled “Relevant.” Conversely, some queries (e.g., “public transportation for the long range”) elicited nearly uniform distributions, suggesting a lack of clarity or consensus around relevance for those query types. These inconsistencies raise concerns about the fidelity, reliability and feasability of reaching a consensus on conceptual queries.

**Does query type condition the distribution and polarization of annotator relevance judgments?** Although a Mann-Whitney

<table><caption>**Table 2:** Inter-annotator agreement.</caption><thead><tr><th>Humanid</th><th>κ (Maj.)</th><th>κ (Avg.)</th><th>ρ (Maj.)</th><th>ρ (Avg.)</th></tr></thead><tbody><tr><td>majority</td><td>1.00</td><td>0.84</td><td>1.00</td><td>0.81</td></tr><tr><td>average</td><td>0.84</td><td>1.00</td><td>0.81</td><td>1.00</td></tr><tr><td>Data Engineer 1</td><td>0.72</td><td>0.69</td><td>0.59</td><td>0.60</td></tr><tr><td>Data Engineer 2</td><td>0.55</td><td>0.44</td><td>0.39</td><td>0.30</td></tr><tr><td>Data Engineer 3</td><td>0.80</td><td>0.77</td><td>0.79</td><td>0.72</td></tr><tr><td>Data Engineer 5</td><td>0.61</td><td>0.67</td><td>0.61</td><td>0.68</td></tr><tr><td>Data Engineer 6</td><td>0.47</td><td>0.53</td><td>0.48</td><td>0.55</td></tr><tr><td>Data Scientist 1</td><td>0.70</td><td>0.76</td><td>0.74</td><td>0.78</td></tr><tr><td>DevOps Architect 1</td><td>0.54</td><td>0.61</td><td>0.59</td><td>0.68</td></tr><tr><td>DevOps Architect 2</td><td>0.80</td><td>0.83</td><td>0.81</td><td>0.83</td></tr><tr><td>DevOps Engineer 1</td><td>0.75</td><td>0.74</td><td>0.71</td><td>0.70</td></tr><tr><td>DevOps Engineer 2</td><td>0.66</td><td>0.67</td><td>0.66</td><td>0.69</td></tr><tr><td>DevOps Engineer 4</td><td>0.65</td><td>0.71</td><td>0.64</td><td>0.73</td></tr><tr><td>DevOps Engineer 5</td><td>0.79</td><td>0.77</td><td>0.80</td><td>0.81</td></tr><tr><td>DevOps Engineer 6</td><td>0.65</td><td>0.68</td><td>0.68</td><td>0.74</td></tr><tr><td>DevOps Engineer 7</td><td>0.61</td><td>0.67</td><td>0.68</td><td>0.80</td></tr><tr><td>DevOps Engineer 8</td><td>0.64</td><td>0.63</td><td>0.68</td><td>0.70</td></tr><tr><td>Developer 1</td><td>0.67</td><td>0.64</td><td>0.69</td><td>0.65</td></tr><tr><td>Developer 1</td><td>0.75</td><td>0.78</td><td>0.70</td><td>0.74</td></tr><tr><td>Developer 1</td><td>0.65</td><td>0.63</td><td>0.64</td><td>0.67</td></tr><tr><td>Developer 1</td><td>0.58</td><td>0.69</td><td>0.63</td><td>0.75</td></tr><tr><td>Developer 1</td><td>0.78</td><td>0.76</td><td>0.75</td><td>0.74</td></tr><tr><td>Product Analyst 1</td><td>0.66</td><td>0.69</td><td>0.59</td><td>0.65</td></tr><tr><td>Product Analyst 2</td><td>0.63</td><td>0.72</td><td>0.64</td><td>0.79</td></tr><tr><td>Product Owner 1</td><td>0.54</td><td>0.61</td><td>0.55</td><td>0.61</td></tr><tr><td>Product Owner 2</td><td>0.61</td><td>0.59</td><td>0.62</td><td>0.56</td></tr><tr><td>Product Owner 3</td><td>0.67</td><td>0.74</td><td>0.66</td><td>0.75</td></tr><tr><td>Product Owner 4</td><td>0.77</td><td>0.79</td><td>0.70</td><td>0.77</td></tr><tr><td>Product Owner 5</td><td>0.56</td><td>0.58</td><td>0.48</td><td>0.49</td></tr><tr><td>Product Owner 6</td><td>0.73</td><td>0.70</td><td>0.72</td><td>0.71</td></tr><tr><td>Product Owner 7</td><td>0.64</td><td>0.59</td><td>0.62</td><td>0.56</td></tr><tr><td>Project Manager 1</td><td>0.59</td><td>0.68</td><td>0.58</td><td>0.68</td></tr><tr><td>Project Manager 2</td><td>0.82</td><td>0.74</td><td>0.74</td><td>0.69</td></tr><tr><td>Project Manager 3</td><td>0.82</td><td>0.77</td><td>0.80</td><td>0.75</td></tr><tr><td>Project Manager 4</td><td>0.68</td><td>0.71</td><td>0.64</td><td>0.66</td></tr><tr><td>Project Manager 5</td><td>0.68</td><td>0.70</td><td>0.72</td><td>0.79</td></tr><tr><td>Project Manager 6</td><td>0.57</td><td>0.55</td><td>0.60</td><td>0.56</td></tr><tr><td>Project Manager 7</td><td>0.56</td><td>0.62</td><td>0.66</td><td>0.72</td></tr><tr><td>Project Manager 8</td><td>0.73</td><td>0.64</td><td>0.74</td><td>0.65</td></tr><tr><td>Project Manager 9</td><td>0.67</td><td>0.63</td><td>0.69</td><td>0.70</td></tr><tr><td>Risk & Compliance Engineer 1</td><td>0.69</td><td>0.71</td><td>0.70</td><td>0.73</td></tr><tr><td>Risk & Compliance Engineer 2</td><td>0.56</td><td>0.61</td><td>0.57</td><td>0.61</td></tr><tr><td>Risk & Compliance Engineer 3</td><td>0.63</td><td>0.58</td><td>0.60</td><td>0.50</td></tr><tr><td>Scrum Master 1</td><td>0.70</td><td>0.74</td><td>0.73</td><td>0.74</td></tr><tr><td>Scrum Master 2</td><td>0.82</td><td>0.73</td><td>0.71</td><td>0.58</td></tr><tr><td>Scrum Master 3</td><td>0.75</td><td>0.78</td><td>0.73</td><td>0.75</td></tr><tr><td>Solution Engineer 1</td><td>0.72</td><td>0.76</td><td>0.74</td><td>0.77</td></tr><tr><td>Solution Engineer 2</td><td>0.82</td><td>0.79</td><td>0.81</td><td>0.77</td></tr><tr><td>System & Network Administrator 1</td><td>0.59</td><td>0.56</td><td>0.65</td><td>0.62</td></tr><tr><td>System & Network Administrator 3</td><td>0.75</td><td>0.73</td><td>0.67</td><td>0.69</td></tr><tr><td>System & Network Administrator 4</td><td>0.56</td><td>0.62</td><td>0.60</td><td>0.66</td></tr><tr><td>System & Network Engineer 2</td><td>0.67</td><td>0.71</td><td>0.69</td><td>0.71</td></tr><tr><td>System & Network Engineer 3</td><td>0.69</td><td>0.69</td><td>0.69</td><td>0.66</td></tr><tr><td>System & Network Engineer 4</td><td>0.60</td><td>0.66</td><td>0.63</td><td>0.74</td></tr><tr><td>System & Network Engineer 5</td><td>0.72</td><td>0.69</td><td>0.65</td><td>0.63</td></tr></tbody></table>

Test showed no significant difference in vote variance between descriptive and conceptual queries—implying comparable levels of annotator disagreement—Figure ?? reveals pronounced divergence in mean vote distributions. Conceptual queries struggle to attract “Perfectly Relevant” judgments (7.5%) compared to descriptive ones (around 15%). Notably, the missing votes shift not to the adjacent “Highly Relevant” category but rather to “Somewhat” or
"Not Relevant," suggesting a polarization effect when expectations of perfect relevance are unmet. This highlights the nuanced challenge of achieving fine-grained alignment on abstract semantic similarity.

<figure><img src="image_4.png" /><figcaption>**Figure 3:** Distribution of relevance judgments per query. ■ indicates a conceptual query.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>**Figure 4:** Distribution of relevance judgments per query type.</figcaption></figure>

Finally, we have conducted a thorough analysis of the consensus observed among human raters. Next, we will examine how language models align with the two aggregated raters.

## 3.2 Language Models as Judges

Table 5 allows us to draw several critical observations from our experiments.

**Can LLMs reliably annotate like human raters?** First, language models perform significantly worse than individual humans raters. Besides the obvious incapable small language models such as Google's Gemma-2b or Meta's LLaMa3.1-1b, Qwen's QwQ-32b and Google's Paligemma also are incapable of producing comprehensive outputs. These deficiencies are not solely attributable to model size, but rather reflect a deeper structural gap between language model training objectives and the nature of human relevance judgment. The best model is gpt-4o which averages a score of 0.69 on both metrics and against both aggregated raters. Gemma-3-27b-it is a great recent addition to the list of models as it rivals quite fairly with proprietary models, though its runtime is significantly greater, it is a lighter model in parameter size, increasing accessibility when compared to LLaMa3.3-70b or OpenAI's models.

**Do LLMs align more closely with smoothed human aggregates or majority-driven decisions?** Language models tend to align better with the Average rater than the Majority voter. Indeed, not one model agrees more with the latter than the former across both Kappa and Spearman. This tendency is not observed with individual humans who seemed to skew uniformly for both aggregates. The Average rater effectively smooths out idiosyncratic judgments, representing the “centroid” of the rating distribution. Language Models which are trained to minimize average loss over massive, heterogeneous corpora—naturally internalize this smoothed signal, gravitating toward mean behavior rather than hard consensus. The Majority voter, by contrast, enforces a hard threshold: only the modal label survives. Models optimized on continuous objectives (e.g. log-likelihood) struggle to replicate such discrete, categorical consolidation, hence their poorer alignment.

**How does model complexity affect annotation performance, and where are its limits?** Figure 5 indicates that all families of models show rapid improvement when you change the complexity of the architecture (parameter size, distillation etc.). We averaged the two metrics (Kappa, Spearman) for the two rates (Majority, Average) as the disaggregation of plot did not introduce a significant observation. However, a plateau is reached and diminishing results are starting to appear when passing the 50B parameter threshold.

<figure><img src="image_6.png" /><figcaption>**Figure 5:** Mean Agreement across both Metrics (Kappa, Spearman) and both Aggregated Raters (Average, Majority) against the Parameter Size (in billions). Note: OpenAI's parameter sizes are only estimates.</figcaption></figure>

**How does modality influence the alignment of LLM judgments with human annotations?** We observe a clear, systematic dependency of model performance on the modality used for the relevance judgment. Table 6 shows that across both gpt-4.1 and gpt-4o, the image-only modality yields the highest agreement with human raters (e.g. GPT-4o: Majority Kappa = 0.68, Average Kappa = 0.72), substantially outperforming captions-only (Majority Kappa = 0.53, Average Kappa = 0.54). This gap indicates that direct access to visual features is crucial for fine-grained relevance judgments, but also that the human-generated captions from MS-COCO have undoubtedly reduced the available informative content required
for evaluation. Contrary to expectations, combining image with captions rarely outperforms the best single modality. For GPT-4o, the image+captions setting (Average Kappa = 0.66) sits between captions-only (0.54) and image-only (0.72). This suggests that naive concatenation can introduce conflicting signals. Other models like Google's Gemma-3-27b-it are more robust to this noise, as just under a point of score was lost in Kappa (Majority) and Spearman (Average).

**What are the computational trade-offs of using different input modalities for relevance judgments?** Nevertheless, adopting images as a modality incurs a non-trivial runtime cost, e.g. gpt-4o shows in caption-only: 0.56s against image-only: 1.98s. Surprisingly, the fusion condition does not always compound this overhead—in some cases like it is the case of gpt-4.1 that runs faster on image+captions (1.17s) than on image-only (2.79s), perhaps due to differences in prompting or early-exit heuristics.

**Can LLMs reach or surpass the agreement ceiling of individual human raters?** The upper-bound is computed by sampling the metric score of individual humans against the aggregated rater, this provides an estimate of the maximum agreement value reachable by any randomly sampled individual rater. An interesting observation is that gpt-4o actually reaches or even surpasses the bootstrapped 95% upper-bound agreement score against the Average voter, meaning it “outperforms” the typical maximum agreement expected from any single human rater. By virtue of its training to minimize average loss over vast, heterogeneous data, gpt-4o may yield judgments that hover very close to the mean human opinion with remarkably low variance. Humans, by contrast, exhibit more contrasted judgments, i.e. even the best-performing individual raters will sometimes diverge from the consensus.

**What are the cost and efficiency trade-offs of deploying LLM-based annotation pipelines at scale?** NVIDIA's platform is generous enough to provide the API for free with the only limitation of 40 requests per minute. OpenAI's pricing model is explicit but the underlying models (parameter size, infrastructure the API runs on etc.) are not available, thus rendering infeasible the projection of GPT's price-per-token to NVIDIA's models. As a side note, gpt-4o consumed more than twice as much input tokens when switching from captions to image. Thus, it takes x4 the time and x2 the cost to run image-based judgement than caption-based judgement.

## 3.3 A closer look at conceptual performance

The main challenge of conQA resides in its conceptual queries. Tables 4, 3 reveal that the dip in model performance on conceptual queries vs. descriptive queries is substantial and non-trivial. This likely reflects the symbolic abstraction gap in LLMs [2], despite their powerful pattern recognition, they still struggle with decontextualized, abstract or even meme-related relationships that are not explicitly grounded in perceptual data.

The models could benefit from a structured knowledge graph or taxonomy that supports symbolic inference beyond distributional co-occurrences. Purely data-driven embeddings tend to blur fine-grained concept boundaries and struggle with counterfactual or hypothetical reasoning that wasn't richly represented in the training corpus. Yet, it is important to note that even humans have a clear struggle in agreeing on the matter.

<table><caption>Table 3: Model Performance on Descriptive queries.</caption><thead><tr><th rowspan="2">Model</th><th rowspan="2">Modality</th><th colspan="2">Majority Voter</th><th colspan="2">Average Voter</th></tr><tr><th>κ (0.78)</th><th>ρ (0.77)</th><th>κ (0.78)</th><th>ρ (0.79)</th></tr></thead><tbody><tr><td rowspan="2">google/gemma-3-27b-it</td><td>captions</td><td>0.63</td><td>0.65</td><td>0.70</td><td>0.69</td></tr><tr><td>image</td><td>0.67</td><td>0.69</td><td>0.71</td><td>0.72</td></tr><tr><td>meta/llama-3.3-70b-instruct</td><td>captions</td><td>0.77</td><td>0.77</td><td>0.78</td><td>0.76</td></tr><tr><td rowspan="2">openai/gpt-4o-2024-08-06</td><td>captions</td><td>0.67</td><td>0.71</td><td>0.72</td><td>0.73</td></tr><tr><td>image</td><td>0.78</td><td>0.77</td><td>0.80</td><td>0.79</td></tr></tbody></table>

<table><caption>Table 4: Model performance on conceptual queries.</caption><thead><tr><th rowspan="2">Model</th><th rowspan="2">Modality</th><th colspan="2">Majority Vote</th><th colspan="2">Average Vote</th></tr><tr><th>κ (0.60)</th><th>ρ (0.63)</th><th>κ (0.61)</th><th>ρ (0.64)</th></tr></thead><tbody><tr><td rowspan="2">google/gemma-3-27b-it</td><td>captions</td><td>0.39</td><td>0.42</td><td>0.35</td><td>0.41</td></tr><tr><td>image</td><td>0.52</td><td>0.53</td><td>0.53</td><td>0.52</td></tr><tr><td>meta/llama-3.3-70b-instruct</td><td>captions</td><td>0.47</td><td>0.44</td><td>0.52</td><td>0.53</td></tr><tr><td rowspan="2">openai/gpt-4o-2024-08-06</td><td>captions</td><td>0.36</td><td>0.38</td><td>0.35</td><td>0.40</td></tr><tr><td>image</td><td>0.55</td><td>0.55</td><td>0.63</td><td>0.63</td></tr></tbody></table>
<table><caption>Table 5: Model Performance by Modality and Voter Type. Values in parentheses indicate the 95% upper-bound estimate using bootstrapping. Bolded values are the best models across all other model-modalities; *underlined* values are the second-best.</caption><thead><tr><th rowspan="2">Model</th><th rowspan="2">Modality</th><th colspan="2">Majority Voter</th><th colspan="2">Average Voter</th><th rowspan="2">Avg. Time (ms)</th></tr><tr><th>κ (0.69)</th><th>ρ (0.68)</th><th>κ (0.70)</th><th>ρ (0.70)</th></tr></thead><tbody><tr><td>01-ai/yi-large</td><td>captions</td><td>0.65</td><td>0.66</td><td>0.71</td><td>0.70</td><td>426.40</td></tr><tr><td rowspan="2">adept/fuyu-8b</td><td>image</td><td>0.07</td><td>0.08</td><td>0.05</td><td>0.04</td><td>1200.24</td></tr><tr><td>image+captions</td><td>0.04</td><td>0.08</td><td>0.04</td><td>0.07</td><td>1533.02</td></tr><tr><td>ai21labs/jamba-1.5-large-instruct</td><td>captions</td><td>0.70</td><td>0.68</td><td>0.70</td><td>0.70</td><td>496.15</td></tr><tr><td>ai21labs/jamba-1.5-mini-instruct</td><td>captions</td><td>0.41</td><td>0.51</td><td>0.46</td><td>0.56</td><td>385.65</td></tr><tr><td>databricks/dbrx-instruct</td><td>captions</td><td>0.52</td><td>0.57</td><td>0.60</td><td>0.63</td><td>381.09</td></tr><tr><td>deepseek-ai/deepseek-r1</td><td>captions</td><td>0.67</td><td>0.69</td><td>0.72</td><td>0.72</td><td>9069.93</td></tr><tr><td>deepseek-ai/deepseek-r1-distill-qwen-14b</td><td>captions</td><td>0.68</td><td>0.68</td><td>0.71</td><td>0.70</td><td>7157.97</td></tr><tr><td>deepseek-ai/deepseek-r1-distill-qwen-32b</td><td>captions</td><td>0.70</td><td>0.71</td><td>0.75</td><td>0.73</td><td>9750.30</td></tr><tr><td>deepseek-ai/deepseek-r1-distill-qwen-7b</td><td>captions</td><td>0.55</td><td>0.61</td><td>0.62</td><td>0.64</td><td>4865.58</td></tr><tr><td>google/codegemma-1.1-7b</td><td>captions</td><td>0.29</td><td>0.38</td><td>0.32</td><td>0.38</td><td>272.95</td></tr><tr><td>google/gemma-2-27b-it</td><td>captions</td><td>0.67</td><td>0.66</td><td>0.74</td><td>0.71</td><td>422.99</td></tr><tr><td>google/gemma-2-2b-it</td><td>captions</td><td>0.49</td><td>0.57</td><td>0.53</td><td>0.58</td><td>237.43</td></tr><tr><td>google/gemma-2-9b-it</td><td>captions</td><td>0.64</td><td>0.66</td><td>0.71</td><td>0.69</td><td>357.98</td></tr><tr><td>google/gemma-2b</td><td>captions</td><td>0.04</td><td>0.10</td><td>0.04</td><td>0.11</td><td>225.22</td></tr><tr><td>google/gemma-3-1b-it</td><td>captions</td><td>0.21</td><td>0.26</td><td>0.20</td><td>0.26</td><td>204.09</td></tr><tr><td rowspan="3">google/gemma-3-27b-it</td><td>captions</td><td>0.63</td><td>0.65</td><td>0.70</td><td>0.69</td><td>21767.46</td></tr><tr><td>image</td><td>0.67</td><td>0.69</td><td>0.71</td><td>0.72</td><td>39364.26</td></tr><tr><td>image+captions</td><td>0.69</td><td>0.71</td><td>0.75</td><td>0.73</td><td>32268.11</td></tr><tr><td>google/gemma-7b</td><td>captions</td><td>0.35</td><td>0.48</td><td>0.40</td><td>0.51</td><td>335.95</td></tr><tr><td rowspan="2">google/paligemma</td><td>image</td><td>-0.01</td><td>-0.01</td><td>-0.01</td><td>-0.01</td><td>291.37</td></tr><tr><td>image+captions</td><td>-0.02</td><td>-0.08</td><td>-0.03</td><td>-0.13</td><td>307.57</td></tr><tr><td>google/recurrentgemma-2b</td><td>captions</td><td>0.02</td><td>0.10</td><td>0.02</td><td>0.10</td><td>481.78</td></tr><tr><td>ibm/granite-3.0-3b-a800m-instruct</td><td>captions</td><td>0.41</td><td>0.43</td><td>0.39</td><td>0.44</td><td>240.10</td></tr><tr><td>ibm/granite-3.0-8b-instruct</td><td>captions</td><td>0.57</td><td>0.54</td><td>0.60</td><td>0.58</td><td>307.57</td></tr><tr><td>ibm/granite-34b-code-instruct</td><td>captions</td><td>0.14</td><td>0.21</td><td>0.15</td><td>0.23</td><td>428.07</td></tr><tr><td>ibm/granite-8b-code-instruct</td><td>captions</td><td>0.02</td><td>0.06</td><td>0.02</td><td>0.07</td><td>335.07</td></tr><tr><td>igenius/colosseum_355b_instruct_16k</td><td>captions</td><td>0.62</td><td>0.66</td><td>0.68</td><td>0.68</td><td>462.65</td></tr><tr><td>igenius/italia_10b_instruct_16k</td><td>captions</td><td>0.37</td><td>0.44</td><td>0.41</td><td>0.47</td><td>292.93</td></tr><tr><td>mediatek/breeze-7b-instruct</td><td>captions</td><td>0.62</td><td>0.59</td><td>0.65</td><td>0.61</td><td>322.75</td></tr><tr><td>meta/llama-3.1-405b-instruct</td><td>captions</td><td>0.71</td><td>0.70</td><td>0.77</td><td>0.73</td><td>595.05</td></tr><tr><td>meta/llama-3.1-70b-instruct</td><td>captions</td><td>0.77</td><td>0.77</td><td>0.78</td><td>0.77</td><td>420.35</td></tr><tr><td>meta/llama-3.1-8b-instruct</td><td>captions</td><td>0.59</td><td>0.58</td><td>0.62</td><td>0.61</td><td>249.60</td></tr><tr><td>meta/llama-3.2-1b-instruct</td><td>captions</td><td>0.04</td><td>0.05</td><td>0.05</td><td>0.07</td><td>197.12</td></tr><tr><td>meta/llama-3.2-3b-instruct</td><td>captions</td><td>0.31</td><td>0.42</td><td>0.37</td><td>0.47</td><td>214.34</td></tr><tr><td>meta/llama-3.3-70b-instruct</td><td>captions</td><td>0.77</td><td>0.77</td><td>0.78</td><td>0.76</td><td>375.10</td></tr><tr><td>Microsoft/phi-3-medium-128k-instruct</td><td>captions</td><td>0.16</td><td>0.34</td><td>0.17</td><td>0.37</td><td>473.51</td></tr><tr><td>Microsoft/phi-3-medium-4k-instruct</td><td>captions</td><td>0.27</td><td>0.48</td><td>0.27</td><td>0.51</td><td>311.90</td></tr><tr><td>Microsoft/phi-3-mini-128k-instruct</td><td>captions</td><td>0.46</td><td>0.47</td><td>0.50</td><td>0.54</td><td>315.47</td></tr><tr><td>Microsoft/phi-3-mini-4k-instruct</td><td>captions</td><td>0.54</td><td>0.56</td><td>0.60</td><td>0.62</td><td>242.86</td></tr><tr><td>Microsoft/phi-3-small-128k-instruct</td><td>captions</td><td>0.51</td><td>0.57</td><td>0.57</td><td>0.61</td><td>1235.42</td></tr><tr><td>Microsoft/phi-3-small-8k-instruct</td><td>captions</td><td>0.56</td><td>0.58</td><td>0.62</td><td>0.62</td><td>241.69</td></tr><tr><td rowspan="2">Microsoft/phi-3-vision-128k-instruct</td><td>image</td><td>0.50</td><td>0.54</td><td>0.53</td><td>0.56</td><td>564.06</td></tr><tr><td>image+captions</td><td>0.56</td><td>0.58</td><td>0.61</td><td>0.64</td><td>582.28</td></tr><tr><td>Microsoft/phi-3.5-mini-instruct</td><td>captions</td><td>0.35</td><td>0.46</td><td>0.42</td><td>0.55</td><td>314.57</td></tr><tr><td>Microsoft/phi-3.5-moe-instruct</td><td>captions</td><td>0.53</td><td>0.58</td><td>0.58</td><td>0.62</td><td>414.75</td></tr><tr><td rowspan="2">Microsoft/phi-3.5-vision-instruct</td><td>image</td><td>0.53</td><td>0.55</td><td>0.55</td><td>0.57</td><td>325.77</td></tr><tr><td>image+captions</td><td>0.45</td><td>0.54</td><td>0.45</td><td>0.58</td><td>339.66</td></tr><tr><td>Microsoft/phi-4-mini-instruct</td><td>captions</td><td>0.39</td><td>0.45</td><td>0.39</td><td>0.47</td><td>252.80</td></tr><tr><td rowspan="2">Microsoft/phi-4-multimodal-instruct</td><td>image</td><td>0.52</td><td>0.57</td><td>0.52</td><td>0.59</td><td>414.42</td></tr><tr><td>image+captions</td><td>0.32</td><td>0.44</td><td>0.33</td><td>0.49</td><td>405.16</td></tr></tbody></table>
<table><caption>Table 5 (Suite): Model Performance by Modality and Voter Type. Values in parentheses indicate the 95% upper-bound estimate using bootstrapping. Bolded values are the best models across all other model-modalities; *underlined* values are the second-best.</caption><thead><tr><th rowspan="2">Model</th><th rowspan="2">Modality</th><th colspan="2">Majority Voter</th><th colspan="2">Average Voter</th><th rowspan="2">Avg. Time (ms)</th></tr><tr><th>κ (0.69)</th><th>ρ (0.68)</th><th>κ (0.70)</th><th>ρ (0.70)</th></tr></thead><tbody><tr><td>mistralai/mamba-codestral-7b-v0.1</td><td>captions</td><td>0.11</td><td>0.14</td><td>0.11</td><td>0.13</td><td>307.85</td></tr><tr><td>mistralai/mistral-7b-instruct-v0.2</td><td>captions</td><td>0.49</td><td>0.55</td><td>0.55</td><td>0.58</td><td>375.06</td></tr><tr><td>mistralai/mistral-7b-instruct-v0.3</td><td>captions</td><td>0.47</td><td>0.56</td><td>0.55</td><td>0.61</td><td>318.73</td></tr><tr><td>mistralai/mistral-small-24b-instruct</td><td>captions</td><td>0.63</td><td>0.69</td><td>0.68</td><td>0.71</td><td>350.37</td></tr><tr><td>mistralai/mixtral-8x22b-instruct-v0.1</td><td>captions</td><td>0.70</td><td>0.68</td><td>0.74</td><td>0.71</td><td>410.18</td></tr><tr><td>mistralai/mixtral-8x7b-instruct-v0.1</td><td>captions</td><td>0.59</td><td>0.62</td><td>0.64</td><td>0.64</td><td>345.12</td></tr><tr><td>nv-mistralai/mistral-nemo-12b-instruct</td><td>captions</td><td>0.54</td><td>0.55</td><td>0.57</td><td>0.57</td><td>346.16</td></tr><tr><td>nvidia/llama-3.1-nemotron-51b-instruct</td><td>captions</td><td>0.75</td><td>0.75</td><td>0.78</td><td>0.75</td><td>383.77</td></tr><tr><td>nvidia/llama-3.1-nemotron-70b-instruct</td><td>captions</td><td>0.70</td><td>0.70</td><td>0.76</td><td>0.75</td><td>381.48</td></tr><tr><td>nvidia/llama-3.1-nemotron-nano-8b-v1</td><td>captions</td><td>0.27</td><td>0.38</td><td>0.35</td><td>0.44</td><td>531.04</td></tr><tr><td>nvidia/llama-3.1-nemotron-ultra-253b-v1</td><td>captions</td><td>0.68</td><td>0.64</td><td>0.72</td><td>0.68</td><td>535.34</td></tr><tr><td>nvidia/llama-3.3-nemotron-super-49b-v1</td><td>captions</td><td>0.74</td><td>0.73</td><td>0.75</td><td>0.74</td><td>352.49</td></tr><tr><td>nvidia/llama3-chatqa-1.5-70b</td><td>captions</td><td>0.59</td><td>0.62</td><td>0.62</td><td>0.62</td><td>406.58</td></tr><tr><td>nvidia/llama3-chatqa-1.5-8b</td><td>captions</td><td>0.23</td><td>0.44</td><td>0.25</td><td>0.43</td><td>261.50</td></tr><tr><td>nvidia/mistral-nemo-minitron-8b-8k-instruct</td><td>captions</td><td>0.57</td><td>0.56</td><td>0.60</td><td>0.58</td><td>411.22</td></tr><tr><td>nvidia/nemotron-4-340b-instruct</td><td>captions</td><td>0.66</td><td>0.70</td><td>0.72</td><td>0.73</td><td>647.87</td></tr><tr><td>nvidia/nemotron-mini-4b-instruct</td><td>captions</td><td>0.45</td><td>0.45</td><td>0.44</td><td>0.44</td><td>217.31</td></tr><tr><td>nvidia/neva-22b</td><td>image<br>image+captions</td><td>0.45<br>0.32</td><td>0.53<br>0.41</td><td>0.54<br>0.39</td><td>0.59<br>0.46</td><td>462.32<br>470.53</td></tr><tr><td>openai/gpt-3.5-turbo-1106</td><td>captions</td><td>0.54</td><td>0.54</td><td>0.58</td><td>0.58</td><td>254.81</td></tr><tr><td>openai/gpt-4-0613</td><td>captions</td><td>0.67</td><td>0.71</td><td>0.72</td><td>0.73</td><td>453.10</td></tr><tr><td>openai/gpt-4-turbo-2024-04-09</td><td>captions</td><td>0.63</td><td>0.71</td><td>0.68</td><td>0.72</td><td>442.45</td></tr><tr><td>openai/gpt-4.1-2025-04-14</td><td>captions<br>image<br>image+captions</td><td>0.65<br>0.78<br>0.69</td><td>0.69<br>0.79<br>0.74</td><td>0.67<br>0.78<br>0.69</td><td>0.69<br>0.79<br>0.72</td><td>490.55<br>2128.92<br>1404.56</td></tr><tr><td>openai/gpt-4.1-mini-2025-04-14</td><td>image<br>image+captions</td><td>0.60<br>0.73</td><td>0.62<br>0.74</td><td>0.64<br>0.77</td><td>0.64<br>0.76</td><td>663.27<br>644.78</td></tr><tr><td>openai/gpt-4.1-nano-2025-04-14</td><td>image<br>image+captions</td><td>0.06<br>0.47</td><td>0.08<br>0.48</td><td>0.06<br>0.50</td><td>0.10<br>0.51</td><td>715.16<br>733.49</td></tr><tr><td>openai/gpt-4o-2024-08-06</td><td>captions<br>image<br>image+captions</td><td>0.67<br>0.78<br>0.75</td><td>0.71<br>0.77<br>0.79</td><td>0.72<br>0.80<br>0.78</td><td>0.73<br>0.79<br>0.78</td><td>537.11<br>2011.04<br>1214.45</td></tr><tr><td>openai/gpt-4o-mini-2024-07-18</td><td>image<br>image+captions</td><td>0.67<br>0.63</td><td>0.70<br>0.64</td><td>0.70<br>0.67</td><td>0.74<br>0.68</td><td>1245.08<br>901.33</td></tr><tr><td>qwen/qwen2-7b-instruct</td><td>captions</td><td>0.54</td><td>0.57</td><td>0.56</td><td>0.60</td><td>286.59</td></tr><tr><td>qwen/qwen2.5-7b-instruct</td><td>captions</td><td>0.58</td><td>0.63</td><td>0.63</td><td>0.64</td><td>258.65</td></tr><tr><td>qwen/qwen2.5-coder-32b-instruct</td><td>captions</td><td>0.66</td><td>0.69</td><td>0.70</td><td>0.69</td><td>390.61</td></tr><tr><td>qwen/qwen2.5-coder-7b-instruct</td><td>captions</td><td>0.54</td><td>0.55</td><td>0.58</td><td>0.58</td><td>289.67</td></tr><tr><td>qwen/qwo-32b</td><td>captions</td><td>0.02</td><td>0.04</td><td>0.03</td><td>0.04</td><td>8154.97</td></tr><tr><td>rakuten/rakutenai-7b-chat</td><td>captions</td><td>0.11</td><td>0.33</td><td>0.11</td><td>0.29</td><td>299.13</td></tr><tr><td>rakuten/rakutenai-7b-instruct</td><td>captions</td><td>0.51</td><td>0.49</td><td>0.53</td><td>0.48</td><td>350.21</td></tr><tr><td>tiiuae/falcon3-7b-instruct</td><td>captions</td><td>0.52</td><td>0.55</td><td>0.56</td><td>0.59</td><td>224.42</td></tr><tr><td>upstage/solar-10.7b-instruct</td><td>captions</td><td>0.65</td><td>0.67</td><td>0.67</td><td>0.67</td><td>387.34</td></tr><tr><td>writer/palmyra-creative-122b</td><td>captions</td><td>0.72</td><td>0.72</td><td>0.75</td><td>0.73</td><td>504.07</td></tr><tr><td>writer/palmyra-fin-70b-32k</td><td>captions</td><td>0.62</td><td>0.64</td><td>0.66</td><td>0.67</td><td>387.17</td></tr><tr><td>writer/palmyra-med-70b</td><td>captions</td><td>0.69</td><td>0.65</td><td>0.71</td><td>0.68</td><td>386.00</td></tr><tr><td>writer/palmyra-med-70b-32k</td><td>captions</td><td>0.55</td><td>0.60</td><td>0.54</td><td>0.62</td><td>390.38</td></tr><tr><td>zyphra/zamba2-7b-instruct</td><td>captions</td><td>0.42</td><td>0.43</td><td>0.48</td><td>0.49</td><td>509.35</td></tr></tbody></table>
<table><caption>**Table 6: Model Performance by Modality and Voter Type.**
Values in parentheses indicate the 95% upper-bound estimate using bootstrapping. Bolded values are the best models across all other model-modalities; *underlined* values are the second-best.</caption><thead><tr><th rowspan="2">Model</th><th rowspan="2">Modality</th><th colspan="2">Majority Voter</th><th colspan="2">Average Voter</th><th rowspan="2">Avg. Time (s)</th></tr><tr><th>κ (0.69)</th><th>ρ (0.68)</th><th>κ (0.70)</th><th>ρ (0.70)</th></tr></thead><tbody><tr><td rowspan="3">google/gemma-3-27b-it</td><td>captions</td><td>0.53</td><td>0.52</td><td>0.54</td><td>0.53</td><td>22.19</td></tr><tr><td>image</td><td>0.60</td><td>0.60</td><td>0.63</td><td>0.61</td><td>41.02</td></tr><tr><td>image+captions</td><td>0.59</td><td>0.57</td><td>0.63</td><td>0.60</td><td>31.22</td></tr><tr><td>meta/llama-3.3-70b-instruct</td><td>captions</td><td>0.63</td><td>0.57</td><td>0.65</td><td>0.63</td><td>0.38</td></tr><tr><td rowspan="3">openai/gpt-4o-2024-08-06</td><td>captions</td><td>0.53</td><td>0.52</td><td>0.54</td><td>0.54</td><td>0.56</td></tr><tr><td>image</td><td>0.68</td><td>0.65</td><td>0.72</td><td>0.70</td><td>1.98</td></tr><tr><td>image+captions</td><td>0.63</td><td>0.61</td><td>0.66</td><td>0.65</td><td>1.22</td></tr></tbody></table>

## 4 Conclusions

In this study, we have investigated the extent to which state-of-the-art language and vision-language models can serve as reliable proxies for human relevance judgments in a multimodal retrieval setting. By re-annotating a carefully stratified subset of the conQA benchmark with a four-point relevance scale and comparing recent models against aggregated human votes, we arrive at several key takeaways:

**Strong alignment on descriptive queries and visual inputs.**
Both proprietary and open-source models achieve high quadratic-weighted Kappa (up to 0.80) and Spearman's (up to 0.79) scores when judging descriptive queries on raw images, they even surpass in some cases the upper-bound estimates.

**Persistent gaps on conceptual abstraction.**
Across all architectures and modalities, performance on conceptual queries lags substantially: even the best models lose more than 15 points from the descriptive task, this difficulty is also testified amongst humans whose upper-bound estimates drop from 15 to 20 points.

**Modality-specific trade-offs.**
Direct visual inputs consistently outperform caption-only and simple modality fusion prompts, but also incur 2-4x higher latency and token costs. Still, researchers are now more than ever able to experiment with these models as a small scale dataset like conQA could highly benefit academic research teams that often lack the expertise or resources for large crowdsourced annotation campaigns.

Collectively, these findings affirm that (V/)LM-as-a-Judge can dramatically streamline evaluation workflows—particularly for perceptually grounded tasks. Still, we insist that conQA is a dataset using MS-COCO documents against everyday life queries, meaning all our conclusions can only be projected in a similar context. Besides, any abstraction of the query revealed catastrophic for the model's and humans' ability to judge. Other critics insist on the dangers of blindly trusting these models without investigating the use-case and experimental setup [8, 23].

## Future Research Directions

* **Enhanced multimodal fusion.** Investigate adaptive fusion strategies to reconcile conflicting signals between text and image representations, rather than naively concatenating inputs [7, 27].

* **Symbolic and structured knowledge integration.** Augment distributional embeddings with knowledge graphs or concept hierarchies to better capture abstract, counterfactual, and metaphorical relationships inherent in conceptual queries. Knowledge-Augmented Generation is a new trend that expands the conventional Retrieval-Augmented pipelines by using knowledge graphs [14].

* **Calibrated self-evaluation and uncertainty quantification.**
Develop methods for models to self-estimate confidence in their judgments—potentially triggering human review only for low-confidence or highly polarizing cases. [28] investigates the logits of language models to rerank predictions on a continuous scale.

By addressing these avenues, future work can push the boundaries of automatic relevance assessment—bringing us closer to fully automated, scalable, and trustworthy evaluation setups across the ever-expanding landscape of multimodal information retrieval.

## References

[1] Zahra Abbasiantaeb, Chuan Meng, Leif Azzopardi, and Mohammad Aliannejadi. 2024. Can We Use Large Language Models to Fill Relevance Judgment Holes?. In *Joint Proceedings of the 1st Workshop on Evaluation Methodologies, Testbeds and Community for Information Access Research (EMTCIR 2024) and the 1st Workshop on User Modelling in Conversational Information Retrieval (UM-CIR 2024) co-located with the 2nd International ACM SIGIR Conference on Information Retrieval in the Asia Pacific (SIGIR-AP 2024)*, Tokyo, Japan, December 12, 2024 (CEUR Workshop Proceedings, Vol. 3854). Praveen Acharya, Charles L. A. Clarke, Fabio Crestani, Xiao Fu, Gareth J. F. Jones, Noriko Kando, Makoto P. Kato, Aldo Lipani, and Yiqun Liu (Eds.). CEUR-WS.org. https://ceur-ws.org/Vol-3854/emtcir-2.pdf

[2] Ali Al-Saeedi and Aki Harma. 2025. Emergence of symbolic abstraction heads for in-context learning in large language models. In *Proceedings of Bridging Neurons and Symbols for Natural Language Processing and Knowledge Graphs Reasoning @ COLING 2025*, Kang Liu, Yangqiu Song, Zhen Han, Rafet Sifa, Shizhu He, and Yunfei Long (Eds.). ELRA and ICCL, Abu Dhabi, UAE, 86–96. https://aclanthology.org/2025.neusymbridge-1.9/

[3] Marwah Alaofi, Paul Thomas, Falk Scholer, and Mark Sanderson. 2024. LLMs can be Fooled into Labelling a Document as Relevant: best café near me; this paper is perfectly relevant. In *Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (Tokyo, Japan) (SIGIR-AP 2024)*. Association for Computing Machinery, New York, NY, USA, 32–41. doi:10.1145/3673791.3698431

[4] Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, E. Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andr'e F. T. Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Surikuchi, Ece Takmaz, and Alberto Testoni. 2024. LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks. *ArXiv abs/2406.18403* (2024). https://api.semanticscholar.org/CorpusID:270738074

[5] Chris Buckley, Darrin Dimmick, Ian Soboroff, and Ellen Voorhees. 2006. Bias and the limits of pooling. In *Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Seattle, Washington, USA) (SIGIR '06)*. Association for Computing Machinery, New York, NY, USA, 619–620. doi:10.1145/1148170.1148284

[6] Yinqiong Cai, Jiafeng Guo, Yixing Fan, Qingyao Ai, Ruqing Zhang, and Xueqi Cheng. 2022. Hard Negatives or False Negatives: Correcting Pooling Bias in Training Neural Ranking Models. In *Proceedings of the 31st ACM International Conference on Information & Knowledge Management (Atlanta, GA, USA) (CIKM '22)*. Association for Computing Machinery, New York, NY, USA, 118–127. doi:10.1145/3511808.3557343

[7] Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. 2024. MLLM-as-a-Judge: assessing multimodal LLM-as-a-Judge with vision-language benchmark. In *Proceedings of the 41st International Conference on Machine Learning* (Vienna, Austria) (ICML '24). JMLR.org, Article 254, 34 pages.

[8] Charles L. A. Clarke and Laura Dietz. 2024. LLM-based relevance assessment still can't replace human relevance assessment. *ArXiv abs/2412.17156* (2024). https://apisemanticscholar.org/CorpusID:274982001

[9] Cyril W. Cleverdon. 1997. The Cranfield tests on index language devices. https://api semanticscholar org / CorpusID : 59960705

[10] Gabriel de Jesus and Sérgio Sobral Nunes. 2024. Exploring Large Language Models for Relevance Judgments in Tetun. *ArXiv abs/2406.07299* (2024). https://api semanticscholar org / CorpusID : 270379707
[11] José Luis Vicedo González and Jaime Gómez. 2007. TREC: Experiment and evaluation in information retrieval. *J. Assoc. Inf. Sci. Technol.* 58 (2007), 910–911. https://apisemanticscholar.org/CorpusID:33367421

[12] Martin Halvey and Joemon M. Jose. 2012. Bridging the gap between expert and novice users for video search. *Int. J. Multim. Inf. Retr.* 1, 1 (2012), 17–29. doi:10.1007/S13735-012-0007-3

[13] J Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. *Biometrics* 33 1 (1977), 159–74. https://apiidiservices.org/CorpusID:11077516

[14] Lei Liang, Mengshu Sun, Zhengke Gui, Zhongshu Zhu, Zhouyu Jiang, Ling Zhong, Yuan Qu, Peilong Zhao, Zhongpu Bo, Jin Yang, Huaidong Xiong, Lin Yuan, Jun Xu, Zaoyang Wang, Zhiqiang Zhang, Wen Zhang, Huajun Chen, Wenguang Chen, and Jun Zhou. 2024. KAG: Boosting LLMs in Professional Domains via Knowledge Augmented Generation. CoRR abs/2409.13731 (2024). doi:10.48550/ARXIV.2409.13731 arXiv:2409.13731

[15] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. In *Computer Vision – ECCV 2014*, David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (Eds.). Springer International Publishing, Cham, 740–755.

[16] Yiqi Liu, Nafise Moosavi, and Chenghua Lin. 2024. LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scores. In *Findings of the Association for Computational Linguistics: ACL 2024*, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 12688–12701. doi:10.18653/v1/2024.findings-acl.753

[17] Shengjie Ma, Chong Chen, Qi Chu, and Jiaxin Mao. 2024. Leveraging Large Language Models for Relevance Judgments in Legal Case Retrieval. CoRR abs/2403.18405 (2024). doi:10.48550/ARXIV.2403.18405 arXiv:2403.18405

[18] Navid Mehrdad, Hrushikesh Mohapatra, Mossaab Bagdouri, Prijith Chandran, Alessandro Magnani, Xunfan Cai, Ajit Puthenputhussery, Sachin Yadav, Tony Lee, Cheng Xiang Zhai, and Ciya Liao. 2024. Large Language Models for Relevance Judgment in Product Search. ArXiv abs/2406.00247 (2024). https://apiidiservices.org/CorpusID:270210321

[19] Arjun Panickssery, Samuel R. Bowman, and Shi Feng. 2024. LLM Evaluators Recognize and Favor Their Own Generations. ArXiv abs/2404.13076 (2024). https://apiidiservices.org/CorpusID:269293311

[20] Sri Devi Ravana, Parnia Samimi, and Parisa Dabir Ashtyani. 2014. Exploring relevance assessment using crowdsourcing for faceted and ambiguous queries. *2014 Science and Information Conference* (2014), 771–776. https://apiidiservices.org/CorpusID:18352681

[21] Stephen Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Framework: BM25 and Beyond. *Found. Trends Inf. Retr.* 3, 4 (April 2009), 333–389. doi:10.1561/1500000019

[22] Juan Manuel Rodriguez, Nima Tavassoli, Eliezer Levy, Gil Lederman, Dima Sivov, Matteo Lissandrini, and Davide Mottin. 2024. Does the Performance of Text-to-Image Retrieval Models Generalize Beyond Captions-as-a-Query?. In *Advances in Information Retrieval*, Nazli Goharian, Nicola Tonellotto, Yulan He, Aldo Lipani, Graham McDonald, Craig Macdonald, and Iadh Ounis (Eds.). Springer Nature Switzerland, Cham, 161–176.

[23] Ian Soboroff. 2024. Don't Use LLMs to Make Relevance Judgments. Information retrieval research journal 1 (2024). https://apiidiservices.org/CorpusID:272828021

[24] Rikiya Takehi, Ellen M. Voorhees, and Tetsuya Sakai. 2024. LLM-Assisted Relevance Assessments: When Should We Ask LLMs for Help? ArXiv abs/2411.06877 (2024). https://apiidiservices.org/CorpusID:273962929

[25] Shivani Upadhyay, Ehsan Kamalloo, and Jimmy Lin. 2024. LLMs Can Patch Up Missing Relevance Judgments in Evaluation. ArXiv abs/2405.04727 (2024). https://apiidiservices.org/CorpusID:269626105

[26] Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. 2024. Large Language Models are not Fair Evaluators. In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 9440–9450. doi:10.18653/v1/2024.acl-long.511

[27] Jheng-Hong Yang and Jimmy Lin. 2024. Toward Automatic Relevance Judgment using Vision-Language Models for Image-Text Retrieval Evaluation. In *Proceedings of The First Workshop on Large Language Models for Evaluation in Information Retrieval (LLM4Eval 2024) co-located with 10th International Conference on Online Publishing (SIGIR 2024)*, Washington D.C., USA, July 18, 2024 (CEUR Workshop Proceedings, Vol. 3752). Clemencia Siro, Mohammad Aliannejadi, Hossein A. Rahmani, Nick Craswell, Charles L. A. Clarke, Guglielmo Faggioli, Bhaskar Mitra, Paul Thomas, and Emine Yilmaz (Eds.). CEUR-WS.org, 113–123. https://ceur-ws.org/Vol-3752/paper7.pdf

[28] Honglei Zhuang, Zhen Qin, Kai Hui, Junru Wu, Le Yan, Xuanhui Wang, and Michael Bendersky. 2024. Beyond Yes and No: Improving Zero-Shot LLM Rankers via Scoring Fine-Grained Relevance Labels. In *Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)*, Kevin Duh, Helena Gomez, and Steven Bethard (Eds.). Association for Computational Linguistics, Mexico City, Mexico, 358–370. doi:10.18653/v1/2024.naacl-short.31

## A Rejected Models

As of April 24th, 2025, here are the available models on NVIDIA that we could not implement in our study:

* aisingapore/sea-lion-7b-instruct: Inconsistent replies (Trained mostly in South-East Asian languages)

* baichuan-inc/baichuan2-13b-chat: Inconsistent replies (Trained mostly in Mandarin)

* bigcode/starcoder2-7b: Domain mismatch (Trained on code data)

* bigcode/starcoder2-15b: Domain mismatch (Trained on code data)

* deepseek-ai/deepseek-r1-distill-llama-8b: No response (API returned no response Apr 10–15, 2025)

* google/codegemma-7b: Inconsistent replies (Trained for code generation)

* google/shieldgemma-9b: Domain mismatch (Content-safety classifier)

* ibm/granite-guardian-3.0-8b: Domain mismatch (Content-safety classifier)

* institute-of-science-tokyo/llama-3.1-swallow-70b-instruct-v0.1: Inconsistent replies (Trained mostly in Japanese)

* institute-of-science-tokyo/llama-3.1-swallow-8b-instruct-v0.1: Inconsistent replies (Trained mostly in Japanese)

* meta/codelama-70b: No response (404 error; region-locked or deprecated)

* meta/llama2-70b: No response (404 error; region-locked or deprecated)

* meta/llama3-8b: No response (404 error; region-locked or deprecated)

* meta/llama3-70b: No response (404 error; region-locked or deprecated)

* microsoft/phi-3-medium-4k-instruct: Inconsistent replies

* mistralai/codestral-22b-instruct-v0.1: No response (404 error; region-locked or deprecated)

* mistralai/mistral-2-large-instruct: No response (404 error; API ID differs from docs)

* mistralai/mathstral-7b-v01: No response (404 error; API ID differs from docs)

* mistralai/mistral-large: No response (404 error; region-locked or deprecated)

* nvidia/llama-3.1-nemoguard-8b-content-safety: Domain mismatch (Content-safety classifier)

* nvidia/llama-3.1-nemoguard-8b-topic-control: Domain mismatch (On/off-topic classifier)

* nvidia/llama-3.1-nemotron-70b-reward: Domain mismatch (Reward model)

* nvidia/mistral-nemo-minotron-8b-base: No response (404 error; region-locked or deprecated)

* nvidia/nemoguard-jailbreak-detect: Domain mismatch (Jailbreak detector)

* nvidia/nemotron-4-340b-reward: Domain mismatch (Reward model)
* nvidia/nemotron-4-mini-hindi-4b-instruct: Inconsistent replies (Trained mostly in Hindi)

* nvidia/usdcode: Domain mismatch (USD file-extension generator)

* nvidia/usdsearch: Domain mismatch (USD file-extension search)

* seallms/seallm-7b-v2.5: Inconsistent replies (Trained mostly in South-East Asian languages)

* snowflake/arctic: No response (404 error; region-locked or deprecated)

* tokyotech-llm/llama-3-swallow-70b-instruct-v01: Inconsistent replies (Trained mostly in Japanese)

* yentinglin/llama-3-taiwan-70b-instruct: Inconsistent replies (Trained mostly in Mandarin)

* black-forest-labs/flux.1-dev: Domain mismatch (Text-to-image generation)

* briaai/bria-2.3: Domain mismatch (Text-to-image generation)

* hive/ai-generated-image-detection: Domain mismatch (Image classification)

* hive/deepfake-image-detection: Domain mismatch (Image classification)

* meta/sam2: Domain mismatch (Image segmentation)

* nvidia/bevformer: Domain mismatch (Image classification)

* nvidia/consistory: Domain mismatch (Text-to-image generation)

* nvidia/cosmos-1.0-7b-diffusion-text2world: Domain mismatch (Text/image-to-video generation)

* nvidia/nemoretriever-parse: Domain mismatch (Image-to-text extraction)

* nvidia/nv-dinov2: Domain mismatch (Image classification)

* nvidia/nv-grounding-dino: Domain mismatch (Object detection)

* nvidia/ocdrnet: Domain mismatch (OCR)

* nvidia/retail-object-detection: Domain mismatch (Object detection)

* nvidia/sparsedrive: Domain mismatch (Object motion modeling)

* nvidia/vila: Domain mismatch (Image captioning)

* nvidia/visual-changenet: Domain mismatch (Image segmentation)

* stabilityai/stable-diffusion-3-medium: Domain mismatch (Text-to-image generation)

* stabilityai/sdx1-turbo: Domain mismatch (Text-to-image generation)

* stabilityai/stable-diffusion-x1: Domain mismatch (Text-to-image generation)

* stabilityai/stable-video-diffusion: Domain mismatch (Text-to-video generation)

* google/deplot: Domain mismatch (Image-to-figure generation)

* microsoft/florence-2: Domain mismatch (Multi-task, not our use-case)

* microsoft/kosmos-2: Inconsistent output format

Received May 5th, 2025; revised June 1st, 2025; accepted June 17th, 2025