Completing Missing Annotation: Multi-Agent Debate for Accurate and Scalable Relevant Assessment for IR Benchmarks
Authors: Minjeong Ban
Jeonghwan Choi
Hyangsuk Min
Nicole Hee-Yeon Kim
Minseok Kim
Jae-Gil Lee
Hwanjun Song
Venue: ICLR 2026 Poster (2026)
Keywords: Information Retrieval
Relevant Assessment
Benchmark
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: A debate-based relevance assessment framework with multiple agents that yields more accurate labeling for certain cases and more reliable AI-to-human escalation for uncertain ones.

Abstract: Information retrieval (IR) evaluation remains challenging due to incomplete IR benchmark datasets that contain unlabeled relevant chunks. While LLMs and LLM-human hybrid strategies reduce costly human effort, they remain prone to LLM overconfidence and ineffective AI-to-human escalation. To address this, we propose DREAM, a multi-round debate-based relevance assessment framework with LLM agents, built on opposing initial stances and iterative reciprocal critique. Through our agreement-based debate, it yields more accurate labeling for certain cases and more reliable AI-to-human escalation for uncertain ones, achieving 95.2% labeling accuracy with only 3.5% human involvement. Using DREAM, we build BRIDGE, a refined benchmark that mitigates evaluation bias and enables fairer retriever comparison by uncovering 29,824 missing relevant chunks. We then re-benchmark IR systems and extend evaluation to RAG, showing that unaddressed holes not only distort retriever rankings but also drive retrieval–generation misalignment. Code and data will be released upon acceptance.

Published as a conference paper at ICLR 2026
COMPLETING MISSING ANNOTATION: MULTI-AGENT DEBATE FOR ACCURATE AND SCALABLE RELEVANCE ASSESSMENT FOR IR BENCHMARKS
Minjeong Ban; Jeonghwan Choi; Hyangsuk Min; Nicole Yeon Kim, Minscok Kim, Jae-Gill Lee, Hwanjun Song!
Korea Advanced Institute of Science and Technology (minjeong.barl ban, hwani.choi, hyangsuk. min, songhwanjun] @ kaisttac.kr
ABSTRACT
Information retrieval (IR) evaluation remains challenging due to incomplete IR benchmark datasets that contain unlabeled relevant chunks. While LLMs and LLM-human hybrid strategies reduce costly human effort, they remain prone LLM overconfidence and ineffective Al-to-human escalation. To address this, we propose DREAM, multi-round debate-based relevance assessment framework with LLM agents, built on opposing initial stances and iterative reciprocal cri- tique. Through our agreement- based debate, it yields more accurate labeling for certain cases and more reliable Al-to-human escalation for uncertain ones, achiev- ing 95.2% labeling accuracy with only 3.5% human involvement. Using DREAM, we build BRIDGE, refined benchmark that mitigates evaluation bias and enables fairer retriever comparison by uncovering 29,824 missing relevant chunks. We then be-benchmark IR systems and extend evaluation to RAG, showing that unad- dressed holes not only distort retriever rankings but also drive retrieval gennrration misalignment. The relevance assessment framework is available [ht.ts:] //gi // ICRR- and the BRIDGE dataset is available
con LaabBRIIDE Benchmark
INTRODUCTION
Information retrieval (IR) systems aim to retrieve relevant text chunks from large document collec- tions in response to user queries, which is a critical task for many downstream applications (Jiang al. 2024 Yang al. 2024 Liu al. 2025). However, IR evaluation remains challenging primar- ily due to the limitations of IR benchmark datasets, which rely heavily on costly human annotations of query-chunk relevance. Since only small subset of text chunks can be labeled, large portions of the corpus remain unjudged, leaving unlabeled yet potentially relevant chunks, so-called "holes" in relevance assessment, which result in potentially misleading evaluation outcomes (Craswell et al., 2020 MacAvaney & Soldaini] 2023 Rassin et al. (024). These holes undermine the reliability of IR benchmarks by obscuring the true effectiveness of retrieval methods, and the problem becomes even more critical in RAG systems, where unreliable retrieval evaluation hinders understanding and optimization of the retriever-generate
interaction (Es et al. 2024, Park et al. 2025).
With the emergence of large language models (LLMs), several efforts have emerged to leverage them to reduce the human cost of constructing IR benchmarks. Recent studies have either fully automated the relevance assessment process (Shivani et al. 2024 Rassin et al. 2024 Park al., 2025 Rahmani ct al. 2025) Ni et al. 2025), or adopted a hybrid strategy in which an LLM handles straightforward cases while humans
engaged only for uncertain instances when the LLM's confidence score is low (Takehi et al. 2025 Li et al. 2022). However, both approaches have inherent limitations due to their strong reliance on single agent serving as the sole judge.
Fully automated pipelines risk propagating the overconfidence of a single model, which often leads large number of mislabeled cases (Tian et al. 2025 Sun al. 2025). In contrast, confidence-
"Equal contribution.
1 "Coresponning Author.

---

Published as a conference paper at ICLR 2026
Mutti-Agent Debate
Result
Relevance Labeling
Agreement
Auturatel Aumobaiion
Opposing
Gold
Incomplete Dutaset
Sparso
Nes-Golk
BRIDGE
Query
Query
Gold Anower(s)
Mulli-Round Dehate
Inclevanee
Amsotation
Gold Answer(s)
Gold Chunkis)
Unaklee Caeilt
Gold Chunk(s)
Heraeking
Dat ds Hokey
Disagroment
Besssingg
Diverse Retrievers
Overview of DREAM pipeline for constructing the BRIDGE benchmark. Multi-agent de- bate with opposing stances conducts a multi-round process, automatically annotating agreement cases and escalating disagreements to humans with debate history.
Figure/ 1:
based escalation attempts to mitigate failures by sending uncertain cases to human annotators. Yet this strategy has notable limitations: the resulting labels are less accurate than direct human anno- tation, even for cases the LLM deems certain, and its miscalibrated confidence scores often cause errors by unnecessarily escalating straightforward cases and failing to escalate genuinely ambigu- ous ones (Kumar et al. 2024]] Chhikara 2025). Consequently, the overall process remains both expensive and error-prone, limiting its effectiveness as a reliable alternative to human annotation.
Moving beyond the constraints of prior single-agent approaches, we suggest multi-agent paradigm (in Figure [ that harnesses LLMs' true strengths, including diverse reasoning, mutual critique, and collective deliberation. Concretely, we introduce DREAM (Debate based RElevance Asses sment with Multi-agentss) a framework that employs multi-ager adversarial de- bate to mitigate holes in IR benchmarks with higher accuracy and lower human cost. Specifically, initialize two agents with opposing stances, one assuming "relevance" " and the other "irrelevance." This design compels agents surface and challenge conflicting evidence, thereby avoiding pre- mature consensus and mitigating single-perspective bias (Koupaee et al.] [025) Oriol et al.] (2025). They then engage in multi-round reciprocal critique, refining evidence and challenging arguments until they either converge to agreement or maintain disagreement.
Unlike recent confidence-basel methods (Takehi et al., 2025 Li al. 2025). our approach lever- ages inter-agent agreement as a direct indicator of rebibility. That is, labels are accepted upon agent agreement and escalated when they disagree, avoiding calibration training or threshold tun- ing. Agreement provides stronger signal of correctness than single model's 's often miscalibrated confidence, while disagreement naturally pinpoints the uncertain cases that require human review. Furthermore, another key distinction of DREAM is its use of the debate history as a consistent mecha- nism across both the multi-agent debate and human annotation stages. During the debate, the history enables agents to reach agreement more efficiently (within two rounds), and when escalation occurs, the same history is provided to humans as a supportive resource, directly assisting them by present- ing organized arguments and evidence rather than leaving them to start from seratch. Through this principled and unified design, DREAM achieves a relevance labeling accuracy of 95.2%, surpassing full-scale non-expert labeling while involving humans in only 3.5% of cases.
To validate and showcase our framework, we construct a new BRIDGE benchmark dataset. Instead of creating yet another IR benchmark from scratch, we re-work widely used two IR benchmarks, namely BEIR (Thakur et al. 2021) and Robust QA (Han et al.] 2023). By applying our DREAM framework, we systematically identify and fill missing relevant chunks (holes), thereby reducing evaluation bias and producing refined benchmark that supports fairer comparisons across retrieval systems as well as more faithful assessment of downstream RAG performance. Specifically, in RAG, improvements in retrieval performance do not fully translate into gains in generation performance. The common explanation that external knowledge often conflicts with the model's internal para- metric knowledge (Longpre et al. [202 Chen et al. 2222 Zhou et al. 2023). While partly valid, we identify another overlooked factor: retrieval performance itself has been misestimated due to holes in IR benchmark datasets (see Section 5.
Our main contributions are: (i) we introduce DREAM, a debate- based relevance labeling framework that achieves high accuracy while keeping human effort and cost minimal; (ii) construct the BRIDGE benchmark by re-labeling BEIR and RobustQA, uncovering 29,824 of missing relevant chunks, equivalent to 428% of the originally annotated 6,976 gold chunks; (iii) we analyze how
2

---

Published as a conference paper at ICLR 2026
holes in existing IR benchmarks distort retrieval performance and rankings, and show that DREAM mitigates this bias; and (iv) we present a new insight into the cause setrieall-enneraie perfor- mance misalignment and address it through our framework.
RELATED WORK
which evaluates IR systems using documents, queries, and human relevance assessments, 18 seminal work in relevance assess- ment. The emergence LLMs has demonstrated strong capabilities for cost-effectively automating data annotation (de Jesus & Nunes, 2024; Thomas et al.) 2024; Upadhyay et al. |025], replacing
Relevance Assessment. The Cranfield experiment (Cleverdon) (.97).
expensive human annotation. UMBRELA (Shivani et al. 2024). D-MERIT (Rassin et al.) 2024),
MIRAGE (Park et al. 2025), and SynDL (Rahmani et al. 2025) have shown that LLMs can
sess document relevance as well as humans, while DIRAS (Ni et al. (025) has demonstrated that fine-tuning smaller LLMs can achieve performance on par with larger models. However, evaluating relevance solely based on LLMs presents limitations, including overconfidence biases and a lack of
nuanced contextual understanding (Faggioli et al. 2023 Clarke & Dietz 2025 Soboroff 2025).
Selective Approaches to Relevance Assessment. Recent work on selective prediction /Varshney ct al., 2022; Stengel-Eskinn & Durme 2023] Srinivasan et al.] (2024) shows that models can reduce unreliable decisions.
errors by abstaining escalating uncertain cases rather than committing
Building on this idea, Al-human hybrid methods (Xu et al. 2025 Sahitaj et al. 2025), which lever- age LLMs for straightforward cases and rely on human intervention for uncertain ones, have been proposed to address inaccuracies in fully automated labeling, such as collaborative annotation (Kim) cet al., 2024) and verification pipelines (Wang et al. 2024). Yet, there has been limited research on their application to relevance assessment. Beyond the confidence- based method, LARA (Takehi al. 2025) calibrates LLM confidence into relevance probability using labeled data to mit- igate miscalibration and overconfidence. Despite these advances, hybrid pipelines remain limited by single-model accuracy and a heavy reliance on human data. contrast, our approach mitigates through opposing opinions and reciprocal critique in multi-round debate, eliminating the need for calibration and the reliance on additional training with human supervision.
Multi-agent Debate. Over a single-agent setup, recent studies have explored utilizing multiple agents with distinct roles that collaborate (Chang et al. [2024] [hen et al. (024). In particular, multi- agent debate has been widely examined, where agents share their answers and rationales, critique one another, and iteratively refine their outputs over multiple rounds (Xiong et al. [2023 Liang al.) 2024a). Such debate-style collaboration has been applied to improve LLM and RAG answer quality (Du et al. 2023 Chen et al. [2024; Khan al. 2024), to handle knowledge confficts in RAG (Wang ct al. (025). ). and to enhance response evaluation (Chan et al. 2023) and data annotation (Tseng et al,, instead leverage multi- agent dehate on Al-Human collaboration for IR relevance assessment. Our framework uses multi- agent debate not only to produce high-quality annotations, but also determine when to escalate uncertain cases to humans and to improve human judgments through providing debate histories.
2025 While existing work focuses primarily AI-AT collaboration,
Connecting IR and RAG. Generation serves as the key downstream task of retrieval, making RAG natural framework that tightly couples retriever quality (Gao et al. (023b). Recent studies have shown that holes in IR benchmarks also undermine the reliable evaluation of RAG, as miss- ing relevant chunks create discrepancy between retrieval and generation performance (Park al.) 2025 Ni et al. (025).. Yet, most RAG benchmark studies continue to focus primarily on joint eval- uation of retrieval and generation (Tang Yang. 2024 Friel et al.) [2024] Krishna et al. 2025), with limited attention to holes; even benchmarks such as DIRAS (Ni et al. (025) and MIRAGE (Park) al. 2025) rely on fully automated --asedd labeling, leaving annotation reliability unverified.
AUTOMATED RELEVANCE LABELING FOR IR
We formulate automated relevance labeling as an answer-awar relevance judgment problem, where the task is to decide if candidate chunk supports the answer to given query q. Formally, given target query q E Q, let A(g) denote the set of possible answers associated with the query q, and C(g) denote the set of candidate chunks considered for the query q. The task is to assign relevance label
3

---

Published as a conference paper at ICLR 2026
for each tuple (g.c) c) where 9 Q and C(g). Here, a query-chunk pair is labeled (relevance) it provides evidential support for at least one answer in A(q), and 0 (irrelevance) otherwise:
1, Ea A(4) such that supports a,
f (g.0)) y V=(0, 00 otherwise.
Our goal is to design a labeling function f together with routing policy, such that (i) certain cases are resolved automatically by f. and (ii) genuinely uncertain cases are escalated humans. The objective is to achieve high labeling accuracy while minimizing the need for human verification.
3.1 DREAM: DEBATE-BASED RELEVANCE ASSESSMENT WITH MULTI-AGENTS
Based on the formulation, we design a labeling function I that leverages inter-agent agreement to automate relevance labeling. To realize this principle, we introduce DREAM in Figure a framework that employs multi-round debate between two LLM agents to surface inter-agent (dis)aagremmet and escalate only genuinely difficult cases. The framework is structured into the following three stages:
Opposing Stance Initialization. We begin by initializing LLM agents with opposing stances to enforce explicit divergence of perspectives. Formally, two LLM agents m, {m1, m2) are instantiated with opposing stances 8; E {81,82), where 81 is defined as relevance ("I think the chunk is relevant the target query.") and 82 is irrelevance ("I think the chunk is not relevant to the target query."). At initialization, m is assigned $1, while m2 is assigned both perspectives are examined. This forced divergence compels each agent defend its stance, preventing premature consensus and mitigates single-perspective bias, which also improves factual grounding and deliberative accuracy (Liang et al., 2024b, Koupace al., 2025). Importantly, we observe no order dependence in the initial stance assignment even when swapping the initial stances of my and m2, Please refer to Appendix B. for details.
ensuring that
Muuti-Roond Debating with Reciprocal Critique. Building on opposing stances, DREAM em- ploys a multi-round debate that compels agents to repeatedly confront and critique opposing perspec- tives, while also allowing them to revise their labeling decisions. During the debate process, each agent comprehensively reviews both its own and the opponent's 's arguments from previous rounds to generate new reasoning. In each debate round j, the two agents m, provided query-chunk pair, its answer set, and the debate history from the previous round h as input. Based on this input, each agent critiques the opponent's s arguments, extracts evidence sentences from the chunk, and produces new relevance label ya with its supporting reasoning -
Specifically, for the first round (j = 1), the discussion history is limited to the initial stance h-1 82), while in subsequent rounds (j > 1). the complete discussion history from the previous round h = ((1-1 ""-"", (r2) ( utilized. After each round concludes, we evaluate inter- agent consensus by verifying whether y4 = vz. If consensus is reached in any round, the debate terminates with the agreed label, which is then used as the final relevance label; otherwise proceeds to the next round. The debate continues until either consensus is achieved between the two agents or a predefined maximum number of rounds, Rt reached. If the consensus is not ultimately reached, this is considered a genuinely uncertain case where the complexity of the relevance assessment exceeds the capability of multi-round debate. Such cases are escalated to human verification. As result, our designed labeling function is formulated as:
vi:
j S s.t. of vi (agreement reached),
DREAM(y. c)
Human (g, c, h), otherwise (persistent disagreement).
Through agent consensus, trivial cases are handled automatically, leaving only uncertain ones for hu- man verification, thereby lowering annotation cost while maintaining accuracy. Refer to the prompts used for the multi-round debate in Appendix A.I
bgreement-based Human Escalation. Persistent disagreement after the maximum rounds R is considered uncertain, case where agents with opposing stances fail to converge. This criterion captures epistemic uncertainty beyond confidence scores, preventing overconfident errors by a single agent and eliminating the need for manually tuned escalation thresholds.
4

---

Published as a conference paper at ICLR 2026
When case is escalated, annotators receive the query-chunk pair and its answer set, along with the debate history, capturing both agents' reasoning and evidence in the final round, as in Eq. 2 This context helps annotators understand the specific sources of disagreement and the nuanced aspects that make the case challenging, leading to more informed human judgmeet. Thus, DREAM maintains high accuracy through principled escalation and enhances human judgment by turning escalation into synergy rather than mere fallback, with quantitative gains shown in Section 3.2.4
3.2 COMPARISON WITH EXISTING RELEVANCE LABELING METHODS
We verify the effectiveness of DREAM compared with alternative auto-labeling methods.
Evaluation Set. We sample 700 query-answer pairs at random, with 100 pairs from each subset: MS MARCO and NQ from BETR (Thakur et al. 202T). p. five domains (Lifestyle, Recreation, Science, Technology, and Writing) from Robust (Han et al., (023). For each query q. a candidate chunk is randomly sampled from candidate set C(q) retrieved using 25 different retrievals. As a result, our evaluation set contains 700 queries, where each query is associated with its answer set A(g) and the candidate chunk c. This corresponds the input in Eq. ( of our problem formulation. Note that the evaluation set is a subset of our refined IR benchmark, detailed in Section 4.1
For this subset, we further collect the ground-trute relevance labels (1: relevance, O: irrelevance), which are adjudicated by expert human annotators, consisting of graduate-level NI.P specialists with complementary academic backgrounds. Refer to Appendix A.. for details.
Metrics. We measure auto-labeling accuracy on non-escalated cases by comparing the estimated labels with the expert ground-truth. truth, and we also report the escalation ratio to humans. A high value of the former indicates reliable automatic judgments, while a low value of the latter reflects reduced annotation cost and efficient use of human effort. We adopt balanced accuracy (bAcc) as our metric for labeling accuracy. For completeness, we report class-wise recall (relevance and irrelevance) and their average as bAcc, which provides more robust evaluation than standard accuracy, especially in relevance assessment tasks where irrelevant cases naturally outnumber relevant ones. The escalation ratio denotes the fraction of cases each method deems uncertain and escalates to humans.
Baselines. We compare DREAM with two agent-based methods and reference method that lies entirely on non-expert annotators recruited via Amazon Mechanical Turk (MTurk). The com- pared methods include: LLMJudge (Shivani et al. 2024 Park et al. 2025), single-agent method that directly assigns relevance labels without escalation, widely used for automated labeling: LARA (Taeehi et al. 2025), confidence-basee escalation method that leverages token probabilities to tomatically assign labels and route uncertain cases to humans; and Human-On ly, a method that assigns each case to three workers MTurk, with the final label determined by majority vote (see Section 4.I. For LARA, it requires specifying an escalation ratio, so we adjust it with values in (3.5%, 12, 25.0%, 50.0%). as they represent different levels of human involvement. The 3.5% setting is aligned with DREAM's escalation ratio to ensure fair comparison.
Implementation. We use 1lama3.3-70B-Instti (with a temperature of 0.0) as the LLM agent for all methods, except for Human Onny.. Thus, DREAM employs two Llama3.3 models to conduct multi-round debates. The maximum debate rounds R is set to 2 by default, as larger values do not produce meaningful gains (see Section 3.2.2 Details including prompts are in Appendix A.3
Regarding LARA, it mitigates miscalibration of LLM token probabilities through online training of logistic regression model, where uncertain (escalated) cases are annotated and used for online training. We apply this setup to our data following the original study (Takchi et al. 2025).
3.2.1
MAIN RESULT: ANNOTATION ACCURACY & ESCALATION RATTO
Figure 2 and Table compare the annotation accuracy and escalation ratio of three automatic relevance assessment methods, evaluated on their non-escalated cases, along with one non-expert human-based method Human-Only which is not fair comparison but is provided as reference point. Although accuracy on non con-easaltted cases can be trivially increased by escalating most cases, the key challenge is achieving high accuracy while keeping escalation ratio low.
5

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of DREAM and two auto- matic labeling methods on non-escalated cases.
LLMinige IARA
Human Only
DREAM (Ouss
Ba Balanced Accuracy
50.2%
0.9
140.%%
Method
Class-wise Recall
bAce
Escalation Ratio
Irreevance Relevance
0.8
Human Only
89.9%
97.8%
93.8%
100.0%
0.5
LLMJudge
50.2%
97.5%
73.9%
0.0%
0.7
0
25
50
75
100
74.5%
89.6%
82.1%
3.5%
Escalation Ratio (%)
91.6%
83.9%
12.5%
Figure 2: Auto-labeling accuracy and escalation ratio of DREAM compared with two LLMJudge and LARA on non-escalated cases.
LARA
80.2%
95.3%
87.8%
25.0%
94.1%
98.4%
96.3%
50.0%
DREAM
91.9%
98.4%
95.2%
3.5%
The results reveal that DREAM attains automatic relevance judgment performance on par with hu- mans, as evident from its balanced accuracy of 95.2%, surpassing the 93.8% by Human Human-Only, while requiring escalation in only 3.5% of the cases. The single-agent baseline, LLMJudge, suf- fers from low recall on irrelevant cases, yielding a much lower balanced accuracy of 73.9% due to the absence of uncertainty- confidence-bart based escalation, achieving higher balanced accuracy with increasing escalation ratio. That is, the confidence- strategy is partly effective: difficult cases escalated to humans while the agent handles higher-confidence ones, though misclassifications still occur.
escalation. Contrarily, LARA outperforms LLMJudge through its
Yet, this escalation proves be sub-optimal. When restricted the same 3.5% escalation ratio as DREAM, LARA falls notably short in annotation quality. To reach performance comparable to DREAM, it requires escalating nearly 50.0% of the cases, which demands extensive human involve- ment and thus lacks scalability. That is, uncertainty estimation in LARA is inaccurate, missing difficult cases while over-escalating easy ones. In contrast, our agreement-based approach with multi-agent dehate enables far more precise escalation, achieving higher annotation quality while automatically labeling substantially more cases than LARA with 50% escalation. In particular, un- like LARA, DREAM removes the need for calibration training and manual escalation tuning, making it more practical and scalable for automated relevance assessment.
In addition, DREAM is cost-effective in terms of labeling cost and latency when compared with Numan-Only: We conduct a comprehensive cost analysis in Appendix B.22 which confirms that DREAM achieves 200 cheaper and 3.5x-7.0x faster than Human-Only.
3.2.2 ABLATION STUDY: DEBATE ROUND AND ADJUDICATION STRATEGY
We conduct an ablation study with DREAM on its two design choices: (i) the number of debate rounds and (ii) the escalation strategy, in which uncertain cases are routed either to human an- notators to LLM adjudicator, the latter abling full automation. Regarding (ii), comparing human annotators (via Mrurk) with L.M adju- dicator as the final judge is important, as it shows the synergy of Al-human collaboration, which leads to higher annotation quality than fully au- tomated adjudication. See Appendix A.4 for the
Table 2:
Overall annotation quality with DREAM
for all cases, where escalated cases adjudi- cated by either an LLM or humans, and all other cases are labeled automatically by DREAM.
Method
Adju- dicator
Class-wise Recall Inrelevance Relevance
bAce
DREAM (R = 1)
LLM
82.9%
97.2%
90.0%
LLM LLM
90.0%
96.7%
93.3%
DREAM (Il 3)
90.8%
95.7%
93.2%
DREAM (R 2)
Human
91.8%
98.4%
95.1%
detailed adjudication setup. Table 2 presents the annotation quality across these settings, illustrating the effect of increasing debate rounds and using different adjudication on overall performance.
First, we increase the maximum round R from 1 to 3 with I.LM adjudication, as this setup removes human inconsistency and clearly reveals the impact of debate depth. The results show that perfor- mance gains saturate after two rounds, with little improvement beyond. Thus, set R. = 2 as the default. Sccond, human adjudication outperforms LLM adjudication with the same maximum debate round. That is, humans resolve uncertain cases more accurately, thereby supporting our Al-to-human escalation strategy to maximize annotation quality.
We further examine agents' consensus stability across multiple debate rounds, understanding how spurious consensus emerges or diminishes is crucial for evaluating the reliability of multi- agent annotations. In this context, also investigate whether increasing the number of debate

---

Published as a conference paper at ICLR 2026
rounds helps reduce spurious agreements and thereby improves the reliability of DREAM's auto- labeling. The experiment in Appendix revells that increasing the number of debate rounds does not consistently reduce the spurious consensus ratio.
3.2.3 ABLATION STUDY: DIVERSE DEBATE CONFIGURATIONS
of interest to examine how increased diversity in debate influences the auto-labeling perfor- mance of DREAM. Hence, we conduct an ablation study examining three factors: (i) increasing the number of agents, (ii) using heterogeneous LLM families, and (iii) adjusting temperature values. For the sake of space, we defer the details AppendixB.4 To summarize, increasing diversity in the debating setup does not yield improved annotation performance. First, increasing the number of agents reduces the likelihood of reaching agreement on relevant cases, which lowers relevance recall and slightly decreases overall annotation accuracy (See Appendix [B.4.]. Second, although heterogeneous model families alleviate model-specific bias, the overall accuracy remains lower due to performance disparities across model types (See Appendix [B.4.2.. Third, higher temperature settings yield longer rationales, which cause the deliberation to drift toward irrelevant decision and ultimately degrade annotation quality (See Appendix B.4.3).
3.2.4 ABLATION STUDY: DEBATE HISTORY AS SUPORTIVV RESOURCE
Table shows the contribution of debate history to human annotation in uncertain escalated cases, con- trasting annotation quality with and without it as a supportive resource. Each case is annotated by three MTurk crowdworkers with majority voting. The re- sults demonstrate that debate history raises IAA, re- flecting greater consistency even in difficult cases, and therefore boosts labeling accuracy (bAcc). This highlights unique advantage of DREAM over confidence- basedd methods, demonstrating genuine Al-human synergy.
Table 3: Comparison of human annotation quality with and without debate history.
Method
1AA Fleiss
Class-wise Recall
bAce
Irrllvance Relevance
History History
0.50
84.9% 89.3%
89.6% 94.6%
87.3%
0.62
92.0%
4 REAL-WORLD APPLICATIONS TO IR BENCHMARK
To demonstrate the utility of our framework. we refine existing IR benchmarks using DREAM (Sec- tion 4.I). Then, we identify how holes in existing IR benchmarks introduce evaluation bias and demonstrate that our approach effectively mitigates this issue (Sections 4.2
4.1 BRIDGE: IMPROVED IR BENCHMARK
A critical issue in IR benchmarks is the presence of holes (Craswell al. 2020 Rassin al. 2024). where truly relevant ("gold") chunks are missing from the labeled evaluation set and thus misclas- sified as irrelevant ("non-gou "'non-gold"). This mislabeling arises because the evaluation set usually built from candidate chunk pool-the union of top-ranked results of few selected retrieval systems- primarily reduce the human annotation cost (MacAvaney & Soldaini] 2023) Rassin et al., 2024). Such holes therefore cause evaluation bias: the performance of retrieval models outside the pool are often underestimated, especially when the pool relies on limited or outdated retrieval systems.
To fill the labeling gap, we refine existing IR benchmarks with DREAM by adding missing relevance labels through large-scale yet low-cost labeling. This process yields 116,622 relevance labels in total, of which 29,824 idenrified "relevant (holes)," at human cost of only about $506. This adds 428% more to the 6,976 originally annotated gold chunks, for a combined total of 36,800.
Target Benchmark. We refine two widely used IR benchmarking resources to address their inher- ent biases and improve evaluation reliability. In detail, we select seven IR benchmark subsets: MS MARCO (MS) and NQ from BEIR (Takur et al., (02T): and Lifestyle (Lifee, Recreation (Rec), Science (Sci), Technology (Tech), and Writing (Writ) from ROBUSTQA (Han et al.) (023). These datasets span diverse domains and corpora of varying sizes and styles, providing a broad and chal- lenging basis for retrieval evaluation. In making this selection, we also prioritize IR benchmarks with query-au annwer pairs, as this structure enables the construction of qquer--chunk-aae answer triplets, which in turn facilitates examining how improvements in retrieval evaluation translate to more reliable sessment of generation in RAG. We sample a total of 3,657 queries from the test sets of the seven subsets, with 550 queries from each dataset except for Science, which contains 357 valid queries.

---

Published as a conference paper at ICLR 2026
Table 4: Hole@ 10 ratios across 25 retrieval systems on the target benchmark prior to refinement with DREAM. Systems with higher percentage of holes than average are marked in bold. A detailed breakdown of the results for each of the seven subsets in Appendix G
Sparse
Dense
BM25
Splade
Contriever
DPR
DimRed
SBERT
Faiss
ANCE
COlBERT
Aggretriever
Arctic
14.6%
19.0%
6.5%
9.5%
11.5%
13.0%
14.5%
15.9%
16.2%
16.6%
17.5%
20.8% %
BM25 Rerank
ANCE Rerank
BM25 Rewrile
TBERT
NBoost
MLM
ELEC
M15
TBERT
NBoost
MLM
ELEC
MI5
HyDE
Q2d
MuG1
17.3%
18.5%
18.5%
18.6%
20.0%
17.7%
18.8%
18.8%
18.8%
20.2%
21.2%
21.3%
22.3%
Abbreviation: Dimited-Dim Reduction, ColBERT-
TBERT- FayHERT, MiM-M ElDCHELECTRA MI3-MontT,, Qud=tue
The collected queries, together with the augmented relevance labels using DREAM (described in detail below), constitute our refined benchmark, which we name BRIDGE (see Appendix CC).
Candidate Pool Construction. To fill the holes in the selected subsets, we construct a pool of 25 retrieval systems, combining both traditional retrievers and more recent ones introduced after the release of the target benchmarks, including sparse (e.g., SPLADE (Formal et al. 2022),, ), dense (e.g., Aretic YYu et al. 2)24)), and their combinations with advanced fusion techniques such as re-ranking (e.g.,
(Li et al.] 2024)) and query rewriting (e.g., MuGI (Zhang et al.] )202a)); as summarized in Appendix D Then, we combine the top-10 retrieved chunks from each of the 25 retrieval systems for selected 3,657 queries, followed by applying simple LLM-bbsed filter to remove typical irrelevant cases (see Appendix (E). This yields 116,622
wanwwr--chume triplets (out of 296,053 initially retrieved), which constitute the candidate set for subsequent assessment.
Relevance Assessment. Each triplet in the candidate pool is considered as input to DREAM, under the same implementation described in Section 3.2 Through this process, 112,566 out of 116,622 cases reached consensus within two rounds of debate and were therefore automatically labeled with high certainty, while only the remaining 4,056 cases of disagreement required human evaluation. Hence, we perform human annotation using MTTurk. Each disagreement case is evaluated by three qualified annotators, and the final label is decided based on the majority vote of their judgments. In our human annotation, the inter-annotator agreement (IAA) measured by Fleiss' Kappa is 0.62, indicating substantial agreement. The recruitment and procedures are detailed in Appendix F
4.2 HOLES AS THE SOURCE OF EVALUATION BIAS
Our benchmark refinement with DREAM reveals that the original benchmarks contain a substantial number of holes with respect to the 25 retrieval systems we considered, indicating that many truly relevant chunks were left unlabeled. Table 4 presents the Hole@ 10 ratio, defined as the fraction of detected holes-relewe
chunks newly tem. The results show that the average Hole@ 10 across the 25 systems is 17.1%, which consider- ably large. Among them, retrieval systems based on advanced fusion approaches, such as rerank and rewrite, as well as recent systems, such as Aggretriever and Arctic, exhibit higher Hole@ 10 ratios than other conventional ones. These highlight that current benchmarks largely underestimate the ef- fectiveness of advanced retrievals due to pervasive unlabeled relevant chunks, leading to evaluation bias. Therefore, benchmark refinement is essential ensure fair and reliable evaluation.
within the top-10 chunks retrieved by each sys-
4.3 BIAS REDUCTION THROUGH FILLING THE HOLES
We validate the bias reduction of our benchmark refinement with DREAM through two metrics: (i) growth rate, which measures the proportion of newly detected holes as the number of retrievers the candidate pool increases, and (ii) marginal contribution, which quantifies the incremental retrieval performance gain when a target retriever added during this progressive expansion (see the equation in Appendix HH). For the refinement to be effective, both metrics should converge toward zero, since no additional retrievers should uncover new holes and further additions should not affect retrieval performance. Figure 3 shows benchmark refinement through DREAM's hole-filling: (a) the growth rate of newly identified gold chunks quickly saturates as more retrievers are added, and (b) the averaged marginal contribution across all retrievers diminishes with increasing retriever diversity, converging near zero. It confirms our refinement eliminates major holes in the original benchmarks, mitigates bias, and ensures reliable evaluation for future retrievers.
3soww
8

---

Published as a conference paper at ICLR 2026
0.48
-a-1 -B-N2
Marginal Contribution
0.16
-e-Hing:10 -0nDD00 Rocallig 10
Growth Rate
0.24
0.08
0.12
0.04
0.00
12
16
20
24
12
20
Number Retrievers in Candidate Pool
Number of Retrievers in Candidate Pool
(a) Growth Rate.
(b) Marginal Contribution.
Bias reduction via incremental addition of retrievals in DREAM's hole filling: (a) reducing growth rate of newly identified holes across seven benchmark subsets as more systems are incorpo- rated into the candidate pool; and (b) diminishing marginal contribution sured by Hit @ 10, nDCG @ 10, and Recall @ 10, as the pool expands. To ensure statistical reliability, results are averaged over 10 runs, each with 25 retrieval systems added in random order.
Figure 3:
target retriever, mea-
Table 5: Retrieval performance using Hit 10 on BRIDGE for 25 retriever systems. Subscript num- bers indicate improvements over the original benchmarks through hole filling. detailed breakdown for each benchmark and results with nDCG @ 10 metric are provided in Tables 19 and 20
Sparse
Dense
BM25
Splade
Contriever
DPR
DimRed
SBERT
BPR
Faiss
ANCE
ColBERT
Aggretriever
Aretic
0.65 0.23
0.80
0.11
0.50
0.54
0.60 0.18
0.66 0.18
0.69 0.18
0.73 0.74 0.77 0.12
0.79
0.87
BM25 Rerank
ANCE
Rerank
BM25 Rewrite
TBERT
NBoost
MLM
ELEC
MT5
TBERT
NBoost
MLM
ELEC
MIS
HyDE
Q2d
MuGI
0.72
0.76
0.75 0.75 0.18 0.78 0.14 0.77 0.16
0.10
0.80
0.16 0.83 0.12 0.67 0.77
0.71 0.25
0.75
BENCHMARKING RETRIEVAL AND GENERATION WITH BRIDGE
We re-benchmark 25 retrieval systems on BRIDGE and extend the evaluation to RAG generation to examine how our hole filling impacts both retrieval and downstream generation performance.
5.1 RE-BENCIMMARKIN RETRIEVAL SYSTEMS AFTER HOLE FILLING
Table presents the Hit@ 10 retrieval success rates of 25 systems on BRIDGE, with their score improvements over the original benchmark before hole filling with DREAM. All systems show higher Hit@ 10 scores after hole filling because previously missing relevant chunks correctly recognized as gold. However, the improvement magnitude varies substantially across systems, which in turn introduces bias in IR evaluation when such holes remain unaddressed.
Among the retrieval systems, advanced fusion with re- we-wiining (BM25 + Rewrite), which previously exhibited the highest Hole @ 10 ratios in Table 4 shows the largest improvement in Hit@ 10 scores after hole filling, indicating that systems with many missing relevant chunks are disproportionately penalized. This confirms that although prior benchmarks reveal evaluation bias due holes, DREAM successfully fills these gaps with 25-retriever pool. As shown in Figure 3 the growth rate of newly identified holes and the marginal contribution of additional retrievers converge to near zero, demon- strating that the bias is effectively resolved and enables fair evaluation over all retrieval systems.
We further analyze retrieval performance changes across the seven subsets in BRIDGE, covering diverse domains. The detailed results and analysis are presented in Appendix
5.2 ENHANCING RETRIEVAL-GENERATTI ALIGNMENT IN RAG EVALUATION
Reliable assessment of RAG systems requires alignment between retrieval and generation. When IR performance underestimated due to benchmark holes, the alignment between retrieval and generation breaks down, leading to biased system evaluation. For example, strong generation may be wrongly attributed to parametric knowledge because relevant chunks were mislabeled as non- relevant, causing retrieval be counted failure even when it actually succeeded.

---

Published as a conference paper at ICLR 2026
Hiug to (Rotrieval)
GPT-to-baal Evaluation (Generation)
Normalized Performanns
1.0
1.0
A MesECCAL
Merriiite acrba Call R ceseepaee
DOEAT
D0825-aan statt
Batchuu
Dio Ricbutton
0.0
0.0
Drkk
Bedection
Retrieval with High Rank
Retrieval with High Ronk
(a) Original Benchmark.
(b) BRIDGE Benchmark.
Figure 4: Normalized retrieval (green) and generation (yellow) performance across RAG systems, using the same Llama3.1 -88--Inttuutt model but different retrievers (labeled by name). Systems are sorted in ascending order of Hit @ 10 scores along the x-axis. Colors indicate rank changes between the original benchmark and BRIDGE: Red for drops and Blue for improvements, from (a) to (b).
To quantify this alignment, we introduce a metric, RAGAlign, which captures how improvements in retrieval translate into generation performance. It is defined for any pair of binary retrieval metric, METRICR &K (e.g., Hit@ K), and binary generation metric, METRICG (e.g., LLM-based binary evaluation). For each query, both metrics provide a binary outcome indicating success fail- ure for their respective tasks. Given N queries in the benchmark data, the score is computed as: RAGAlign@ @ K = + Σ 1 [METRICR; = METRICG;], which is the proportion of queries where retrieval and generation outcomes agree, reflecting their alignment. We instantiate the metric with Hit@ 10 and GPT-4o- based binary evaluation as the two metrics (see Appendix
Alignment Improvement in BRIDGE. Table 6 re- ports RAGAlign @ 10 scores on BRIDGE. comparing results before and after hole filling with DREAM. For each query, we use the top-10 results from 25 retrieval systems, and the generation metric is computed from answers generated by Llama3.3-88 --nsstuutttt Instruct for each query-chunk pair under a RAG pipeline (refer to Ap- pendix D for details). The results show that hole filling with DREAM substantially strengthens eetrieval-generatte erage, RAGAlign@ 10 improves by 0.14 compared to the benchmark before refinement. That is, our benchmark, BRIDGE, provides a clearer link between retrieval and generation, addressing the limitation of earlier benchmarks. This trend remains consistent even when using other LLMS such as Qwen2.5-7B ----tBIntttt and Gemma-2-9b-it See Appendix J.3 for details.
Table 6: RAGAlign @ 10 score gains over seven subsets of BRIDGE.
Version
BEIR
ROBUSTQA
Avg.
MS
NQ
Life
Roc
Toch Writ
Original |0.59 0.73 0.73|0.70 0.76 0.74 0.67 0.74 0.70
BRIDGE |0.88 (0.84 0.81
0.81 |0.84
alignment in RAG evaluation. On av-
Impact of Better Alignment on RAG Evaluation. Tigure4comprre normalized retrieval (green dots) and generation (yellow squares) performance across RAG systems, with (a) showing the orig- inal benchmarks and (b) the refined BRIDGE, After hole filling, substantial system-level rank shifts observed: 20 out of 25 systems change their retrieval rankings, with some (e.g., SBERT, ANCE) dropping in position and others (e.g., BM25+MuGI, SPLADE) rising, indicating that the original benchmarks distorted retrieval system comparisons.
Beyond retrieval ranking shifts, the figure also suggests improved retrieval-generation alignment from (a) to (b). In the original benchmark (a), the green and yellow curves show noticeable gaps for several systems, while in BRIDGE (b), the two curves align more closely with smaller gaps. The strong alignment also supported by Pearson correlation coefficient of 0.985 in BRIDGE. This indicates that retrieval gains more reliably translate into generation performance and thus DREAM offers stronger basis for the evaluation of RAG systems.
6 CONCLUSION
In this paper, we proposed DREAM, a debate-based relevance assessment pipeline that achieves high accuracy and effective Al-to-human escalation with minimal human intervention, without requiring additional training. Building on this pipeline, we constructed BRIDGE benchmark dataset by fill- ing missing relevant chunk in existing datasets, thereby mitigating bias in IR evaluation. Through re- ce-benchmarking 25 IR systems, we further demonstrate that unaddressed holes cause systematic retrieval underestimation and misalignment between retrieval and generation performance.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This paper addresses relevance labeling using LLM agents and raises no ethical concerns from methodological standpoint. In conducting human annotation, we maintained clear communication with all crowd-sourced annotators. These annotators were compensated above-minimum wage com- pensation and provided explicit consent for their contributions to be used in our study. No personally identifiable information was collected throughout the process, ensuring complete annotator privacy. For details on human annotation are provided in Appendix BRIDGE benchmark dataset.
We will also publicly release our
REPRODUCIBILITY STATEMENT
Every experiment conducted public datasets specified in Appendix [ and the preprocessing procedure is described in Section 4.1 We also utilized publicly available retrieval systems as listed Appendix D The multi-round dabate-based labeling process and the answer generation for RAG evaluation were implemented using various LLMs, covering both open-source and proprietary cate- gories. For open-source models, we utilized Llama3. 1-series, - and Qwenes5-series. For proprietary models, used GPT-4o-mini and GPT-4o through their paid API services. De- tailed description about model configurations and checkpoints is provided Appendix K We will also publicly release our implementation of DREAM.
ACKNOWLEDGMENTS
This work was supported by the National Research Foundation of Korea (NRF) grant funded by Ministry of Science and ICT (No. RS-2022-NR068758 and No. RS-2024-00334343) and by the Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by Ministry of Science and ICT (No. RS RS-2024-0044508)
REFERENCES
Angana Borah and Rada Mihalcea. Towards implicit bias detection and mitigation in multi-agent Ilm interactions. In EMNLP, 2024.
Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better Ilm-based evaluators through multi-agent debate. In
ICLR, 2023.
Chia-Yusa Yuan Chang, Zhimeng Jiang, Vineeth Rakesh, Menghai Pan, Chin-Chia Michael Yeh, Guanchu Wang, Mingzhi Hu. Zhichao Xu, Yan Zheng, Mahashweta Das, and Na Zou. Main-rag: Multi- agent filtering retrieval-ad auggenttd generation. In ACL, 2024.
Hung-Ting Chen, Michael Q. Zhang, and Eunsol Choi. Rich knowledge sources bring complex knowledge conflicts: Recalibrating models to reflect conflicting evidence. In EMNLP, 2022.
Justin Chih-Yan Yao Chen, Swarnadeep Saha, and Mohit Bansal. Reconcile: Round-table conference improves reasoning via consensus among diverse lims. ACL, 2024.
Prateek Chhikara. Mind the confidence gap: Overconfidence, calibration, and distractor effects in large language models. arXiv preprint arXiv:2502. 1028, 2025.
Charles L.A. Clarke and Laura Dietz. Llm-based relevance assessment still can't replace human relevance assessment. In EVIA, 2025.
Cyril W. Cleverdon. The cranfield tests index language devices. Aslib Proceedings, 1997.
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Voorhees. Overview of the tree 2019 deep learning track. arXiv preprint arXiv:2003.07822 078200, 2020.
Gabriel de Jesus and Sérgio Nunes. Exploring large language models for relevance judgments in tetun. In SIGIR, 2024.
Yilun Du Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. In ICMI., 2023.

---

Published as a conference paper at ICLR 2026
Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. Ragas: Automated evaluation of retrieval augmented generation. In EACL, 2024.
Guglielmo Faggioli, Laura Dietz, Charles L. A. Clarke, Gianluca Demartini, Matthias Hagen, Clau- dia Hauff, Noriko Kando, Evangelos Kanoulas, Martin Potthast, Benno Stein, and Henning Wachsmuth. Perspectives on large language models for relevance judgment. In ICTIR, 2023.
Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. From distillation to hard negative sampling: Making sparse neural ir models more effective. In SIGIR, 2022.
Robert Friel, Masha Belyi, and Atindriyo Sanyal. Raghench: Explainable benchmark for retrieval- augmented generation systems. arXiv preprint arXiv:2407.11005 2024.
Luyu Gao. Xueguang Ma, Jimmy Lin, and Jamie Callan. Precise zero-shot dense retrieval without relevance labels. In ACI., 2023a.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997 2023b.
Rujun Han, Peng Qi, Yuhao Zhang, Lan Liu, Juliette Burger, William Yang Wang, Zhiheng Huang, Bing Xiang, and Dan Roth. Robustqa: Benchmarking the robustness of domain adaptation for open-domain question answering. In ACL, 2023.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv22122009) 2021.
Feihu Jiang, Chuan Qin, Kaichun Yao, Chuyu Fang, Fuzhen Zhuang, Hengshu Zhu, and Hui Xiong- Enhancing question answering for enterprise knowledge bases using large language models. In DASFAA, 2024.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. TinyBERT: Distilling BERT for natural language understanding. In EMNLP, 2020.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In EMNLP, 2020.
Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Ed- ward Grefenstette, Samuel R. Bowman, Tim Rocktäschel, and Ethan Perez. Debating with more persuasive Ilms leads to more truthful answers. ICML, 2024.
Hannah Kim, Kushan Mitra, Rafael Li Chen, Sajjadur Rahman, and Dan Zhang. Meganno+: A LLM collaborative annotation system. In EACL, 2024.
Mahnaz Koupaee, Jake W Vincent, Saab Mansour, Igor Shalyminov, Han He, Hwanjun Song, Raphael Shu, Jianfeng He, Yi Nian, Amy Wing-mei Wong, et al. Faithful, unfaithful or am- biguous? multi-agent debate with initial stance for summary evaluation. In NAACL, 2025.
Satyapriya Krishna, Kalpesh Krishna, Anhad Mohananey, Steven Schwarcz, Adam Stambler, Shyam Upadhyay, and Manaal Faruqui. Fact, fetch, and reason: unified evaluation of retrieval- augmented generation. In NAACL, 2025.
Abhishek Kumar, Robert Morabito. Sanzhar Umbet, Jad Kabbara, and Ali Emami. Confidence under the hood: An investigation into the caniidence-probabiliy alignment in large language models. In ACL, 2024.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: A benchmark for question answering research. In TACL, 2019.
Canjia Li, Andrew Yates, Sean MacAvaney, Ben He, and Yingfei Sun. Parade: Passage representa- tion aggregation fordocument reranking. ACM Trans. Inf. Syst., 2023.
12

---

Published as a conference paper at ICLR 2026
Minghan Li, Honglei Zhuang, Kai Hui, Zhen Qin, Jimmy Lin, Rolf Jagerman, Xuanhui Wang, and Michael Bendersky. Can query expansion improve generalization of strong cross-encoder rankers? In SIGIR, 2024.
Yukun Li, Sijia Wang, Lifu Huang, and Liping Liu. Graph-based confidence calibration for large language models. Transactions Machine Learning Research, 2025.
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang. Yan Wang. Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi- agent debate. In EMNLP. 2024a.
Tian Liang Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujuu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking large language models through multi- agent debate. In EMNLP, 2024b.
Sheng-Chich Lin, Jheng-Hong Yang, and Jimmy Lin. In-batch negatives for knowledge distillation with tightly-coupled teachers for dense retrieval. In RepLANLP, 2021.
Sheng- Chich Lin, Minghan Li, and Jimmy J. Lin. Aggretriever: A simple approach aggregate textual representations for robust dense passage retrieval. Transactions of the Association for Computational Linguistics, 2022.
Yu-An Liu, Ruqing Zhang, Jiafeng Guo, and Maarten de Rijke. Robust information retrieval. In WSDM, 2025.
Zhenghao Liu, Han Zhang, Chenyan Xiong, Zhiyuan Liu, Yu Gu, and Xiaohua Li. Dimension reduction for efficient dense retrieval via conditional autoencoder. In EMNLP, 2022.
Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. Entity-based knowledge conflicts in question answering. In EMNLP, 2021.
Sean MacAvaney and Luca Soldaini. One-shot labeling for automatic relevance estimation. In SIGIR, 2023.
Luke Merrick, Danmei Xu, Gaurav Nuti, and Daniel Campos. Arctic-embed: Scalable, efficient, and accurate text embedding models. arXiv preprint arXiv:2405.05374, 2024.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. Ms marco: A human generated machine reading comprehension dataset. In NIPS, 2016.
Jingwei Ni, Tobias Schimanski, Meihong Lin, Mrinmaya Sachan, Elliott Ash, and Markus Leippold. Diras: Efficient I1m annotation of document relevance for retrieval augmented generation. In NAACL, 2025.
Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. Document ranking with pre-
trained sequence-to-sequencc model. In EMNLP, 2020.
Mare Oriol, Quim Motger, Jordi Marco, and Xavier Franch. Multi-agent debate strategies to enhance requirements engineering with large language models. arXiv preprint arXiv:2507.0595 2025.
Deonna M Owens, Ryan A Rossi, Sungchul Kim, Tong Yu, Franck Dernoncourt, Xiang Chen, Ruiyi Zhang, Jiuxiang Gu, Hanieh Deilamsalehy, and Nedim Lipka. multi-llm debiasing framework. arXiv preprint arXhv:2409.13888 2024.
Chanhee Park, Hyconseok Moon, Chanjun Park, and Heuiseok Lim. Mirage: A metric-intensive benchmark for petrieval-augmented generation evaluation. In NAACI, 2025.
Hossein A. Rahmani, Xi Wang, Emine Yilmaz, Nick Craswell, Bhaskar Mitra, and Paul Thomas. Syndl: A large-scale synthetic test collection for passage retrieval. In WWW, 2025.
Royi Rassin, Yaron Fairstein, Oren Kalinsky, Guy Kushilevitz, Nachshon Cohen, Alexander Libov, and Yoav Goldberg. Evaluating d-merit of partial-annotation on information retrieval. In EMNLP, 2024.
13

---

Published as a conference paper at ICLR 2026
Nils Reimers and Iryna Gurevych. Sentence-bert. Sentence embeddings using stamese bert- networks. In EMNLP, 2019.
Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and be- yond. Foundations and Trends@ Information Retrieval, 3(4):333-388) 2009.
Ariana Sahitaj, Premtim Sahitaj, Veronika Solopova, Jiaao Li, Scbastian Möller, and Vera Schmitt. Hybrid annotation for propaganda detection: Integrating Ilm pre-annotations with human intelli- gence. NLP4PI, 2025.
Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. Col- bertv2: Effective and efficient retrieval via lightweight late interaction. In NAACL, 2022.
Weizhou Shen, Chenliang Li, Hongzhan Chen, Ming Yan, Xiaojun Quan, Hehong Chen, Ji Zhang, and Fei Huang. Small Ilms are weak tool learners: A multi-llm agent. In EMNLP. 2024.
Upadhyay Shivani, Pradeep Ronak, Thakur Nandan, Craswell Nick, and Lin Jimmy. Umbrela: Umbrela is the (open-source reproduction of the) bing relevance assessor. arXiv preprint arXiv:2406.06551 06519, 2024.
Ian Soboroff. Don't use Ilms to make relevance judgments. Information Retrieval Research, 2025.
Tejas Srinivasan, Jack Hessel, Tanmay Gupta, Bill Yuchen Lin, Yejin Choi, Jesse Thomason, and Khyathi Raghavi Chandu. Selective "selective prediction": Reducing unnecessary abstention in vision-language reasoning. In ACL, 2024.
Elias Stengel-Eskin and Benjamin Van Durme. Did you mean...? confidence-based trade-off's in semantic parsing. EMNLP, 2023.
Fengfei Sun, Ningke Li, Kailong Wang, and Lorenz Goette. Large language models are overconfi- dent and amplify human bias. arXiv preprint arXiv:2505.0215), 2025.
Rikiya Takehi, Ellen M. Voorhees, Tetsuya Sakai, and lan Soboroff. LIm-assisted relevance assess- ments: When should we ask Iims for help? In SIGIR, 2025.
Yixuan Tang and Yi Yang. Multihop-rag: Benchmarking retrieval-augmenned generation for multi- hop queries. In COLM, 2024.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In NeurIPS, 2021.
Cole Thienes and Jack Pertschuk. Nboost: Neural boosting search results. https: wttp///rrarraaaaaaoaaa
2019.
Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. Large language models can accu- rately predict searcher preferences. In SIGIR, 2024.
Zailong Tian, Zhuoheng Han, Yanzhe Chen, Haozhe Xu, Xi Yang, Hongfeng Wang, Lizi Liao, et al. Overconfidence in llu-ae-a-judge: Diagnosis and confidence-driven solution. arXiv preprint UrXiv:2508.0622. 2025.
Yu-Min Tseng, Wei-Lin Chen, Chung- Chng-Chi Chen, and Hsin-Hsi Chen. Evaluating large language models as expert annotators. In COLM, 2025.
Shivani Upadhyay, Ronak Pradeep, Nandan Thakur, Daniel Campos, Nick Craswell, Ian Soboroff, and Jimmy Lin. A large-scale study of relevance assessments with large language models using umbrela. In ICTIR, 2025.
Neeraj Varshney, Swaroop Mishra, and Chitta Baral. Investigating selective prediction approaches across several tasks in IID, OOD, and adversarial settings. In ACI. 2022.
Han Wang, Archiki Prasad, Elias Stengel-Esking and Mohit Bansal. Retrieval auggenete generation with conflicting evidence. In COLM, 2025.
14

---

Published as a conference paper at ICLR 2026
Liang Wang, Nan Yang, and Furu Wei. Query2doc: Query expansion with large language models. In EMNLP, 2023.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: deep self- attention distillation for task-agnostic compression of pre-trained transformers. NeurIPS, 2020.
Xinru Wang, Hannah Kim, Sajjadur Rahman, Kushan Mitra, and Zhengjie Miao. Human-Ilm col-
laborative annotation through effective verification of Ilm labels. In CHI, 2024.
Kai Xiong, Xiao Ding, Yixin Cao, Ting Liu, and Bing Qin. Examining inter-consistnncy of large language models collaboration: An in-depth analysis via debate. In EMNLP, 2023.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In ICLR, 2021.
Yifei Xu, Tusher Chakraborty, Emre Kiciman, Bibek Aryal, Srinagesh Sharma, Songwu Lu, and Ranveer Chandra. Rlthf: Targeted human feedback for Ilm alignment. In ICML, 2025.
Ikuaa Yamada, Akari Asai, and Hannaneh Hajishirzi. Efficient passage retrieval with hashing for open-domain question answering. In ACL, 2021.
Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Gui, Ziran Jiang, Ziyu Jiang, et al. Crag-comprchensive rag benchmark. In NeurlPS, 2024.
Puxuan Yu, Luke Merrick, Gaurav Nuti, and Daniel Campos. Aretic-embed 2.0: Multilingual re- trieval without compromise. arXiv preprint arXiv:2412.045006 2024.
Le Zhang, Yihong Wu, Qian Yang, and Jian- Nie. Exploring the best practices of query expan- sion with large language models. In EMNLP, 2024a.
Le Zhang, Yihong Wu, Qian Yang, and Jian Yun Nie. Exploring the best practices of query expan- sion with large language models. In EMNLP, 2024b.
Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhao Chen. Context-faithhin prompting for large language models. In EMNLP, 2023.
15

---

Published as a conference paper at ICLR 2026
A AUTOMATED RELEVANCE LABELING DETAILS
A.1 MULTI-AGENT MULTI-ROUND DEBATE DETAILS
The prompt templates used for our multi-round debate are provided in Tables 28 Each template includes placeholders that instantiated differently for the two agents. We designate each agent as "Agent A" or "Agent B" and use clear self-referential markers such as "You are Agent A" and "Agent A (You)" when the agent refers itself, while using "Agent B" when referring to the counterpart. This explicit labeling maintains clear role separation and ensures that each agent consistently argues from its designated position while directly addressing and challenging the opposing viewpoint.
Multi-Round Protocol. Table 27 provides the prompts for the first round debate. In the first round, each agent is assigned an initial stance and is made aware of it through the coll-condiiionnng Table 28 shows the prompts for the subsequent round debate. In subsequent rounds, before deciding a new reasoning and label, each agent must consult the dehate history, including the previous round's reasoning outputs and the recorded label for that round, and then update, defend rebut its position.
Prompt Operation. For each debate turn, the agent (i) carefully reads the query and all candidate answers to ensure full comprehension; (ii) reads the given chunk; and (iii) determines whether the chunk fully supports any candidate answer under the evaluation guidelines. The guideline is in- tended to constrain the agent's role to an answer-aware query-chunk relevance assessment, ensuring that the agent only verifies whether chunk fully supports any candidate answer to the target query. In particular, the rules explicitly prohibit judging answer correctness, query-answer validity, or an- swer-answer consistency, thereby narrowing the task to strict answer-aware relevance assessment. To ground the judgment, the agent must quote the decisive evidence from the chunk. Finally, the agents provide rationale covering their query-chunk interpretation, response to the opposing agent's position, and clear justification for the final decision.
A.2 EXPERT ANNOTATION PROTOCOL
To evaluate relevance labeling methods in Section 3.2 we collect the ground-truth relevance labels (1: relevance, irrelevance) for 700 query-chunk-anee ansser triplets. We recruit seven graduate-level NLP specialists with complementary academic backgrounds (i.e., computer science, artificial intel- ligence, business, cconomics, and social science) and compensate them $15 per hour. All annotators complete five practice examples to align guideline interpretation. To facilitate adjudication, each triplet is paired with supporting and opposing reasons generated by Llama3. 1aama3.1-880 Instruut encourag- ing balanced consideration before final judgment. A senior researcher spot-checks a random subset to verify label consistency and resolve potential misjudgments in ambiguous cases. This protocol ensures high-quality ground-truth labels for evaluating pipeline accuracy. Through expert annota- tion. the 700 samples are identified as 212 relevant and 488 irrelevant chunks.
A.3 EXPERIMENT IMPLEMENTATION DETAILS
For the L.MMJudge, we adopt single-agent setting using the prompt in Table 17 DREAM employs two agents prompted with Table 27 28 engaging in up to two rounds of debate. If the two agents reach an agreement, their consensus is taken the final label; otherwise, the case is escalated to a human annotator for adjudication. We implement LARA based on the official code released on GitHubt where employ the simple binary prompt setting.
A.4 LLM ADIUDICATION SETUP DETAILS
Adjudication is triggered only when the two agents fail to reach consensus on a label. The adjudicator is implemented with Llama3.3. - with temperature 0.0. The full prompt in Table 29 During adjudication, the adjudi- cator receives the debate outputs produced at the end of the corresponding round from both agents as discussion history, along with the query, answers, and candidate chunk.
We detail our LLM-based adjudication setup Section 3.2.2
https://githh/e
yaal///aiii LARRA
16

---

Published as a conference paper at ICLR 2026
Table 7: Detailed statistics of the seven datasets in BRIDGE. Each dataset categorized by do- main and chunk length, including the number of corpus and queries, average number of chunks and answers per query, and average word counts for queries, chunks, and answers.
Dataset
Domain
Chunk Length
# Corpus
e Query
Avg. #
Avg. AQQ
Avg. Wards
Query
Chunk
Answer
MS MARCO
8,841,823
550
16.77
1.11
5.83
56.58
13.67
Wehsearch
Short
NQ
2,68 (468
Lifestyle
Lifestyle
119,461
550
6,61
1.47
10.26
152.77
9.46
Recreation
Recreation
166,975
550
4.63
1.46
8.74
137.67
6.94
Science
Science
Long
1,000,000
357
16.09
1.37
8.82
127.74
7.95
Technology
Technology
638,509
Writing
Writing
199,994
550
8.52
1.57
8.26
120.45
8.34
Table 8: Dataset-level chunk statistics across the two-stage debate process. The table shows: (1) the dataset name, (2) the initial pool of candidate chunks be labeled, (3) the number of chunks
retained after irrelevant chunk filtering, and (4-7) the number of chunks that reached or failed to reach consensus after Rounds and 2 of agent-based debate.
Dotasct
Candidate Chunks
After Irrelevant
After Debate Round
After Debate Round
Chank Filtering
# Consensus
Disagreement
Consensus
Disagreement
MS MARCO
35,579
20,137
17,893
2.244
1,382
862
NQ
41,656
11.126
10,558
568
422
146
Lifestyle
13.159
1,394
Recreation
44,466
11,207
10,283
924
672
252
Science
33,966
19.011
16.525
2,486
1,646
840
Technology
51,104
20,876
18,491
2,384
1,552
832
Writing
47,239
19,712
17,529
2,182
1,473
709
Total
296,083
116,622
104,438
12,182
8,126
4,056
B DETAILED STUDY ON DEBATING FRAMEWORK
IMPACT OF STANCE ORDER
In Table we compare the final debate labels for all consensus cases under the "Relevant-fire (RF)" and "Irrelevantffirt (IF)". Across 676 queries, 96.7% (654/676) of predictions remain identical after swapping the agent order, and only 3.3% (22/676) change. Importantly,the flips oc- cur in both directions: eight cases switch from relevant to irrelevant, and 14 cases switch in the opposite direction. If the model were simply fol- lowing the first agent, label changes would be heavily skewed toward one side. The nearly sym- metric flip pattern instead demonstrates de- tectable dependence on the initial stance or first- speaker order.
Table 9: Row corresponds to the final labels obtained when agents tart from the "Relevant- first (RF)" stance, and column corresponds to the final labels obtained when agents start from the "Irrelevant-firs (IF)" stance. The value rep- resents the how consistent the final labels are across the two settings.
RF II
Irrelevant
Relevant
Total
Invlevant
393
14
407
Total
401
275
676
Moreover, as shown in Table 10 the bAce essentially stable across different reason- ing orders, indicating that the overall eval- uation quality not influenced by which agent speaks first. This indicates that the debate outcomes are not affected by fist- speaker bias order-dependent effects.
Table 10: Comparison of bAce and escalation ratio be- tween "Relevant-first (RF)" and "Irrelevant- first (IF)".
Method
Recall (Relevance) (Imeeleance)
Recall
bAce
Escalation
Relovant First (RF)
98.4% 97.3%
91.9% 90.1%
95.2% 93.7%
3.8%
Irrelevant First (IF)
4.4%
B.2 LATENCY COMPARISON
To examine the trade-off between annotation accuracy and annotation cost, we compute the cost and latency per sample (i.e., single mmery-chunk-answe triplet) and compare method DREAM
17

---

Published as a conference paper at ICLR 2026
against both Human-Only and LIMJudge in Table I1 We omit the cost report for LARA, as it relies on the same single-agent model as LLMJudge and therefore incurs identical cost and latency.
The human or API cost reports the hu- man labor cost for Human-Only and API cost (based input and output token costs under Deep Infra pricing) for both LLMJudge and DREAM. Latency is mea- sured in two settings: Non-parallel latency corresponds to a single API session, where we sum the generation times of all partici- contrast, parallel latency
Cost and latency of labeling methods. We report human labor cost for Human-Only and API cost for both LLMJudge and DREAM. Non-parallel latency denotes processing time under a single API session, while parallel latency reflects execution with multiple concurrent API sessions.
Table 11:
Non-parallel
Parallel Latency (scc)
Human or Cost (S)
bAcc (%)
Escalation Ratio (%)
Method
Latency (soc)
pating agents. corresponds to multiple API sessions, mea- sured as the wall-clock time when all agents generate their outputs concurrently. The re- ported results are averaged over 147 sam- ples and used
Human Only
0.1
30
30
| 93.8
100.0
LLMJudge
0.000066
1,292
1.292
DREAM
0.000470
8.583
4,291
95.2
3,5
for LLMJudge and DREAM.
Human-Only yields strong labeling accuracy (bAce), but this performance comes with very high human cost and latency. On the other hand, LLMJudge minimizes the cost and latency but its bAce significantly lower than Human-Onll y. indicating that it struggles to reliably identify relevance labels. In contrast, DREAM achieves even higher bÂce while being 200x cheaper and 3.5x-7.0x faster than Human-One Only. This shows that DREAM achieves a favorable balance between cost and performance, enabling scalable high-quality annotation that would be infeasible with humans alone.
B.3 AGREEMENT STABILITY AND ANNOTATION QUALITY ACROSS MULTI-ROUND DEBATE
A central validity concern in multi-agent debates is whether consensus reflects genuine convergence or spurious agreement due stochastic fluctuations. To assess the stability of consensus, we examine whether round-1 agreements persist when the debate continues for additional rounds, and we compute the correspond- ing spurious consensus rate. We sample 651 instances that reach consensus in round and continue the de- bate up to round 5, with discussion history consisting only of previous-round outputs.
Table 12: Retention of round consensus and spurious consensus rate across subse- quent rounds (R2-R5). Persistence ratio denotes the proportion of cases that remain consensus in each later round.
Round
R1
R2
R3
R4
R5
Persistence ratio (%)
96.5
96.2
96.2
96.0
TTottl Consensus
651
651
651
651
625
Spurious
5.0%
3.9%
5.0
%
5.1%
5.1%
Table 12
shows that 96% of round consensus is pre- served in round 2, and this level of stability continues through round 5, indicating that the consensus can be regarded as stable. However, increasing the number of debate rounds does not consistently reduce the spurious consensus rate. The lowest rate appears at R2 (3.9)), after which it rises again, reaching 5.1% at R5. That is, more rounds eventu- ally introduce additional noise into the discussion, making incorrect agreements more likely rather than less.
In detail, from R1 to R2, observe cases where spurious consensuses initial disagreements corrected R2, because these errors are relatively easy for LLMs to reassess once the opposing ar- guments are explicitly surfaced. However, extending the debate beyond two rounds pushes the cases that remain ambiguous even after R2 toward an incorrect "irrelevance" consensus, as evidenced by the drop in recall (relevance) while irrelevance recall remains stable. These results demonstrate that using more than two rounds carries risk of amplifying spurious consensus.
B.4 IMPACT OF IMPROVED DIVERSITY
DREAM uses two Llama3.3- 3-B Instrrctt models with a temperature of 0.0 as its LLM agent. To in- vestigate whether increasing diversity in the debating setup improves annotation quality, conduct an ablation study with DREAM on its two design choices: (i) increasing the number of agents, (ii) using heterogeneous model families, and (iii) using non-zero temperature for each agent.
18

---

Published as a conference paper at ICLR 2026
B.4.1 INCREASING AGENT NUMBER
We first examine the impact of scaling the number of debating agents from two to four. Specifically, compare three configurations: (1) 2-Llama, which uses two 10mma33.--0B-Inttruue models (our default setting); (2) 4-L1 ama, which em- models; and (3) 2 12-Llame ama 2 2-Qwen, which combines two 1lama3.3-70B-Imm
Table 13: Comparison of annotation quality un- der different numbers of debate agents. 2L+2Q represents a heterogeneous configuration com- bining Llama and Qwen models.
ploys four 1lama3.3-700-Imse
Method
Class-wise Rocall Relevance
bace
Escalation Ratio
Inrlevance
2 11ame.
91.9%
98.4%
95.2%
3.5%
and two Qwen2.5-72B- Instruct models, with each model family initial- ized with both relevant and irrelevant stances.
4-I.1
97.2%
91.4%
94.3%
11.2%
2L 2Q
97.3%
92.0%
94.7%
13.0%
As expected. increasing the number of agents naturally reduces the consensus rate (100% escalation ratio), since more models must reach agreement before a final label can be assigned. Al- though this stronger agreement requirement could be expected improve labeling accuracy (bAce) on non non-escalated cases, we find that bAcc actually decreases. As more models participate, the de- bating process becomes increasingly conservative toward relevant cases, making agreement on them harder to reach compared to irrelevant ones. This leads to higher number of disagreements on relevant examples, which in turn reduces recall for the relevant class and overall labeling accuracy.
B.4.2 USING HETEROGENEOUS LLMS
We next investigate whether introducing hetero- geneity in model families can mitigate inherent biases and improve annotation quality. We evalu- ate four configurations: (1) 2 2-L1 ama (our de- fault); (2) 2- -wwen, using two Qwen2.5-72B- Instruct models; (3) Llama (Relevant) Owen
Table 14:
Comparison of annotation quality across debate settings with different model fam- ilies. L+Q uses Llama the relevant initiator and Qwen as the irrelevant initiator, Q+L uses the opposite initialization.
Method
Class wise Recall Relevance
bAce
Escalation Ratio
) where Llama is initial- ized with the relevant stance and Qwen with the irrelevant stance; and (4) Owen (Relevant) +
Irrelevance
2-1
91.9%. 95. 1%
98.4%
95.2% 89.0%
3.5%
2-Cwen
82.8%
2.6%
Llama (Irrelevant), the reverse configura- tion of (3).
92.3%
96.0%
91.4%
93.7%
6.5%
Our analysis reveals distinct biases in each model family: the 2-L1ame ama baseline exhibits higher recall for the relevant class, while the 2 2-Qwen baseline shows higher recall for the irrelevant class, indicating that Llama leans toward predicting "relevant" and Qwen toward "irrelevant". The heterogeneous setup reduces these biases, achieving higher recall for the irrelevant class than 2-L1 ama and higher recall for the relevance class than 2-Qwen. However, its bAce remains lower than 2-Llama because Qwen's 's weaker performance drags down overall accuracy. This suggests that heterogeneous debate is beneficial only when all participating models are individually strong.
B.4.3 ADJUSTING TEMPERATURE VALUES
We also vary the sampling temperature under the 2- Llama configuration assess its impact on debate quality. Under higher temperatures, models pro- duce often longer reasoning because the sampling distribution becomes flatter, making low- low-probability tokens more likely to be selected. This lengthen- ing causes the debate to drift toward irrelevance, which turn makes the final decision more conser- vative. Although recall for the irrelevant class im-
Table 15: Comparison of annotation quality across debate settings with different model temperatures.
Temp.
Class-wise Recall
bAcc
Escalation Ratio
Inrlevance
Relevance
0.0
91.9%
98.4%
95.2%
3.5%
93.5%
92.7% 91.9%
93. 1%
3.8%
1.0
93.3%
92.6%
3.1%
proves slightly, the substantial drop in recall for the relevant class leads to an overall decline in labeling accuracy (bAcc).
19

---

Published as a conference paper at ICLR 2026
Table 16: Publicly available retrieval model links used in this paper.
Model
Public Model Checkpoint (Link)
BM25
Robertson
2009
/pppp wttp//ro/ecc//ssseeeooo pctlpyscriii
SBERI
Kemmers & Gurey vel
2019
hipps www.gngggeeeee.... ace conentence conentencee-ranallee tamnsooreee/mmmmmrrr hawpaarrrodduuubbbb-b-----a----
NBooss.
Thienis Periscux
hlips///ugggnn sice conbousipp
base uncused mismaacco
DPR
Karpukhin el ai.
4020
http://hegpin place envsentence transi crmnsssommme//ieeeeee htt.eh/www.....cc.eoommmemmeeeee
Faiss
D5155
[Karpukni
2020
/ convaaceooo www.ooo/seccectttaaii
Tiny BERT
[Ta 2010
miips//hugggnge
encolesims
BERT-L
MiniLM
(Wang of 202011
htpp: h/hhggggggggng gingrace wttpoons-cooddmmmm-ee caaeee/mmmmmmo--mmaa Miniooo--MiiLaa M-LII-
MonoT5
niips //wggggegaacc corcastorini moonst--bss---smnne
BPR Yamala ct m. 2021
ht h/w.gnngggee...... conncome
cove-covid--ae base base--sssaeeeeee mismarce missaace--issiibett--aaa distilbert-tas-
Contnever [iracani
hllps Whuggingi wwt.aacccccccceeeeeeeeeeee cov/acehenn//oonnneee
TCT-COL Lin
2021
hitps whw.gggggggnggg covcastorin/tttoolle wateeer-----------e- 2-hhop-mamareo
ANCE Xiong ain K0M1
mtp /wwggg consentenc--trae new mismar waw----aaa------------
Dim Reduchon
2022
mips /huggigggggg prace
oemery dishhhe- base--tt--------
SPLADE (Formal et of (222
heps www.gghgggagcccccccccco
Ageretriover (Lan
formu aeerelreve ever--ocooueeeee
ELECTRA et 2UAS
hi http://ugggggeee curenos encodde mancu electra base
HyDE
20/2
hiips Wtphh/phhhhi comet//httoooottoo//oooooooooo
Q2D
Wang et
N724
hmps htps: Marki
MuGI Zhang oral
1422-bb
htw.///thaa..coom com cont/ezzann Kewee MuGi
Arctic Mernic
mips hips://ugggggaccc.oo
C DETAIL OF BRIDGE BENCHMARK DATASET
C.I SOURCE DATA
MS MARCO (Nguyen et al. 2016) contains real anonymized Bing queries paired with manually curated answers reflecting real-world search behavior. NQ (Kwiatkowski et al., 2019) consists of Google queries linked to Wikipedia pages with human-annotated answer spans. LoTTE (Santhanam al. 2022) provides long-tail, topic-stratified QA data from the StackExchange community, which enables robust out-of-domain evaluation. It covers five domains: Lifestyle (e.g., cooking, sports, and travel), Recreation (e.g., gaming, anime, and movies), Science (e.g., math, physics, and biology), Technology (e.g., Apple, Android, UNIX, and security), and Writing (e.g. English). These domains introduce diverse question styles and content types, offering a comprehensive evaluation setting that spans both factual and user-generated queries.
C.2 BRIDGE STATISTICS DETAIL.
preseenss detailed statistics of the 7 datasets in BRIDGE, spanning 6 diverse domains includ- ing web search, science, and lifestyle. The datasets are categorized as either short- or long-fom based on the average chunk length, with those exceeding 100 words per chunk classified as long- form. The benchmark contains 3,657 queries. Each domain contributes 550 queries, and each query paired with an average of 10.08 gold chunks. Answer lengths range from 2.14 to 13.67 words, covering both concise and elaborate responses. We also provide the chunk statistics observed during the relevant assessment, spanning from candidate chunk pooling to debate round 2, in Table g
presents
D RETRIEVAL SYSTEMS
Throughout BRIDGE's construction and experimentation, utilize 25 RAG systems, as described in Section 4.1 This configuration covers sparse, dense retrievers, as well as fusion techniques such re-ranking and query rewriting. This varied setup facilitates comprehensive analysis in our experimental evaluations. These are publicly available retrieval models and methods (see Table 16
E IRRELEVANT CHUNK FILTERING
To enhance the efficiency by filtering out obviously irrelevant cases, discard clearly irrele- vant chunks prior to conducting relevance assessment. It employs three LLMs, i.e., GPT-4o-mini. and Sreen..5-72B-Istiii
it helps reduce the risk of over-pruning due to single-model bias (Owens et al. 2024 Borah & Mihalcea 2024). The prompt is presented in Table 17. Since the dataset consists of query-chunk-anssee answer triplets, consider both query and
1lama3.3-70B-Insttuu
20

---

Published as a conference paper at ICLR 2026
Table 17: Prompt for LLMJudge and irrelevant chunk filtering.
Determine if DOCUMENT contains enough information answer QUERY with any the AN- SWERs.
QUERY: {query}
DOCUMENT: {chunk)
ANSWERs:
(ground-truth answers)
If any ANSWER is supported by the DOCUMENT, return true. If all ANSWERs not supported, return false. Do provide explanations or additional information.
Respond strictly in this format: ("is.suppan siisssupportedd true/false)
ground-truth answer when assessing chunk relevance. Each chunk is evaluated by three LLMs for sufficiency in answering the query given the ground-truth answer, and discarded only if all agree it irrelevant, thereby reducing the candidate pool from 296.053 116,622.
F HUMAN ANNOTATION
F.1 RECRUITMENT AND COMPENSATION
We apply strict qualification criteria for MTTurk annotators to ensure the quality of human annotation. Participation is restricted to MTurk workers with a HIT approval rate above 90% and at least 500 approved HITs to filter for experienced workers. Additionally, all annotators must achieve mini- mum score of 90 on our custom English comprehension exam designed to simulate actual annotation tasks, ensuring both fluency in English and familiarity with the task format.
Annotators are compensated $7.5 per hour, rate that exceeds the U.S. federal minimum wage. All procedures follow standard ethical guidelines, including informed consent, with instructions clearly stating that annotations will be used solely for academic research and consent obtained prior to par- ticipation. To protect worker privacy, we not collect any personal or sensitive data, including detailed demographics such age gender. As the study poses minimal risk and ensures par- ticipant anonymity, we did not seek ethics board approval accordance with standards commonly considered exempt from review.
F.2 ANNOTATION PROCESS
Human Annotation. Each chunk evaluated by three qualified MTurk workers, who are provided with the target query, a set of its possible answer(s), the candidate chunk, and the debating history that produced disagreement. Annotators are instructed determine whether the chunk contains sufficient evidence to support the gold answer(s), following unified annotation guideline. The debating history provided to help annotators understand why the agents led disagreement and to highlight potential ambiguities, thereby enabling more informed and consistent labeling decisions. Final labels are determined by majority vote among the three annotators.
Attention Check. We insert 10% of attention checks into each HIT. These are designed to re- semble real tasks, but have objectively verifiable answers to ensure high-quality annotations on crowd-sourcing platforms. Specifically, we construct attention checks by pairing a ground-truth chunk from existing datasets with a query-gold answer pair and two conflicting arguments from unrelated domains. Attention check chunks are intentionally correct and must be labeled as rele- vant candidate chunks. Annotators are required to pass all attention checks for their responses to be accepted. Attention checks are randomly placed to prevent predictability and have moderate dif- ficulty to maintain both fairness and effective filtering. This strict filtering improves overall label consistency without significantly increasing cost.
21

---

Published as a conference paper at ICLR 2026
G HOLES AS THE SOURCE OF EVALUATION
Table 18 shows a detailed breakdown of the Hole@ 10 ratios across 25 retrieval systems for each of the seven subsets. The results indicate that MS MARCO exhibits the highest Hole @ 10, which can be attributed to its large corpus size.
Table Hole@ 10 ratios across 25 retrieval systems and the existing benchmark. Systems with higher percentage of holes than the average across retrieval systems marked in bold.
Dataset
Method
Model
Avg.
MS
NQ
Life
Rec
Sci
Tech
Writ
BM25 SPLADE
23.3%
12.3%
10.5%
7.1%
21.5%
14.2%
13.3%
14.6%
Sparse
41.3%
18.2%
12.4%
8.8%
23.0%
15.3%
14.3%
19.0%
Contriever
16.
5.1%
3.9%
3.7%
6.4%
4.9%
5.7%
6.5%
DPR
22.6%
12.4%
5.2%
3.2%
10.6%
5.2%
7.5%
9.5%
Dim Reduction
35.0%
9.0%
8.1%
5.0%
9.4%
7.3%
6.7%
11.5%
SBERT
BPR
35.9%
12,9%
8.8%
6.2%
16.4%
10.6%
10.5%
14.5%
Dense
Faiss
39.5%
14.3%
10.7%
7.4%
17.7%
11.6%
10.3%
15.9%
ANCE
37.8%
13.4%
11.2%
7.4%
18.3%
12.5%
12.6%
16.2%
TCT-COLBERT
39.2%
15.4%
10.8%
7.1%
19.1%
12.7%
12,2%
16.6%
Aggretriever
39,9%
15.7%
10.8%
8.3%
20.2%
14.2%
13.4%
17.5%
Arctic
41.1%
18.4%
13.4%
10.9%
24.7%
20.2%
17.2%
20.8%
TinyBERT
35.8%
16.2% %
11.7%
8.2%
21.7%
14.1%
13.3 %
17.3% %
NBoost
37.2%
17.8%
11.9%
9.7%
23.1%
14.9%
15.1% %
18.5
BM25
MiniL.M
38.3%
18.2%
12.0%
9.5%%
22.3%
14.5%
14.6 %
18.5%
ELECTRA
36.9%
17.9%
13.0%
9.4%
23.4%
14.9%
14.4 %%
18.6%
Rerank
BiRYBERT
21.1%
NBoost
39,2%
17.4%
12.7%
9.5%
22.8%
15.3%
14.7%
18.3%
ANCE
MiniLM
41.3%
17.7%
12.7%
9.1%
21.4%
15.0%
14.5%
18.8%
ELECTRA
39.7%
17.7%
13.2%
9.2%
22.4%
15.1%
14.5
18.8%
MonoTS
40.4%
18.2%
14.1%
10.2%
25.5%
17.3%
15.6%
20.2%
HyDE
36.1%
25.2%
14.0%
8.7%
28.0%
20.9%
15.7 %
21.2%
Rewrite
BM25
Q2D
36.9%
22.3%
14.4%
9.1%
20.7%
18.0%
21.3%
MuGi
38.0%
24.8%
14.8%
9.1%
28.7%
21.3%
19.6%
22.3%
Avg.
36.2%
16.3%
11.4%
7.9%
20.6%
14.1%
13.2%
17.1%
H EVALUATION FORMULA
To demonstrate the bias reduction of BRIDGE refined by DREAM, we measured on two metries.
Growth Rate. To assess whether the candidate pool constructed with multiple retriever is saturated enough, we compute the growth rate that is the proportion of newly detected relevance chunks (holes) as the number of retrievers in the candidate pool increases.
GR(m) (H(Cm. ( -1(
m 2
|H(Cm-1)
where Cm denotes a candidate pool constructed with subset of m retrievers, H(Cmm) denotes the set of unlabeled relevant chunks discovered by the candidate pool Cm. H(Cm-11 denotes the set of unlabeled relevant chunks discovered by the previous candidate pool Cm-1-
Marginal Contribution. To examine whether the dataset annotated with multiple retrievers remains robust for evaluating a retriever not included in candidate pool, evaluate each retriever on dataset annotated with the results of the other 24 retrievers, excluding itself. The marginal contribution is defined as the absolute performance difference between the dataset annotated with the target retriever and the dataset where excluded. The marginal contributions across all retrievers then averaged and illustrated in Figure
MCM (r) Perf p. Perf Dsm (r) Sm S R.V. (r).
22

---

Published as a conference paper at ICLR 2026
Table 19: Retrieval Hit 10 performance on BRIDGE for 25 retrievers over 7 datasets, with im- provements (1) over original benchmarks in subscripts. Best performance in bold.
Dataset
Method
Model
Avg.
MS
Life
Rec
Sci
Tech
Writ
Spanse
BM25
0.68
0.61 0.11
0.34
f
0.77 0.19
SPLADE
0.89 0.25
0.82 0.13
0.81 0.02
0:77 0.27
0.81 0.181
0.66 0.12
0.87 0.05
0.80 0.11
Contriever
0.61
0.48 0.21
0.52 0.08
0.51 0.66
0.45 0.23
0.38 0.26 f
0.58 0.08
0.50
DPR
0.61 0.35 f
0.72 0.12
0.57 0.16
0.44
0.50
0.32 0.17
0.64
0.54 0.19
Dim Reduction
0.79
0.56
0.59 0.70 f
0.52
0.45 f
0.58 0.14
0.10
SBERT
0.84 0.30
0.66 0.15
0.63 0.11 7 0.16
0.61 0.24
0.49 0.18 T
0.70 T
0.66 0.18
Dense
BPR
0.82 f
6.16
0.70
?
0.25
T
0.75 0.15
0.18
Faiss
0.27 0.23 f
0,74 0.121
0,77 0
0.72
0.68 0.221
0,58 0.19
0.14
0.73 0.16
ANCI
0.85 0.28 f
0.72 0.15
0.78 f
0.74 1
0.74 0.24
0.57 0.17 f
0.77 0.13 T
0.74 0.17
TCT-COL BERT
0.89
0.78 0.18
0.80
0.74
0.76
0.61 0.16
0,81
0.77 0.12
Aggretriever
0.88 0.29
0.79 0.14
0.80 1
0.75 0:41
0.77 0.20
0.68 0.14 T
0.85
0.79 0.13
Arctic
0.10
0.82 f
0.15
0.12 1
TBERT
0.80 f
0.72 0.12
0.71 0.74
0.72 0.25
0.55 T
0.80 T
0.72 0.17
NBoost
0.82 0.30
0,75 0.12
0.79 0.02
0.76
0.77 0.21
0.61 0.13
0.04
0.76 0.12
BM25
MiniLM
0.83 0.28
0.75 0.12
0.79
0.75 006
0.74 0.24
0.58 0.17 1
0,840.14
0.75 0.16
ELEC
0.82
0.75
0.18
0.75
0.57 0.18 f
f
0.75
Mono'15
0.84 0.32
0.77 0.13
0.81 0.04
0.77 0.0
0.80
0.14 +
0.86 0.07
0.78 0.14
Rerank
TBERT
0.86 0.26
0.77 0.12
0.79 f
0.75 0.8
0.76 0.23
0.19
0.13
0.77 0.16
NBoost
f
0.13
0.85 1
0.79 0.0
0.790.155
0.66 0.11
0.86 T
0.80
ANCE
MiniLM
0.87
0,80 0.121
0,84 f
007
0.78
0.65 0.18
0,86
0,80 0.10
EL
0.87 0.27
0,80
0.84
0.78 1
0.78
0.64 0.18 1
0.80 0.86
MonoTS
0.89
0.82
0.87 0.02
0.81
0.17
0.70
0.88
0.83
HyDE
0.2
0.16
0.66 f
0.58
0,39
0.53 0.31 f
0.69 0.23
0.27
Rewrite
BM25
Q2D
0.78 0,40
0.81 0,15
0,72
0.63
0.68 0,37
0.31 0,58
0.79 0.20
0.71 0.25
MuGI
0.81 0.39
0.83 0.13
0.75 0,18 t
0.68 0.12
0.71 0.36
0.61 0.31 f
0.83 0.19 f
0.75 0.24
Avg.
0.82 0.31 f
0.75 0.14
0,760.10 1
0.70
0.71 0.25
0.79 0.19
0.73 0,17
where R. denotes the full set of retrievers, r E R. the target retriever, S., a subset of m re- trievers excluding DA denotes a dataset annotated with retriever set A, and Perf (()) denotes the performance of retriever r on dataset D (e.g., Recall@k, Hit@k, nDCG@k).
1 RETRIEVAL BENCHMARKING DETAIL
The comprehensive retrieval evaluation presented in Tables 19 and 20 demonstrates contrasting pat-
terns between Hit@| 10 and nDCG@ 10 metrics following label correction. While Table 19 shows consistent positive improvements across all 25 retrieval systems, with query rewriting approaches (HyDE, Q2D, MuGI) achieving the most substantial gains averaging 0.25 to 0.27, Table 20 reveals a markedly different pattern. Most retrieval methods exhibit performance decreases in nDCG@) 10, with only MS MARCO showing consistent improvements and query rewriting methods demonstrat- ing modest gains. This phenomenon can be attributed to the nDCG metric's structural characteris- tics. When additional relevant chunks are identified lower ranking positions through label correc- tion, the ideal DCG (IDCG) increases more substantially than the actual DCG, resulting in lower nDCG@n @k scores despite improved recall. These findings suggest that while the original bench- mark systematically underestimated retrieval performance as evidenced by Hit@ 10 improvements, nDCG@ 10 may not be well-suited for evaluation scenarios with expanded ground truth annotations. particularly when multiple relevant chunks are present.
J ALIGNMENT EVALUATION DETAILS
This section provides detailed formulations of the metrics used to evaluate retrieval performance, generation quality, and their alignment in RAG systems. We employ binary retrieval metrics (Hit@k), bin-bynary retrieval metries (nDCG@k), and GPT-4o-based binary evaluation for gen- eration assessment.
23

---

Published as a conference paper at ICLR 2026
Table 20: Retrieval nDCG@ 10 performance on BRIDGE for 25 retrievers evaluated on datasets, with improvements (1) over original benchmarks in subscripts. Best performance in bold.
Dataset
Method
Model
Avg.
NQ
Rec
Tech
BM25
0.25 0.97 1
0:27 0.02 1
0.32 -2.000444 L
0.30 0.30-00.3 4
0.22 not 1
0.17 0.07
0.35 -0a5 1
0.27 0.00 f
Sparse
SPLADE
0.46
0.45
0.43 4
0.43
0.27 -0.10
0.24-0.04
0.42
0.39 -006
Contriever
0.16 1
f
-0.04
-0.3
-0.6 1
DPR
0.26 1
0.35 -0.07
0.22 00005 1
0.18 -0.04
0.11 -0.03 1
0.08 -0.01
0.23 0.05 1
0.20 -003
Dim Roduction
0.36 0.98 1
0.22 0.01
0.32 0.06 1
0.27 0.07
0.11 0.05 1
0.12-0.03
0.21 0.06
1
0.23 0.03
SBERT
0.38 0.96
0.27
0.31 0.08
0.29
0.16
0.14
0.27 0.08
0.26
BPR
0.38 0.96 1
0.33 .0.03
0.34 -0.08 1.
0.33 -0.08
0.19 -0.66 J
0.16-0.04
0.33 0.09
0.29 2005
Dense
Raiss
0.44 f 0.36-0.03.
0.40
0.38
0.22-00094 J
0.19
0.34
0.33 4006
ANCE
0.41 0.07 f
0.34
0.41 0.08 1
0,38 -0.09
0.23 -0.11
0.20 -004 4
0.37 0011
0.35 4006
TCT-COL BERT
0.44 0.84 1
-0.3
0.41
0..6
1
0.39 .0.10
-0.99 J.
0.20
0.12.
0.35
Aggretriever
0.42 0,08 1 0.41 00.04
0.42
0,10
0.40
0.24 0.09
0,24 0,04
0.12 0,40
0.36 0.06
Arctic
0.43 0.10 1
0.49 0.05
0.12 4 0.50
0.54 0.09
0.31 0.12 1
0.35 0.05
0.13 0.49
0.44 0.07
TBERT
0.00 1
0.40
0.38
0007
0,38 00.08
0.25 0.07
0.19 0.02
0.39 0.10
0.34 -004
NBoost
0.41 f 0.44
1
0.44
0.22 0.0
0.12
0,38
BM25
MiniMM
0.42 0.07 f
0.44
0.42 0.10
0.43 00.99
0.26 0.09 1
0.22 0004 4
0.12 0.42
0.37 4005
0.40 1
0.43
-0.7
0.43 .007 4
0.27 -0.88 1.
0.22 -003
-0.1-0111
0.37
Rerank
MonoTS
0.42 0.08
0.00
0.45 0.08
0.46
0.30 0.30-0.07
0.25 -0.02
0.44 00.1
0.40
TBERT
0.43 0.07
0.42
0.41 -0.09
0.40 -0.09
0.26-0.11
0.22 0.23
0.40-0.12
0.36 -006
NBoost
0.43 1
0.46-0.03
0.47 10.10
0.45 -0.00
0.28 -0.1
0.25 00.04
0.44-
0.40 -0.0
ANCE
MiniLM
0.46 f
0.00
0.46
0.45
0.27
0.25
0.14
0.40
ELEC
0.44 0.08 T
0.45
0.46
0.44-6.09
0.27 -0.12
0.25 -004
0.43-0.13
0.39 -006
MonoTS
0.44 0.08 1
0.46 0.46-0.02
0.50 0.09 1
0.49 0.08
0.31 0..11
0.28 0.04
0.45 0.13 1
0.42 0.06
HyDR
0.31 0.14 f
0.49 0.06 1
0.32 0.03
0.29 0.29 1
0.10
0:21 0.09 1
0m3
0.31 0.07 1
Rewrite
BM25
Q2D
0.34 0.13 t
0.45 0.02
0.37
0.30
0.26
0.22
0.41
0.01
0.34
MuGl
0.36 0.13 1
0.51 0.00
0.39 0.00
0.35 0.01
0.28 0.05
0.23 0.06
0.43 0.01
0.36 0.04 1
Avg.
0.38 0.08
0.40-0.01
0.39 0,00
0,38 -0.0
0.23 0.005
0.20-0.01
0.38 0.07
0.34 0000
J.1 RETRIEVAL METRIC
@it@k. Hit@k k evaluates whether at least one of the top-k retrieved chunks contains the gold chunk(s). This metric is a binary metric that reflects success within the top-ranked candidates.
Hit@k 1(rankrel S <k),
where rank denotes the rank of the first relevant chunk, and 1 is the indicator function that returns if the condition holds, and 0 otherwise.
nDC@@k. nDCG@k accounts not only for the existence of gold chunks in the top-k retrieved chunks, but also their ranks. It assigns higher scores when relevant chunks appear earlier in the ranking, providing a position-sensitive measure of retrieval quality.
nDCG@k= DCG@k
IDCG@k'
where
2rel:
DCG@k
logg (i + 1)²
2rel; 1
IDCG@K= =
loga (i 1)*
rel, denotes the relevance score of the i-th retrieved document, rel, represents the relevance scores
sorted in descending order, and IDCG@k @k is the ideal DCG@k, representing the maximum possible DCG@k under the optimal ranking.
24

---

Published as a conference paper at ICLR 2026
Table 22: RAGAlign@ 10 score gains of BRIDGE for 1eem2..-7BBInstruue
Table 23: RAGAlign @ 10 score gains of BRIDGE for Gemma-2-99
Version
BEIR
ROBUSTQA
Avg.
Version
BEIR
ROBUSTQA
Avg.
MS
NQ
LLiee
Rec
Sci
Tech
Wri
MS
NQ
L
Rec
Sci
Tech
Writ
Original 0.58 0.71 [0.67 0.71 0.66 0.66 0.69 0.66
Original
0.60 0.74 0.74/0.73 0.73 0.68 0.68 0.72 0.69
BRIDGE 0.83 0.80 [0.71 0.75 0.75 0.76 0.73 0.76
BRIDGE
0.85 0.86 |0.76
0.78 0.76 0.76 0.75 0.78
J.2
GENERATION METRIC
We employ GPT-4o-based binary evaluation to assess the correctness of generated answers against ground truth annotations. The evaluation prompt structured to determine whether the predicted answer captures the core content of any ground truth answer, returning a binary True/False judgment. Table 25 presents the specifie prompt template used for this evaluation.
J.3 ALIGNMENT METRIC
We extend the RAGAlign metric from binary non-binary variables, enabling the computation of non-binary metrics such as nDCG@k and Recall@k.
RAGAlign @k k between Binary Variables. When both retrieval and generation metrics are binary, measure alignment the proportion of queries where both components agree in their evaluation outcomes:
RAGAlign@K 1 N N Σ METRICR, = METTRIGG]]
where N denotes the total number of queries in the dataset. METRICR; E {0, 1} is the binary retrieval outcome, and METRICG, {0, }} is the binary generation outcome for the i-th query. is the indicator function that outputs if the condition is satisfied, and 0 otherwise. This metric captures how consistently retrieval and generation evaluations align.
RAGAlign @k between Non-binary and Binary Variables. Point-biserial correlation a special form of Pearson correlation used when one variable is binary and the other is non-binary. This allows us to measure alignment between continuous retrieval metries (e.g, nDCG@k) and binary generation outcomes:
RAGAlign@k Rí - Ro
71
S.E(R) n(n 1)
where Rt and Ro are the mean METRICR for METRICG=1 and METRICG=0 respectively, S.E the standard error of METRICR, and n,, ni. n are the corresponding sample sizes. The point- biserial correlation can be computed the Pearson correlation between non-binary variable and binary variable coded as and
Result. To investigate whether the RAGAlign@ 10
RAGAlign @ 10 score gains over seven subsets of BRIDGE measured
Table 21:
gains on BRIDGE generalize other LLM genera- tors, we additionally evaluate the retrieval-generation alignment rate on two further generators, Qwen2.5-7B- Instruct and Gemma-2-9b-in in Table 22 and Table 23 Both models exhibit the similar trend as Llama-3.1 1--8--
by nDCG@ 10
Version
BEIR
ROBUSTQA
Avg.
NQ Life
Tech
Instruct, demonstrating that the observed gains are not tied to a single generator configuration. We will include these results in the revised version.
Original 10.16 0.38 0 0.39 0.43 0.28 0.34 0.33
BRIDGE J0.38 0.51 0.45 |..4 0.49 0.45 0.51 0.42 0.46
Table demonstrates the nDCG@ 10-based RAGAlign 10 scores comparing BRIDGE to the orig- inal benchmark. Consistent performance improvements are observed across all datasets, with MS MARCO showing the most substantial gain from 0.16 to 0.38. The RobustQA subdomains exhibit more moderate, but positive improvements across all areas. On average, the RAGAlign @ 10 score improved from 0.33 to 0.46, confirming that chunk label correction significantly enhances the accu- racy of retrieval performance evaluation.
25

---

Published as a conference paper at ICLR 2026
Table 24: Details of the model checkpoint and hardware used for implementation.
Model Type
Model Name
Hugging Face Checkpoinis & Official API Version
Hardware & Precision
Proprictary
GPT-4o-mini GPI-4o
got-fo-mini-2224-07-88 gpt-fo-2024-08-06
OpenAI API Default settings OpenAl API & Default settings
LLMS
Open-
Llama3.1-88 -88--nst.
cettalllama/Lllange 1-8B-Instrroc
NVIDIA L40S 48GB (x1) & BF16 NVIDIA LAUS 48GB (x4) & BF16
source
metulllama/LLannee 1.ammmmm--0---mmsttttt--
LLMs
Qwen2.5.72B Inst.
Qwen Instruct
NVIDIA L4DS 48GB (x4) & BF16
J.4
GENERATION PHASE
Our generation setup employs consistent approach across all systems to isolate the impact of retrieval quality. We utilize a single large language model Llama-3. 1-1-..1181 ----IBsttrut with a standardized prompt template that incorporates the top- 10 retrieved chunks context. This uniform generation configuration ensures that performance variations can be directly attributed to differences in retrieval quality rather than generation capabilities. The specific prompt structure enforces striet JSON formatting and prevents the model from incorporating external knowledge beyond the pro- vided context. For the generation prompt, see Table 26
K LLM IMPLEMENTATION DETAILS
Table 24 provides comprehensive details of model versions and hardware specifications used in our experiments. Open-source models are sourced from Hugging Face with specific checkpoint versions, and we detail the exact hardware configuration, GPU memory and precision settings used for each model deployment. Proprietary models are accessed via official APIs including OpenAI API and AWS Bedrock with documented API versions and parameters.
Table 25: Prompt for
binary evaluation.
Your task evaluate
of the PREDICTED ANSWER based the GT ANSWERs
Instructions:
Read the QUERY and then compare the GT ANSWERs and the PREDICTED ANSWER
Check
ANSWER includes any core content
there mulliple GT ANSWERS and the PREDICTED ANSWER includes the content of least one of (hem, oulput "True".
### QUERY: {quory}
HOH GT ANSWERS: (ground truths)
### PREDICTED ANSWER: {prediction)
Strictly aulput False
Table 26: Prompt for RAG answer generation.
Answer the given QUERY only using the information provided in the Multiple CONTEXTss Do not include any assumptions, general knowledge, information found Multiple CONTEXTs.
QUERY: (query)
Multiple CONTEXTs: (retrieved documents)
Do provide any explanation additional text.
Respond strictly following JSON format:
**no relevant information** found CONTEXTs:
{" ("Answer": "No yelevant information found.) ")
rtelevant information" exists CONTEXTs, short form:
"nnswe"" "your answer")
26

---

Published as a conference paper at ICLR 2026
Table 27: Prompt for initiating the first-round debate, containing both initial stances.
System Instruction
You are (agentl) ), and your task is to determine whether the given chank FULLY answer to the query with AT LEAST ONE the answers. There is also other agent, {agent2). assigned the same task you. You are provided query. answers, and chunk Kquecy>< ese//eeyy> <ans>>< </ans> lugs, and <doc></ <oo/>>>/>>>>>//////////////>>>>> doz> lags respectively. There can be multiple answers to the query, each listed with an ordered number. You can access argument and (agent2) argument. There are also important guidelines to help your decision in <quide><//ee //quide> and keep that mind for whole discussion process.
discussion history
[agen12) in <hi ary>s/ /hyy tags. You can refer your previous
Target Input
<query>
[query}
query>>
<answer>
[answer)
</answer>
<doc>
[chunk)
</doo>
<histoor>>
(agentA): J: Yes. The chunk contains complete information to construct at least one answer to the target query.
{agentB): J: No. chunk does not contain complete information any to the target query.
</history>
Debate Guidelines
You have carefully read query and its and fully understand the and fully understand it, With your understanding of the chank and answers, you have to analyze the discussion history and engage critically discussion. 'hen, you think critically whether the chunk can fully justify any answers with all the guidelines sguide>> souidd>>//guiieee /guide>.
each Afterthat, read the given chunk
<guide de>
1. The chunk hus the scupe the scope the The chunk with specilic example cannol, justify general delinition, and vice versa.
2. The chunk
without contradiction with 3. The chunk must directly provide the answer, without implications, oppositional logic or common-scusse reasoning. chunk musi express same concept is not sufticient.
contain key contents of the and present consistent
same intent, scupe and practical instruction. simple match on theme terminology
5. Base your decision solely on whether chunk provides complete information for the answers, regardless of any additional information.
6. Do mark the answer supported hased surface level similarity, for example both mention word, without checking for matching intent and guidance.
7. Do
correcneess, relevance query. consistency between answers.
8. Each answer must be evaluated independently.
</ guide>>
Explain your reasoning process step by step, including how you interpreted the chunk, and engaged with the other agent's position. TF applicable, include reference sentence from the chunk to support your reasoning. Conclude with a cleur justification of your final decision. Determine whether the piven chunk is fully supporting any of the answers "response"
yes: The chunk contains complete information to construct at least one answer to the target query. contain complete information for any answer target query.
10: The chunk
Output Template
Provide your output as a properly formatted JSON object. No additional explanation.
"reference": ('reference sentence If exisss"" ...]. "reason": "Explain your reasoning process step by step. Conclude with clear justification of your final docision. (Max 100 words)". "response"; "yes or no"
27

---

Published as a conference paper at ICLR 2026
Table 28: Prompt for the subsequent-round debate, incorporating the discussion history from the previous round.
System Instruction
You are (agentl) J. and your task is determine whether given chank FULLY answer to the query with AT LEAST You are provided query, answers, and chunk in <query>< ceeeers///uuerry /qquery> lags, <ans>> </an>> lags, and <doc></ doc> lags respectively. There can be multiple answers to the query, each listed with an ordered number. You also access the discussion history with [agen12] in wtttrrrs//itttttoeeeeettt argument and (agent2) argument. There are also important guidelines to help your decision in <guide><////eo.oo </qui des tags and keep that mind for discussion process.
the answers.
There is also other agent, {agent2), assigned the same task
lags. You refer your previous
Targtt Input
equezy>
{query}
</ query>
<answer>
[answer}
</answer>
<doc>
{chunk]
</doe>
<historr>>
(agentA): ): (agentA reasoning)
[agentB]: {agentB reaasoing
</history>
Debate Guidelines
You have to carefully read the query and its answers and fully understand the core content of each answer. Afterthat, read the given chunk and fully understand With your undorstanding the chonk and answers, you have to analyze discussion history and engage critically with the discussion. Then, you have think critically whether the chunk fully justify any of with all of the guidelines in eguide><<d <e>></guide> <guide de>
1. The chunk has the same as the scope of the answer. The chunk with a specific example camnot justify generul definition, and vice versa.
2. The chunk must contain the key contents of the answer, and present consistent context without contradiction with the answer.
3. The chunk must directly provide the answer, without implications, oppositional logic or common-sense reasoning.
4. The chunk
express same concope with the same intent, scope and practical instruction. simple match theme terminology is not sufficicnt.
answers, regardless any additional information. 6. Do not mark the answer as supported based on surface level similarity, for example both mention same word, without checking for matching intent
Base your decision solely whether chunk provides complete information
guidance.
7. Do not assess answer correctness, relevance query, consistency between answers.
8. Each answer must be evaluated independently. e/quide>
Explain your reasoning process step by step. including how you interpreted the chunk, and engaged with the other agent's is position. If applicable, include reference ventence from chunk support your reasoning. Conclude with clear justification your decision. Determine whether the given chunk is fully supporting any of the answers in "response"
yes: The chunk contains complete information to construct least one answer to the target query.
Do: The chunk does contain complete information for any answer the target query.
Output Template
Provide your output as a properly formatted JSON object. additional explanation.
{ ("reference": ['reference sentence lifexsiss".... "reason": "Explain your reasoning process step by step. Conclude with a clear justification of your final decision. (Max 100 words)", "respoose"" "yes or no"
28

---

Published as a conference paper at ICLR 2026
Table 29: Adjudicator prompt to resolve the disagreement between two agents.
System Instraction
You are an ADJUDICATOR tasked with making final determination chunk-answerr-uurry alignment. You are given a query, answers, a chunk, and discussion history between two evaluater agents with upposing views. Determine whether the chunk contains sulficient infor- to fully substantiate AT The answers are propesed responses to the query.
candidate answers the given query.
chunk should provide complese evidentiary support for constructing/validalit at least one answer.
You are NOT evaluating answer quality. only answer-query grounding.
Target Input
</query>
[answer]
</answer>
<doc>
{chunk}
</doe>
<hiatorys
{bistory}
</history>
Adjudication Guidelines
You have carefully read query and its and fully understand the and fully understand it. Summarize both agents' positions and evaluate their reasoning quality. Assess how well cach agent followed the evaluation criteria below in pouide><//giide Then, you have critically whether the chunk can fully justify any answers with all of the guidelines in suuide></guides <qui de>
each Afterthat, read the given chunk
You musl stick all the guidelines below for your decision:
I. The chank has the same scope as the scope of the answer. The chunk with a specific example cannot justify general definition, and vice
2. The chunk must contain the key contents of the answer, and present a consistent context without contradiction with the answer.
chunk must directly provide the answer, without implications, oppositional logic or commoon sense reasoning.
4. The chunk must express the same concept with the same inteat, scope and practical instruction. simple match on theme terminology
is not sullicient.
5. Base your decision solely whether the chunk provides complete information for the
regardless of any additiionl information. 6. Do not mark the answer as grounded based on surface-level similarity, for example both mention the same word, without checking for matching intent and guidance.
</ guide>
agents' arguments, and applied evaluation guidelines. Conclude with clear justification of your final decision. Determine whether the given chunk is fully suhstantiating any of the answers in "response". yec The chunk contains complete information to
Explain your reasoning process step by step. including how you interpreted the chunk, engaged
least n0: The chunk does not contain complete information for any answer to the target query.
to the target query.
Output Template
Provide your output a properly formatted ISON object. No additiioal explanation.
{ ("refeerncc"": I'reference sentence if exissss" ...]. "reason" " "Explain your reasoning process step by step. Conclude with a clear justification j your final decision. (Max 100 words)", "response": "yes no")
29


Reviews:
[{"review_id": "lviICzg5eq", "reviewer": "Reviewer_z9oR", "summary": "The paper proposes DREAM, a debate-based framework for completing missing annotations in information retrieval (IR) benchmarks. It introduces a multi-agent relevance assessment process where two LLMs with opposing stances critique each other to reach agreement, using disagreement as a natural signal for human escalation. Building on DREAM, the authors create BRIDGE, a refined IR benchmark that fills missing relevance labels and enables fairer evaluation of retrieval and retrieval-augmented generation (RAG) systems.", "questions": "See above", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper is well-written and easy to follow\n- The method itself is intuitive and fitting for the problem itself and seems to outperform baselines and work well empirically.", "weaknesses": "- Could also compare with this work and related works which use multi-agent debate to improve performance of RAG systems and contrast with these: https://arxiv.org/abs/2504.13079, https://arxiv.org/abs/2501.00332 \n- Could clarify on how the quality of the benchmark is impacted by the choice of number of agents and which model families these models come from.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4VSFjKiZdL", "reviewer": "Reviewer_MSma", "summary": "This paper proposes DREAM, a multi‑agent debate framework for completing missing answer‑aware relevance annotations in IR benchmarks. Specifically, two LLM agents are initialized with opposing stances, debate for up to two rounds, and auto‑label a query–chunk pair upon consensus; persistent disagreements are escalated to human annotators together with the debate history to aid adjudication. DREAM targets accurate, low‑cost query–chunk relevance labeling and reports higher accuracy than single‑agent LLM‑as‑judge and confidence‑based escalation baselines.\n\nUsing DREAM, the authors construct BRIDGE, a refined benchmark built from subsets of BEIR and RobustQA that substantially reduces unlabeled relevant “holes.” BRIDGE adds 29,824 previously unlabeled relevant chunks—a 428% increase over the originally annotated 6,976 gold chunks—bringing the total to 36,800. The paper shows that such holes introduce systematic evaluation bias that can underestimate retriever effectiveness, and it introduces RAGAlign to measure retrieval–generation alignment in RAG systems. Results indicate stronger retrieval–generation alignment on BRIDGE than on the original benchmarks, and ablations on the expert‑annotated subset support the choice of two debate rounds and the utility of providing debate history for AI–human synergy.", "questions": "Please see weaknesses.\n\nAnother following up question considering W.1 and W.2 together: \n1. If you increase debate diversity—by adding more agents, using heterogeneous base models, or allowing non‑zero temperatures—does the rate of spurious consensus decrease, and does end‑to‑end accuracy improve at comparable escalation cost?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper identifies and addresses a critical bottleneck in existing IR benchmarks: the incompleteness (i.e., holes) of current relevance-annotated datasets leading to unreliable retrieval performance evaluation of the RAG system.\n2. The proposed multi-agent debate annotation pipeline is intuitive and clearly explained. \n3. The constructed BRIDGE benchmark uncovers 4 times previously unlabeled relevant chunks compared to the original gold chunks with relatively low cost, which should be meaningful for the IR community.", "weaknesses": "1. **Homogeneous, 2-agent setting for multi-agent debate**. In DREAM, both agents are Llama-3.3-70B-Instruct with temperature=0. Although this is a standard minimal setting for multi-agent debate, what if we use more than 2 agents or heterogeneous-model (i.e., different LLMs for each agent) to increase the diversity during debating? Will this bring more accurate annotation?\n2. **Agreement treated as reliability without analyzing \"wrong-but-agree\"**. There is discussion of persistence of agreement across debate rounds in section A.5. However, there are no analysis to quantify the rate of spurious consensus (cases where agents agree yet the consensus contradicts expert labels).\n3. **RAGAlign analysis tied to a single generator configuration**. The RAGAlign@K analysis in section 5.2 only focus on Llama3.3-8B-Instruct. It is not shown whether the observed retrieval–generation alignment gains on BRIDGE hold for other LLM generators. Also, there seems to have a generator version discrepancy between Figure 3 (Llama‑3.3‑8B) and Appendix I.4 (Llama‑3.1‑8B).\n4. **Will the more accurate chuck annotation help generation performance?** While RAGAlign improves, the paper does not report whether denser, more accurate chunk annotations translate into better end‑task generation metrics under the same generator and prompts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XtXTjcba84", "reviewer": "Reviewer_M8G8", "summary": "The focus of the paper is on IR evaluation, where the authors propose an LLM multi-round debate solution. The solution is motivated by the fact that current IR benchmarks have \"holes\", i.e. data that is not labelled as relevant but might be relevant. Past work has used LLMs for relevance assessment but used single judge, which suffers from overconfidence. The authors point out that this hurts the performance of hybrid systems that send uncertain cases to humans: to do this well, models must be calibrated/ambiguity-aware. \n\nThe paper introduces DREAM to use adversarial debate to decide when to escalate to human judgment. The debate is based on conflicting stances, only escalating when agents agree. They introduce a benchmark based on existing datasets, using DREAM to reduce evaluation holes.\n\nThe paper finds that DREAM reduces the escalation ratio while improving accuracy on the evaluation set. Moreover, DREAM can be used to refine real-world IR datasets and fill holes, lowering the human cost by only human-annotating uncertain cases. The refined real-world IR data is used to evaluate systems for RAG, with improvements to a RAG model.", "questions": "- Order-dependence: LLMs are often prone to sycophancy and there's a documented trend of models picking whichever answer goes first. Does it matter which agent goes first? It would be good to have an ablation showing stance order doesn't matter. \n- How do you evaluate consensus? Are models instructed to format answers in an extractable way? \n- What was the reason for human disagreement on human eval? \n- Have you thought about distillation? Past work, e.g. https://arxiv.org/abs/2402.01620 distills multi-agent interactions into a single model. In principle, you could distill the better classifications from the DREAM debate pipeline into a single judge model that at test time produces a single decision. \n\nI'm willing to increase my score if these weaknesses/questions can be addressed.\n\nOther minor points:\n- L174: the opponents -> the opponent's\n- L092-094: the common explanation...: this needs a citation. I believe another more common explanation for this is that the models are ignoring their contextual knowledge in favor of the parametric knowledge (https://aclanthology.org/2021.emnlp-main.565/, https://aclanthology.org/2022.emnlp-main.146/, https://aclanthology.org/2023.findings-emnlp.968/).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Strong intrinsic results: the paper shows that DREAM greatly reduces the amount of human annotation needed for the same accuracy\n- Downstream utility: the paper demonstrates the utility of DREAM on augmenting a real IR dataset, where human annotation cost would be high with baseline approaches. \n- Human evaluation: the evaluation set is vetted by human expert annotators\n- Baselines: the method compares to LLM-only and confidence-based baselines that support its claims.", "weaknesses": "- No discussion of latency cost: compared to LLM-as-judge baselines, this debate approach is much more expensive, requiring multiple model calls across multiple rounds. While the method saves human cost, the trade-off here should be discussed. \n- Novelty/major missing related work section: the paper is missing most related work on multi-agent debate, including https://arxiv.org/abs/2309.13007, https://arxiv.org/abs/2305.14325, https://arxiv.org/abs/2504.13079 which have covered stances, divergent thinking, and evidence conflict. The application of debate to IR evaluation seems novel but the debate methods themselves less so. \n- Dataset size: the size of the evaluation dataset seems quite small compared to the original datasets (BEIR and RobustQA) which have tens of thousands of examples. \n- This paper would benefit from interaction with the extensive literature on selective prediction, which has the same motivation (e.g. https://aclanthology.org/2022.findings-acl.158/, https://arxiv.org/abs/2303.16857, https://arxiv.org/abs/2402.15610) \n- Comparison to LARA would be easier to understand as a plot with different confidence thresholds/escalation ratios on the x axis", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lRPFSIgXAS", "reviewer": "Reviewer_58tj", "summary": "The paper aims to address missing relevance annotations (“holes”) in IR benchmarks that distort evaluation and hinder RAG alignment. It proposes DREAM, a multi-agent debate framework where two LLMs take opposing stances (Relevant vs. Irrelevant) and iteratively critique each other until reaching consensus or escalating to humans. Using this method, the authors rebuild benchmarks into BRIDGE, filling large annotation gaps and revealing that many retrieval improvements were previously underestimated. The results show DREAM effectively automates reliable labeling and that completing these “holes” leads to more accurate and faithful IR and RAG evaluations.", "questions": "1. By default, the debate runs with two rounds of alternating arguments (R=2). I’m curious whether the authors explored what happens with more rounds — for instance, does extending the debate yield deeper reasoning or just redundancy? Some analysis on how performance scales with R could offer useful insights into the debate dynamics.\n\n2. Since both agents (m₁ and m₂) are initialized from the same Llama3.3-70B-Instruct model, I wonder if there’s any risk of “shared bias” or reinforcement from their common pretraining knowledge, even though they take opposing stances. Would using models from different sources (e.g., different architectures or training data) introduce useful diversity, or might it hurt the framework’s stability and its strong human-level accuracy?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper highlights an often-overlooked issue — that retrieval performance itself has been misjudged because of annotation holes in IR benchmarks. This brings a fresh perspective to understanding the mismatch between retrieval and generation performance.\n\n2. I really like how the multi-agent debate setup helps counter problems like single-model overconfidence or failed confidence-based escalation, where easy cases get escalated and truly hard ones slip through.", "weaknesses": "One thing I wonder is whether the framework considers multi-hop reasoning cases — where the answer can only be inferred by combining multiple chunks. For example, if one passage says “Jack’s mother is Jane” and another says “Jane’s husband is Bob,” neither chunk alone directly answers “Who is Jack’s father?”, but together they clearly provide the evidence. It’d be interesting to see how DREAM handles or could be extended to capture this kind of compositional relevance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)