Jury-and-Judge Chain-of-Thought for Uncovering Toxic Data in 3D Visual Grounding
Authors: Kaixiang Huang
Qifeng Zhang
Jin Wang
Jingru Yang
Yang Zhou
Huan Yu
Guodong Lu
Shengfeng He
Venue: NeurIPS 2025 poster ()
Keywords: LLM
Chain-of-Thought
3D Vision Grounding
LLM-as-a-Judge
Primary area: deep_learning
TL;DR: We propose Refer-Judge, a Jury-and-Judge Chain-of-Thought framework that leverages MLLMs to uncover toxic data in 3D visual grounding

Abstract: 3D Visual Grounding (3DVG) faces persistent challenges due to coarse scene-level observations and logically inconsistent annotations, which introduce ambiguities that compromise data quality and hinder effective model supervision. To address these challenges, we introduce Refer-Judge, a novel framework that harnesses the reasoning capabilities of Multimodal Large Language Models (MLLMs) to identify and mitigate toxic data. At the core of Refer-Judge is a Jury-and-Judge Chain-of-Thought paradigm, inspired by the deliberative process of the judicial system. This framework targets the root causes of annotation noise: jurors collaboratively assess 3DVG samples from diverse perspectives, providing structured, multi-faceted evaluations. Judges then consolidate these insights using a Corroborative Refinement strategy, which adaptively reorganizes information to correct ambiguities arising from biased or incomplete observations. Through this two-stage deliberation, Refer-Judge significantly enhances the reliability of data judgments. Extensive experiments demonstrate that our framework not only achieves human-level discrimination at the scene level but also improves the performance of baseline algorithms via data purification. Code is available at https://github.com/Hermione-HKX/Refer_Judge.

Jury-and-Judge Chain-of-Thought for Uncovering Toxic Data in 3D Visual Grounding
Kaixiang Huang'
Qifeng Zhang'
Jin Wang!*
Jingru Yang*
Yang Zhou'
Huan Yu²
Guodong Lu
Shengfeng He² SSingapore Management University
'Zhejiang University
Carnegie Mellon University
Scene-level Message
This is brown armehair. It next to another armehair.
Find the Only Object matches the description!
M
This white It is brown table.
Taxic Data?
Solid Data?
Paradox!
Amihguity!
The Next To relation Symmetrical. Description matches both chairs.
Refer-Judge
Multiple objects fit the description, uniqueness cannot be guaranteed
Figure I: 3D visual grounding aims to localize a unique target object in scene given a single referring expression, but existing datasets often contain ambiguous paradoxical samples that hinder this goal. We propose Refer-Judge, an agentic, deliberation- based framework that detects and filters toxic data through structured pulti-perspective reasoning and corroborative refinement.
Abstract
3D Visual Grounding (3DVG) faces persistent challenges due to coarse scene-level observations and logically inconsistent annotations, which introduce ambigui- ties that compromise data quality and hinder effective model supervision. To address these challenges, we introduce Refer-Judge, novel framework that har- nesses the reasoning capabilities of Multimodal Large Language Models (MLLMs) identify and mitigate toxic data. At the core of Refer-Judge Judge Chain-of of-Thuught paradigm, inspired by the deliberative process of the judicial system. This framework targets the root causes of annotation noise: ju- rors collaboratively assess 3DVG samples from diverse perspectives, providing structured, multi-faceted evaluations. Judges then consolidate these insights using Corroborative Refinement strategy, which adaptively reorganizes information to correct ambiguities arising from biased or incomplete observations. Through this two-stage deliberation, Refer-Judge significantly enhances the reliability of data judgments. Extensive experiments demonstrate that our framework not only achieves human-level discrimination the scene level but also improves the performance of baseline algorithms via data purification. Code available https://githun wtt.cooo///ce/e http//ermiiinn HwweooneeeeeHHeeee TirX/Refer_Judgee
Jury-and-
"Corresponding author. dujcomOzju.ee edu.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
3D Visual Grounding (3DVG) aims to localize a unique target object within complex 3D scene based on a natural language referring expression. As a critical bridge between human perception and machine understanding of real-world environments, it has driven the development of large-scale datasets such as the widely adopted ScanRefer 7 laying the groundwork for algorithmic advances.
To support generalizable learning, 3DVG datasets must capture diverse scenes and densely anno- tate referable objects. However, meeting this demand places heavy burden on annotators, who must repeatedly write descriptions while navigating incomplete and often ambiguous scene repre- centations-namlle sparse 3D point clouds and disjointed 2D frames. This process has led to non-negligible number of toxic samples, whose descriptions degrade training quality and compromise evaluation reliability. These toxic samples typically arise from two sources: 1) Logical paradoxes. where the description is internally inconsistent and fails to isolate a single target (c.g.. "... next to another ... creates symmetrical ambiguity); and 2) Referential ambiguities, where vague or under-specified descriptions match multiple similar objects. Figure 1 illustrates these cases: the left side shows a logically paradoxical reference, while the right highlights under-specification due to insufficient scene observation.
Detecting such toxic data requires deep, multimodal reasoning across both 3D scene structure and natural language. Yet large-scale human validation is costly, time-consuming, and difficult to
scale [26 (4]. Recent advances Multimodal Large Language Models (MLLMs) [ 31 33 B 20
have shown remarkable capabilities in perception and reasoning [44 35). opening up new possibilities for data evaluation. For example, SceGround [T]] enhances MLLM-basek understanding using adaptive 2D renderings and achieves competitive zero-shot performance on 3DVG tasks. However, current research largely treats MILMS as perceptual engines or grounding backbones 49 16 9.
overlooking their potential to identify hidden toxic samples. This gap highlights a critical yet underexplored direction: employing MLLMs for principled data quality assessment in 3DVG.
Meanwhile, MLLMs are increasingly recognized as capable evaluators. JudgeLM (55], for instance, fine-tunes LLMs assess output reliability, while others [50 6 45
explore Chain-of-Thought (CoT) prompting 37 to improve aligmment with human judgment. Despite encouraging results, these approaches are generally limited pairwise comparisons or simple content scoring, and they falter when faced with the structural complexity of scene-level reasoning. Successors like CoCoT 45 and GoT [5] target fragmented, open-domain tasks (c.g., single questions or isolated image-text pairs) and are ill- ill-equipped to trace the nuanced origins of toxic data in 3DVG. The lack of a targeted, coherent judgment mechanism capable of deep multimodal reasoning leaves toxic sample detection in 3DVG an unsolved problem.
To address this challenge, we introduce Refer-Judge, novel agentic deliberation-baser system for uncovering toxic annotations in 3DVG. Unlike prior MILM evaluators, Refer-Judge adopts a Jury-and- Judge Saain-of-Thhugll paradigm inspired by judicial deliberation. Given scene and its corresponding referring expression, multiple Jurors analyze the sample from four core perspectives: Logic, Consistency, Distinguishability, and Ambiguity. This structured division of reasoning enables jurors to identify logical contradictions and ambiguities from diverse angles, forming a comprehensive diagnosis of sample quality.
Mirroring the real-world jury system, this division of reasoning promotes both coverage and spe- cialization in evaluating heterogeneous scene-level inputs. The outcomes from the jurors are then passed to a panel of Magistrate Judges, each responsible for consolidating feedback within a specific dimension, ensuring internal coherence and resolving intra-aspect contradictions. To synthesize a final verdict, one District Judge performs high-level arbitration through targeted re-evaluation. Central to this stage is our proposed Corroborative Refinement strategy, which goes beyond majority voting or self-consistency heuristics [34] Instead of relying on static consensus, Magistrate Judges adaptively reorganize visual and textual evidence based on juror insights, retrieving auxiliary context and correcting uncertain reasoning caused by blurred or incomplete observations. This process yields judgments that are both context-aware and resilient to scene complexity. By integrating multi- role, multi- puuttipperspectvve reasoning without reliance on aase-specific fine-tuning or external perception modules (e.g., object detectors or renderers), Refer-Judge delivers robust scene-level assessments. Extensive experiments demonstrate that Refer-Judge achieves human-comparable performance in identifying toxic samples, enhancing the quality of supervision, and advancing the reliability of 3DVG benchmarks.

---

In summary, our main contributions are threefold:
We identify and systematically analyze the prevalence of paradoxical and ambiguous samples in existing 3DVG datasets, and present Refer-Judge, novel evaluation method capable of uncovering toxic annotations without reliance on task-speeific fine-tuning or auxiliary perception modules.
We propose Jury-and-Judee Chain-of- Chain-of-Thought approach, wherein jurors perform targeted, multi- perspective reasoning and a panel of judges, culminating in high-level refinement, delivers robust, structured judgments through our Corroborative Refinement mechanism.
We validate the effectivvness of Refer-Judge through extensive experiments, showing both its alignment with human judgments and its ability to improve baseline 3DVG model performance when trained on filtered data.
2 Related Work
3D Visual Grounding and Multimodal Large Language Models. 3DVG focuses on localizing unique target object in a 3D scene based on natural language referring expression. Early approaches primarily adopt a detection-then-matte pipeline, emphasizing improved alignment between visual
(36 29]. With the rapid advancement of Large Language recent efforts have explored the potential of MLLMs for 3DVG LS I.. For instance, 3D-LLM [IS] integrates point cloud representations into LLMs and shows strong performance across diverse 3D reasoning tasks. SeeGround [19 leverages dynamically rendered 2D views to enhance open- -oce-vocbbulary grounding, achieving competitive zero- shot results. And CoT3DRef interpretable Seq2Seq staged prediction. However, while these approaches expand the capability of MI.LMs, they largely overlook the quality of the underlying datasets. Paradoxical and ambiguous annotations, often caused by human error or limited scene observability, remain unaddressed and continue to degrade both model training and evaluation reliability.
and linguistic modalities (7 |2 51| 46
47
Models (LLMs)
tasks|49 149
19
28
further introduce the idea of CoT into 3DVG to form a more
LLMs as Evaluators. A growing body of work explores the use of LLMs as evaluators, often referred as LLM-as-a-Judge, to approximate human judgment in various domains [4 43 example, LLLLM-as-a-Jude-- [6] evaluates MLLMs' recision-making highlights their limitations in handling fine-grainee reasoning. FINCON 43 applies a multi-agent LLM setup to assess financial risk, while -.G-rrodddrr [40) uses MLLMs to score candidate views in 3DVG, enhancing zero-shot grounding performance. Although these studies demonstrate the promise of LLMs as evaluators, most are restricted to shallow or fragmented inputs, such as isolated prompts or image-text pairs. The use of MILMS for structured, scene-level evaluation in complex multimodal tasks like 3DVG remains relatively unexplored.
For capabilities across tasks and
Chain-of Tfouhhh Reasoning, Ohain-of-Thoughe (CoT) prompting enhances LLM reasoning by encouraging step-by-step inference (37] Initial methods such as vanilla CoT introduce sequential logic to improve answer reliability. More advanced techniques like Tree-of-The ofeh-uh- (ToT) 21),
Graph-of- Traph-of-Thought (S), and Hierarchy- of Thought ()] enable models to explore multiple reasoning paths or build graph-based reasoning structures. In multimodal settings, extensions such as G-Co' 23 and MC-CoT (38] combine visual and textual modalities for more comprehensive analysis. Despite these advances, most CoT-based methods are tailored for narrow-scope tasks, including arithmetic reasoning or short image-text question answering [53 48
45
50, 52). Their
refinement strategies generally involve simple path selection, answer re-ranking, or majority voting (5) 34 30 27 24 These mechanisms are insufficient for evaluating rich, scene-level content where ambiguity stems from complex visual and linguistic interplay. In contrast, our proposed Refer-Judge enables distributed reasoning across multiple perspectives and introduces corroborative refinement for robust scene-level judgment in 3DVG.
3 Methods
To address the issue of toxic samples in 3D visual grounding (3DVG) datasets, we propose Refer- Judge, a novel Jury-and-Judge Chain-of-Thoughl framework that enables structured, scene-level reasoning over multimodal inputs. We begin by formulating the I.M-as-aaJudge task in the context of 3DVG data exploration and present an overview of our framework in Section 3.1 Section 3.2
details our Jusy- distributed evaluation strategy, which decomposes the assessment into multiple
3

---

Zugical
J
Description
D
CCCO
Description Group
Confilonc
Cooconalidatto
Andy
D
3
thir
Approgation
Fure
Dosgis
July
Mensae
3D Scene
jull
Eshaenes
Eviguooess
Final Judgment
thr
ranonale
D
J sag
raturto yrnboo penados
Retonale the
Mensag unl jasn
Ratioade
Antiqqiit
Raskmack
Figure 2: Overview of Refer-Judge. Jurors evaluate the scene and referring expression across four complementary dimensions. Their judgments then refined and integrated generate final scene-level judgment.
aspects. Section 3.3 describes how the Judge module refines and integrates the reasoning paths to produce final judgment.
3.1 Problem formulation and method overview
Consider an annotated 3D visual grounding (3DVG) dataset consisting of a 3D scene, where a set of images I from multiple viewpoints represents different regions of the environment, and natural language description D refers to a specific target object Or- Our ultimate goal to construct an agentic, deliberation-based judgment system fASLLas (I, D) that robustly and effectively identify toxic data caused by paradoxes and ambiguities. The system outputs discrete quality scores to reflect the reliability of each data sample, enabling a clear and interpretable evaluation of 3DVG data quality 3
In contrast to prevailing LLM-based evaluators that focus solely on analyzing text-image pairs with limited complexity, the proposed Refer-Judge approach towards multifaceted CoT analysis for the scene-level multimodal information, thereby enabling comprehensive data judgment. As demonstrated in Figure jurors (Ji, Je. Ja. Ja) in Jury Jjury are employed in parallel to access 3DVG data from four key perspectives: logical, consistency, distinguishabiiity, and ambiguity. To further enhance judgment reliability, the Judge system Jjudge applies two Magistrate Judges, J and Jmmgg tasked with aggregating and refining the outputs of the corresponding jurors across multiple image-description pairs. These judges adaptively identify uncertain or conflicting sub-evaluations and reorganize them by referencing more consistent information, thereby mitigating errors caused by limited viewpoints or incomplete observations. Finally, a District Judge Jois integrates all streams of reasoning and adjudicates the final quality score. Notably, due Refer-Judge involving multifaceted assessment, we observed an interesting CoT phenomenon, where the same thought can be shared across different inputs and serve distinct judgment processes. We term this phenomenon twin thought, as marked in Figure
3.2 Jury Reasoning
Aligned with the common causes of annotation errors in 3DVG datasets, the Refer-Judge initially introduces four jurors to comprehensively evaluate the heterogeneous scene-level information, as shown in Figure 3 Specifically, the textual modality focuses on identifying potential logical paradoxes within the referring expression, while the visual modality then uncovers ambiguity arising from incomplete or biased observations.
Logical Juror. The natural language referring expression is central to identifying the target object within the scene. To prevent self-contradictory or logically invalid descriptions, we introduce logical Juror Jj to assess the logical soundness of the input description. Given that logical reasoning is fundamental ability of large language models, and 3DVG descriptions typically short and simple, we adopt a direct strategy based on in-context learning to generate both judgment score A, and rationale Ri, formulated as:
(Ai, Ri) D)
(I)

---

Logioal Juror
Consistency Juror
comerneeennnn Description
Reter
specific ofject
sentences Description Group depiet
from geven Descriptiors
object, judging the logical uniloriity descriptio,
Duscriptionn
Monddyy logioll paradox the deso
descripiios.
scts at tabic by another chelt.
The desriction runch analpii x. This chairs B
2: There is char that under table & the bettom left maa..
Deacription G
R.
filtering
Score
Rationall: D,, 11 and 2)
Scone
chair the table
Agric the loft lesaNaationn
and L Score
This is traswoorby..
deseniriin table
cfeer i next chat then the chee' & also claic.6 ixporsabbee unipuell identity the tright chol.
Aetloneric Siree the rerd redetoo symmmttrical
Agrce on relotion with
1) and iJ;s &Z Score
> -onsider the ofeee chull oear to in D. the deccrorice ( stso by another chair solldd
door
Distinguishebility Juror
Ambiguity Juror
Scone
A.R
Rarlonanle
Peix Thongeh
Rulated oljects
A.R.
present The reletionctip
ludging how
Description
P
Rationae:
accuraiely manches the
description..
8
matchee the Image...
description This chale. table by smother
fulflling both decription.
The contrinss
A. R.
Hiltering
A.R
cosmuree description
p
Magistrate Judge of Ambiguity
This sttets at table by
A. eatiorades correlations and inconsletndiee. deternine anomaloue entiiis
Suspicious tD: UL. 1L.,4... S..
Scone R.R
Magistrate Judge of Distingnishubiiity
From
Conrabrratvee
(esspizion:) aisistnce.j .)
R
P
RR
A.
R
corne rationalies es,
and issonsistencios, detarmine Corroborative Pair Set
Felinement
A.R
District Judge
For each pair. the irformation assistance entry chould help correct the problematic entry...
A.R R
Corroborative Rain
Emanee stischims 1.
A.R.
"Consisteny"
Twie Thought
A.
"Dietagilhhbiiity" "Ambiguity"
SCORE
①
Figure 3: The detailed architecture of Refer-Judge. Submodules, distinguished by color, illustrate the hierarchical reasoning structure and information flow from parallel analysis to final decision maaking within the Jury-and-Judge Chain-of-Thone -Too--hf--houht..
where P/ is the task-specific prompt provided to the juror J1-
Consistency Juror. Beyond assessing the target description D isolation, we further introduce a consistency evaluation that leverages alternative descriptions of the same object across the dataset [7], denoted as Description Group G [) .... D }. These descriptions serve as cireumstantial evidence to evaluate the consistency of D, helping to prevent potential paradoxes across data entries.
The Consistency Juror Jo is designed as two-step ffain-of-thougeee process. Since expressions within G may themselves be logically flawed, the first filtering step J evaluates the internal consistency of this group. Removing descriptions with low confidence scores A', through a threshold thra- resulting in refined subset g Next, J performs cross-validation between the description D and g,, producing the consistency score Ag and rationale R.- The process is formulated as:
(A,, Jé = i [1,2], AG E A,, A,, > Uhra}
(Ac, R.)= J: D)
where pf and p denote the prompts used by Jf and J respectively. Ay and R, are the confidence score and rationale sets for each candidate sentence in the description group.
Distinguishability Juror. Among all scene-level images I, the subset of views that contain the referred object or, denoted as Ipos = .... [I,.....I?, C I, serves as the positive set essential for assessing distinguishability. Specifically, we align the RGB frames with annotations to identify images where is visible, ensuring that all views in Ipos depict the intended referent. The verification process by human experts further involves: 1) checking whether the object described appears the image, and 2) confirming that the expression refers to unique object.
5

---

Mimicking human observation, the Distinguishabiltty Juror Ja also applies a two-stage CoT. jointly evaluating existence and uniqueness derive a comprehensive distinguishability assessment. Initially, pair, resulting in set of existence scores A. and rationales Re- Based on thre,, Ja filter out a subset Zyo, that meets the conditions. Subsequently, Já further evaluates whether each remaining image contains unique object satisfying the description, producing uniqueness scores A., and rationales Ry, In summary, the reasoning chain is expressed as follows:
Já traverses each image-descripee
(Ae, Re) - ( (Já(P) T D) I Ipos}, Ippos { i [1. 2) Ac: A thre) = R.)= (P) I D) 1 = pos]
(3)
(A,, Ru) where Pj and Pa denote the used prompts.
Ambiguity Juror. While Ja focuses on confirming object identification within positive views, the Ambiguity Juror Ja provides a complementary perspective by evaluating whether other similar objects in the scene also satisfy the description, revealing potential ambiguity.
Notably, we introduce the concept of twin thought: under identical prompting conditions used in existence analysis, altering the input to achieve a distinct evaluation purpose. Specifically, J. applies the same existence assessment as Jz. but on a misleading view set Imris = { .... 1% S I, the misleading set is formed by selecting images from the same scene that do not contain the ground-true truth object but include other instances of the same object category of Or. If high confidence is assigned to Tmiss it contradicts the exclusivity requirement of the 3DVG task, as follows:
(Aa; Ra)) = (Ja(Pa, Fin D) J E [miss]
where An and R., represent the sets of resulting scores and rationales, which serve as strong indicators of potential ambiguity.
3.3 Judge Refinement and Aggregation
consolidate the distributed reasoning paths, Refer-Judge then introduces the Judge system Jjudge introduces Magistrate Judges for refinement, followed by a District Judge that aggregates all juror outputs from multi-dimensional and heterogeneous assessment to produce a final, holistic quality scene-level judgment.
As shown in Figure 3
Magistrate Judge of Distinguishability. Due to incomplete or occluded views of the target object, both steps of the Distinguishability Juror Ja may produce unreliable assessments. Therefore, identi- fying and refining flawed evaluations becomes natural necessity. Unlike existing CoT refinement approaches based on majority voting or self-ranking 34 5.. the proposed Magistrate Judge of Distinguishability Janag moves beyond the simple selection and optimization of static consensus. Instead, it introduces a Corroborative Refinement mechanism, which focuses on retrieving auxiliary information and reorganizing inputs to enable more reliable re-evaluation.
Take the refinement of existence assessment as an example. The magistrate first examines the previous outputs (Ae, Re) and constructs Corroborative Pair Ser Ce by analyzing the associations and confficts within the evaluation set. Each element (suspicion, assistance) corresponds to pair of image indices (i.j) from Ipos. where the suspicion sample 12 is considered potentially unreliable, and the corresponding assistance sample I provides visual evidence to support re-ealluaion (e.g.. combining complete view with an occluded one). Through stitching operation sti(-) and re- assessment via Jÿ. the refined existence outputs (Aree, RRTe) replace the originals, mitigating the effects of unreliable observations. Concretely, sti(-) horizontally concatenates the aligned edges of suspicion and assistance views to form a richer, context-aware visual prompt for final assessment.
On the other hand, the refinement of uniqueness judgment serves as the rwin thought of existence refinement, performing a similar analysis of the corroborative pair C., reorganization of visual information, and re-evaluation. The overall procedure is summarized as:
Cleu) = - (p Afe,u): Ricut): Irel {sti(Ig, 13) (i,5) Clau)
[c.u]
Atele Reell))
JJ (le (p (ple.u) ,I, E
(5)
(Atenu);Ren R(enu)) (epluce((AAeaa)) Rie,ut)s ([ou) Reef (e,))))
(Aa, Ra) = (Ag @ An, R. @ R.))

---

where Pinagg denotes the prompt used for identifying suspicious assssstan pairs, Aa and Ra denote the final distinguishability score set and rationale set, with @ representing the message combination. Meanwhile, replace(-) substitutes the original (possibly erroneous or low-confidence) score and rationale produced by individual jurors with the refined outputs, ensuring more accurate and globally consistent results.
Magistrate Judge of Ambiguity. Unlike the Distinguishability judgment, which operates over Ipos focused on the target object, the ambiguity assessment involves scattered distractor views with limited inter-image correlation. Therefore. the Magistrate Judge of Ambiguity Jane simplifies the refinement process by discarding corroborative pair analysis and directly identifying set of suspicious assessments Sa. As simplified version to Ja Jany, the procedure defined as:
So = Ja (P (Paieg Aa Ra), In [Im l S
(Are1 R(e)) = (Ja(Pa. 1, D) 1 Ine (Aas R..) = replace ((An: Ra), (Are,, R)
(6)
District Judge. After distributed evaluations and localized refinements, the final decision rendered by the District Judge Juse, which integrates the reasoning paths provided by all preceding components. As an aggregation thought module, Jdis focuses solely on analyzing the outputs of the Jurors and Magistrate Judges, without directly revisiting the original scene-level inputs (I, D). This design mirrors the real-world Jury-and-Judg system, preserving interpretability while avoiding interference from redundant original messages. The process is follows:
(A,, Jdis (prioe (AL, Rt) (Ac. Re), (Aa, Ra), (Au, Ra))
where pdis is the prompt guiding the final judgment, and (A,, R2) denote the final judgment score with rationale. Altogether, the proposed Refer-Judge represent a comprehensive and trustworthy assessment of the 3DVG data, effectively uncovering latent paradoxes and ambiguities.
Experiments
We evaluate the performance of the proposed Refer-Judge follows: Section 4.1 describes the experimental setup and implementation details; Section 4.2 presents the main results of Refer-Judge well the performance of baseline models after toxic data removal via Refer-Judge; Section provides a detailed analysis of ablation studies on the proposed methods.
4.1 Setups and Implementation Details
Datasets. We conduct experiments on the proposed ScanRefer-Justice dataset to verify the effec- tiveness of Refer-Judge. Built upon the widely-used ScanRefer benchmark (7). ScanRefer-Justice introduces reliable 3DVG judgments annotated and verified by human experts, while emphasizing complex scenes with numerous similar objects, providing an accurate method assessment. Detailed statistics and construction procedures are provided in Appendix/M AA, Additionally, we evaluate baseline models the ScanRefer dataset to demonstrate how identifying and removing toxic annotations improves model performance.
Baselines. To validate the generality and effectiveness of Refer-Judge, we apply it across various representative Multimodal Large Language Models (MLLMs), including GPT-4o [88. GPT-4.1- mini (25). Grok-3 (39], Gemini-2.5 Pro 12). and LLaMA-3.2 11B [13]. To assess the impact of data purification on 3DVG, we consider multiple baseline models: ScanRefer 77, the pioneer method in
3DVG that established the detect-then-ment match paradigm; 3DVLP [7] and ConcreteNet [32), which
reflects superior performance in 3DVG and multiple downstream tasks.
Evaluation Metrics. Following JudgeLM (55), we apply agreement rate, precision. recall, and F1-score to verify the effect of toxic data identification. The RMSE and MAE are also reported to better quantify the alignment with human judgment on fine-grained scoring. To evaluate 3DVG
performance, following the experiment setting in [7], we choose Acc@0.25 and A.c@0.5 as the
evaluation metries. A more precise definition of the evaluation metric is provided in Appendix B
7

---

Table 1: Quantitative results on ScanRefer-Justice dataset. The Bold denotes the best performance. The Human Performance is statistically obtained during the verification process of the annotation.
Model
Agreement f
Precision
Recall
F1 1
RMSE
MAE
GPT-4o 18
82.77
82.95
85.77
84.33
2.69
1.71
GPT--411DD 1imii
25
81.81
82.64
83.66
83.14
2.82
1.94
Grok-3
39
81.14
81.03
84.66
82.81
3.07
1.84
Gemini-2.5 Pro 12
77.01
78.53
78.39
78.53
3.15
2.20
LLAMA-3.2-IIB
67.88
67.67
76.83
71.96
3.71
2.73
Human Performance
84.87
90.43
82.92
86.51
Table 2: Quantitative results on ScanRefer dataset. (a) shows results under the original dataset
splitting sets. (b) shows results under separated toxic and purified samples.
(a) Quantitative results on ScanRefer dataset. The models are verified on the original validation set.
Unique
Multiple
Overall t Acc@0.5
Method
Acc@0.25
Acc@0.5
Acc@0.25
A.c@0.5
Acc@0.25
TGNN TV
68.61
56,80
29.84
23.18
37.37
29.70
Instance
75.72
64.66
29.41
22.99
38.40
31.08
TDVG-Tranffommr 51
81.93
60.64
39.30
28.42
47.57
34.67
SecGround 19
75.7
68.9
34.0
30.0
44.1
39.4
Vi-ViSTA
81.6
75.1
43.7
39.1
50.6
45.8
ScanRefer 7
76.33
53,51
32.73
21.11
41.19
27.40
Refer-Judge
(4.57(43.24)
54.31
34.15(-142)
22.69(-1.58)
42.96(+1.7))
28.83(-143)
3DVLP 47
85.18
70.04
43.65
33.40
$1.70
40.51
Refer-Judge
(6.29(+1.11)
12.19(+2.15)
44.24:-0.59)
34.88(+144))
52.39(+069)
42.1
NoncreteNet
82.39
75.62
41.24
36.56
48.91
43.84
+ Refer-Jugge
(4.14()1775)
(9.57(43.95)
(1.97(-0.73)
36.16.4040)
49.94(-1003)
(4.55(-071)
(b) Quantitative results on toxic and purified data ScanRefer val set.
Toxic data Acc@0.25
Unique (purified) Acc@0.25
Multiple (purified) 1 Acc@0.25
Overall (purified)
Method
Acc@0.5
Acc@0.5
Acc@0.5
Acc@0.25
Acc@0,5
ScanRefer
21.69
14.58
76.89
50.57
34.96
21.80
43,91
27.94
Reder-Judge
18.76(299)
13.67.091
79.50.226)
55.41
36.07.1.11
24.44:2244
45.33
31.04:-3.m
3DVLP
22.91
17.75
85.97
70.02
46.44
36.02
54.86
43.01
Refer-Judge
22.11c0.00
15.46)229)
86.41:4044)
72.27
47.14:
37.436-14)
55.50..06
44.85
ConereteNet
26.33
22.37
82.73
75.68
42,32
37.42
50,92
45.24
Refer-Judge
24.18:215)
21.92.1105)
84.03
79.23(v3.5))
43.03
37.97.a39)
51.4:0a55)
46.45/-1an
Implementation Details. To ensure generality, all MLLMs are used with default configurations. Proprietary models, such as GPT-4o, are accessed through APIs. LLAMA-3.2 is deployed with the released checkpoints. Our experiments are conducted on computational platform equipped with Intel(R) Xeon(R) CPU E5- 2680v3 @2.50 GHz CPU x2, 128G memory. and RTX 4090 GPU x8. The inference of LLAMA is conducted using a dual-GPU setup.
4.2 Main Results
Comparison on ScanRefer-Justice Benchmark. We first evaluate the proposed Refer-Judge approach on the ScanRefer-Justice datasets. As shown in Table [ the quantitative results cover the performance instantiated with various MLLMs to comprchensively assess its effectiveness and adaptability. Notably, all models are evaluated in a zero-shot manner without fine-tuning on ScanRefer- Justice. Specifically, the results reveal strong correlation between the identification of toxic data and the underlying reasoning and perception capabilities of backbones. Among them, Refer-Judge (GPT-4o) achieves human-level judgment capability, slightly lagging with human experts by- -2.1 1 % in agreement score, while achieving recall of over 85%. The subsequent experiments are all based the GPT-4o-driven Refer-Judge. Additional case studies presented in Appendix [
Comparison on ScanRefer Benchmark. We further investigate the impact of applying Refer-Judge to identify and filter toxic data from the ScanRefer training set. As shown in Table 2a all baseline

---

Table 3: Ablation of the Jury System. Bold Table 4: Ablation of the Judge System. Bold denotes best performance.
denotes best performance.
Method
Agmtt
Prec.
Rec. 1
F1
Method
Agmt.
Prec.
Rec.
RMSE
MAE
Jury system (Ours)
CR (Ours)
82.77
82.95
85.77
84.33
2.69
1.71
Mix logical
80.52
80.34
64,70
82.46
observation
78.65
Goll 3
81.55
82.73
83.25
82.99
2.66
1.78
Mix four branches
75.67
76.03
79.91
77.91
Average
74.24
74.63
78.85
76.68
3.27
2.21
83.00
Table 5: Ablation study of module design.
The Cost shows the token required to complete the ScanRefer- Justice dataset (Million). Bold denotes best performance.
Refer-Judge
Agreement
Precision
Recall
F1 f
RMSE +
MAE + Cost +
Log.
Obs.
Ref.
V
/
81,55
83.44
82.18
82.80
2.91
1.93
108. 78
2
V
/
81.72
83.15
83,00
83.08
2.80
1.83
82.41
54.20
58.93
50.50
54.39
5.10
3.72
9.41
4
/
82.77
82.95
85.77
84.33
2.69
1.71
112.79
methods benefit significantly from the purified training data, achieving consistent improvements. For example, 3DVLP+Refer-Judee outperforms the bascline by +1.6% Overall A.c@0.5, achieving competitive 3DVG performance.
To better understand the effects of toxic data, further separate the validation set into purified and toxic subsets using Refer-Judge. As reported Table 2b the performance gain becomes even more evident, with +3.10% +1.84% and +1.21% % improvements in Overall Acc@0.5, respectively. On the other hand, the baseline model achieves better performance on toxic samples, which are inherently unable accurately predict. This discrepancy may arise because the original training set provides implicit priors aligned with the validation distribution, particularly benefiting the toxic samples, exhibiting inflated scores, which highlight the critical role of toxic data exposing making reliable evaluations for 3DVG. More results and analysis are left in Appendix
4.3 Ablation study
Ablation of the Jury System. We first examine the effectiveness of the Jury system in Refer-Judge. As shown in T the best performance is achieved when all four jurors jointly assess the sample from multiple perspectives, including linguistic logic and visual observation. This confirms that structured decomposition and distributed evaluation significantly enhance targeted reasoning over complex scene-level 3DVG data.
Ablation of the Judge System. Next, we analyze the contribution of our proposed Corroborative Refinement strategy (CR) within the Judge module. As shown in Table 4 our method outperforms alternative strategies such as Self-consistency with Chain-of-Thouge re-evaluation of problematic results (GoT (5), and naive score averaging. While CoT-SC yields comparable agreement score, its reliance on majority voting often leads to polarized decisions. which fails fine-grained scoring metrics like RMSE and MAE. These findings highlight the importance of adaptive refinement through auxiliary information in understanding the complex 3D scene.
(CoT-SC) (34), ranking and
Ablation of Model Design. Finally, we summarize the impact of each component performance and cost. As presented Table visual observation (e.g.. Distinguishability and Ambiguity jurors) contributes the most performance but also dominates token cost due scene-level image analysis. In contrast, logic-related assessments have a relatively lower impact, as visual jurors may implicitly assess textual logic. Meanwhile, the refinement the Judge system provides consistent improvements, helping the Refer-Judge achieve human-comparable performance in uncovering paradoxical and ambiguous 3DVG data. More detailed results are left in Appendix C
5 Conclusion
In this paper, we introduce Refer-Judge, novel framework for uncovering paradoxes and ambiguities 3DVG datasets, along with high-quality benchmark for evaluating data-centric methods. By leveraging a Jury-and-Judge Chain-of- Chain-of-Thought paradigm, Refer-Judge effectively harnesses the

---

reasoning and perceptual capabilitiess of MLLMs, achieving human coaann-oomaabllee performance in identifying toxic samples. Moreover, baseline models trained on the purified data consistently exhibit improved performance, underscoring the practical value of exposing and filtering flawed annotations. We hope this work offers a new lens on dataset reliability in 3D scene understanding and serves as foundation for more trustworthy training and evaluation practices.
Limitations. While Refer-Judge demonstrates strong effectiveness, its computational cost remains a barrier to large-scale deployment. Future work could focus on optimizing the judgment process and leveraging more efficient MLLM architectures to enhance scalability and extend applicability to larger and more diverse 3D datasets.
Acknowledgment. This work is supported by the "Pioneer" and "Leading Goose" R&D Program of Zhejjang (2024C01020, 2025C01091), Key Technology Breakthrough Plan Project of "Science and Technology Innovation Yongjiang 2035" (20257058, 20247295).
References
[1] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Al- tenschmidt, S. Altman, and S. Anadkat. technical report. arXiv preprint arXiv:3303.08774 2023.
[2] Achlioptas, A. Abdelreheem, F. Xia, M. Elhoseiny, and L. Guibas. Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. In European conference on computer vision, pages 422-440. Springer, 2020.
[3] Y. Bai, J. Ying. Y. Cao, X. Lv, Y. He, X. Wang, J. Yu, K. Zeng, Y. Xiao, and H. Lyu. Benchmark- ing foundation models with aagguagemmodellaaa--eeimii Advances in Neural Information Processing Systems,
2023.
[4] E. M. BAKR, M. A. Mohamed, M. Ahmed, H. Slim, and M. Elhoseiny. Cot3dref: Chain- of-thoughts data-efficient 3d visual grounding. In The Twelfth International Conference on Learning Representationss 2024.
[5] M. Besta, N. Blach, A. Kubicek, R. Gerstenberger, M. Podstawski. L. Gianinazzi, J. Gajda, T. Lehmann, H. Niewiadomski, Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17682 17690, 2024.
[6] D. Chen, R. Chen, S. Zhang, Y. Liu, Y. Wang, H. Zhou, Q. Zhang, Y. Wan, P. Zhou, and L. Sun. allm-aas-a-jddee Assessing multimodal juddee-a-jude with vision- Lsiounlanguage benchmark. arXiv preprint arXiv:2402.00 04788, 2024.
[7] D. Z. Chen, A. X. Chang, and M. Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In European conference on computer vision, pages 202-221. Springer, 2020.
[8] D. Z. Chen, Q Wu, M. Nicßner, and A. X. Chang. D net: A unified speaker- listener architecture for 3d dense captioning and visual grounding. In European Conference Computer Vision, pages 487-505. Springer, 2022.
[9] Chen, X. Chen, C. Zhang, M. Li, G. Yu, H. Fei, Zhu, J. Fan, and T. Chen. LI3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26428-26438, 2024.
[10] A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. NieBner. Scannet: Richly- annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and patlern recognition, pages 5828-5839, 2017.
[11] I.. Geng and J. Yin. Viewinfer3d: 3d visual grounding based on embodied viewpoint inference. IEEE Robotics and Automation Letters, 2024.
10

---

[12] Google DeepMind. Gemini 2.5: Our most intelligent ai model. URL URU_ttps::/blog.ggggl technology/ wowpollgggggggleeeeeoeooo deppgiieddeemindd/ gemanigeenniin pemini-model--ee thaeellttiinniggugpee -pdates--march-22225 march- 2025.
[13] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur,
A. Schelten, and A. Vaughan. The llama herd of models. arXiv preprint arXiv:2407.21783, 2024.
[14] Gu, X. Jiang, Z. Shi, H. Tan, X. Zhai, C. Xu, W. Li, Y. Shen, S. Ma, and H. Liu, A survey on Ilm-as-a-judge. arXiv preprint arXiv:2411.15544 2024.
[15] Y. Hong, H. Zhen, P. Chen, S. Zheng, Y. Du, Z. Chen, and C. Gan. 3d-llm: Injecting the 3d world into large language models. Advances Neural Information Processing Systems, 36:20482 200442220444, 2023.
[16] H. Huang, Y. Chen, Z. Wang. R. Huang, R. Xu, T. Wang. L. Liu, X. Cheng, Y. Zhao, and
J. Pang. Chat-scene: Bridging 3d scene and large language models with object identifiers. In The Thirty-eighth Anmual Conference on Neural Information Processing Systems, 2024.
[17) P.-H. Huang, H.-H. Lee, H.-T. Chen, and T.-L.. Liu. Text-guided graph neural networks for referring 3d instance segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1610-1618, 2021.
[18] A. Hurst, A. Lerer, P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, and A. Radford. Gpt-4o system card. arXiv preprint arXiv:2410.21276. 2024.
[19] R. Li, S. Li, L. Kong, X. Yang, and J. Liang. Seeground: See and ground for zero-shot open- vocabulary 3d visual grounding. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2025.
[20] H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892 -4916, 2023.
[21] J. Long. Large language model guided tree-of-thought. arXiv preprint arXiv:2305.08200 2023,
[22] J. Luo, J. Fu, X. Kong, C. Gao, H. Ren, H. Shen. H. Xia, and S. Liu. 3d-sps: Single-stage 3d visual grounding via referred point progressive selection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16454-16463, 2022.
[23] Y. Ma, Y. Cao, J. Sun, M. Pavone, and C. Xiao. Dolphins: Multimodal language model for driving. In European Conference on Computer Vision, pages 403-420. Springer, 2024.
[24] A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, Wiegreffe, U. Alon, N. Dziri,
S. Prabhumoye, Y. Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534-46594 2023.
[25] OpenAL. Gpt-4.1 mini model card. https::/pllttfomm.opo wwtm//ocs/modee/////go....o
mini 2025. Accessed: 2025-05-10.
[26] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, Mishkin, C. ZZang, Agarwal,
K. Slama, and A. Ray. Training language models to follow instructions with human feedback. Advances neural information processing systems, 35:27730-27744, 2022.
[27] D. Paul, M. Ismayilzada, M. Peyrard, B. Borges, A. Bosselut, R. West, and B. Faltings. Refiner: Reasoning feedback intermediate representations. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, pages 1100-1126, 2024.
[28] Z. Qi, Z. Zhang, Y. Fang, J. Wang, and H. Zhao. Gpt4scene: Understand 3d scenes from videos with vision-language models. arXiv preprint arXiv:2501.01428, 2025.
[29] x. Shi, Z. Wu, and S. Lee. Aware visual grounding in 3d scenes. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 14056-14065. 2024.
11

---

[30] N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao. Reflexion: Language agents with verbal reinforcement learning Advances in Neural Information Processing Systems, 36:8634 8652, 2023.
[3]] G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, and K. Millican. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
[32] O. Unal, C. Sakaridis, S. Saha, and L.. Van Gool. Four ways to improve verbo-visual fusion for dense 3d visual grounding. In European Conference Computer Vision, pages 196-213. Springer, 2024.
[33] Wang.
S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, and W. Ge. Qwen2-vi: Enhancing vision-lanuuag model's perception of the world at any resolution. arXiv preprint 1IX1:2249912211 2024.
[34] X. Wang. J. Wei, D. Schuurmans, Q. V. Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023.
[35] Y. Wang. S. Wu, Y. Zhang, W. Wang. Z. Liu, J. Luo, and H. Fei. Multimodal chain-of-thought reasoning: A comprehensive survey. arXiv preprint arXiv:2503. 12605, 2025.
[36] Z. Wang, H. Huang, Y. Zhao, L. Li, X. Cheng, Y. Zhu, A. Yin, and Z. Zhao. 3drp-net: 3d relative position-aware network for 3d visual grounding. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10612-10625, 2023.
[37] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q Le, D. Zhou, et al. Chain-of- thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24837,, 2022.
[38] L. Wei, W. Wang, X. Shen, Y. Xie, Z. Fan, X. Zhang, Z. Wei, and W. Chen. Mc-cot: A modular collaborative cot framework for zero-shot medical-vqa with Ilm and milm integration. arXiv preprint arX1v:2410.04522, 2024.
[39] xAI. Grok-3 beta release. httpa//t...tt../....a///////rrr.
2025. Accessed: 2025-05-10.
[40] J. Yang, X. Chen, S. Qian, N. Madaan, M. Iyengar, D. F. Fouhey, and J. Chai. -mngrounder: Open-vocabulary 3d visual grounding with large language model as an agent. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 7694-7701. IEEE, 2024.
[41] F. Yao, C. Tian, J. Liu, Z. Zhang, Q. Liu, L. Jin, S. Li, X. Li, and X. Sun. Thinking like an expert: Multimodal gppergraph-of-thoughh (hot) reasoning to boost foundation modals. arXiv preprint arXiv:2308. 06207, 2023.
[42] S. Yao, D. Yu, Zhao, I. Shafran, T. Griffiths, Y. Cao, and K. Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:1 1809-1118888 11822, 2023.
[43] Y. Yu, Z. Yao, H. Li, Deng, Y. Jiang, Cao, Z. Chen, J. Suchow, Z. Cui, and R. Liu. Fincon: A synthesized Ilm multi anlti-agent system with conceptual verbal reinforcement for enhanced financial decision making. Advances in Neural Information Processing Systems, 37:137010-137045 2024.
[44] Z. Yuan, X. Yan, Y. Liao, R. Zhang, S. Wang, Z. Li, and S. Cui. Instancerefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contex- tual referring. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1791-1800, 2021.
[45] D. Zhang, J. Yang, H. Lyu, Z. Jin, Y. Yao, M. Chen, and J. Luo. Cocot: Contrastive chain-of- thought prompting for large multimodal models with multiple image inputs. CoRR, 2024.
[46] H. Zhang, C. A. Yang, and R. A. Yeh. Multi-object 3d grounding with dynamic modules and language-informed spatial attention. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
12

---

[47] T. Zhang, S. He, T. Dai, Z. Wang, B. Chen, and S.-T. Xia. Vision-language pre-traie treiiiinn with object contrastive learning for 3d scene understanding. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 7296-7304, 2024.
[48] X. Zhang, C. Du, T. Pang, Q. Liu, W. Gao, and M. Lin. Chain of preference optimization: Improving chain-of-thought reasoning in llms. Advances in Neural Information Processing Systems, 57:333-356 2024.
[49] Y. Zhang, Z. Gong, and A. X. Chang. Multi3drefer: Grounding text description multiple 3d objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15225-15236, 2023.
[50] Z. Zhang, A. Zhang, M. Li, G. Karypis, and Smola. Multimodal chain-of-thought reasoning in language models. Transactions on Machine Learning Research, 2024.
[51] I. Zhao, D. Cai, L. Sheng, and D. Xu. 3dvg-transformer: Relation modeling for visual grounding on point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2928-2937, 2021.
[52] C. Zheng, D. Liang, W. Zhang, X.-Y. Wei, T.-S. Chua, and Q. Li. A picture is worth graph: blueprint debate paradigm for multimodal reasoning. In Proceedings of the 32nd ACM International Conference Multimedia, pages 428, 2024.
[53] G. Zheng, B. Yang, Tang, H.-Y. Zhou, and S. Yang. Ddcot: Duty-distinct chain-of-thougee prompting for multimodal reasoning in language models. Advances in Neural Information Processing Systems, 36:5168-555 2023.
[54] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Lin, Z. Li, Li, and E. Xing. Judging Ilm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595 2023.
[55] L. Zhu, X. Wang, and X. Wang. Judgelm: Fine-tuned large language models are scalable judges.
In The Thirteenth International Conference Learning Representations, 2025.
[56] Z. Zhu, X. Ma, Y. Chen, Z. Deng, S. Huang, and Q. Li. 3d-vista: Pre-trained transformer for 3d vision and text aligmment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2911-2921, 2023.
13

---

A ScanRefer-Justice Dataset
Assessment Criteria
The sample has obvious logical paradoxes or observational ambiguities.
Multiple objects clearly fit the description
34.5%
The sample has logical paradoxes or extent.
observational ambiguities Toxic in the rigorous evaluation..
6
00.9%
sample credible.
There is only one object in the scane that description
4:15
10+
ScanRefer
Scan Justice
Figure 4: Complexity distribution of ScanRefer and ScanRefer-Justice datasets.
Figure 5: Assessment Criteria for anno- tating ScanRefer-Justice.
To evaluate the effectiveness of our proposed method in identifying toxic samples, we introduce a new benchmark dataset: ScanRefer-Justice This dataset is constructed upon the widely used ScanRefer dataset which contains 51, 1,583 referring expressions for objects from ScanNet [T0 scenes.
A.1 Data Selection
Due to the cost and complexity of conducting rigorous human evaluation across the entire ScanRefer dataset, it is impractical to manually verify all samples while ensuring annotation quality. Moreover, we observe that the occurrence of toxic data is highly correlated with scene complexity. Over- representation of simple samples (i.e., only few objects of the same class) can lead skewed data distribution and biased algorithm evaluation.
To better reflect this complexity, categorize the ScanRefer samples into three levels based on the number of same-class objects with the target one: Normal (1-3 instances), Hand (4-15), and Complex (16 more). As shown in Figure 4 choose 3,001 data from the ScanRefer training and validation sets to form ScanRefer-Justee covering 162 scenes. The sampling focuses on increasing coverage of more complex scenes better assess the performance of 3DVG data judgment algorithms.
Specifically, ScanRefer-Justie significantly rebalances the data composition compared to ScanRefer. While complex samples constitute only 6.3% in ScanRefer, they account for over 20% in ScanRefer- Justice. Combined with hard samples, more than 50% the dataset consists of challenging examples, capturing wider range of possible sources of annotation errors.
A.2 Data Collection
To ensure consistency and rigor in data labeling, we employ trained internal annotators (graduate students), rather than outsourcing, was done in the original ScanRefer annotation process. The annotation pipeline consists of two stages: (i) Annotation and (ii) Verification.
Annotation. Annotators first observe the complete 3D point cloud scene using Open3D visualization tools. This step excludes both the referring expression and ScanNet's object labels to reduce prior bias. Observers are required to spend at least 20 seconds examining the scene to ensure comprehensive understanding.
Next, expanding the annotation process of ScanRefer 77. we provide up to six 2D reference views per sample compensate for incomplete details in the 3D point cloud reconstructions, where three images containing the target object and three with distractor objects of the same category. Category definitions aligned with the original setting of ScanRefer. Meanwhile, the corresponding referring expression is presented as on-screen subtitles.
Based this information, annotators assess each sample's quality from two perspectives: the Logical Completeness of the description and the Uniqueness the visual context. Throughout the process, the annotator revisit the 3D scene and consult additional ScanNet frames for cross-refcrencing
14

---

Normll
14.7% 17.8%
90.2%
Complex
45.9%
53.6%
77.2% 87.9%
Figure 6: Distribution of reliability scores in the ScanRefer-Justie dataset.
0.35
2.55
2.5
0.5
Todl Padto
02
Deectipiion
Rarge
Figure 7: Relationship between reliability score and sentence length in ScanRefer-Justice dataset.
To ensure consistency in scoring, we require annotators to follow 9-point Likert scale based on predefined Assessment Criteria (higher scores mean higher data quality), shown in Figure
Verification. After the initial round, the annotators carry cross-checks validate the consistency of the annotations and estimate human error rates toxic data identification (as reported in Table I During this stage, the previous annotation score displayed together with the referring expression.
A.3 Data Statistics
Figure iillusttates the score distribution in ScanRefer-Justie Benefits to our targeted sampling. the dataset achieves balanced distribution of reliable and toxic samples. Specifically, 45.9% of samples are rated as clearly erroneous, 7.7% as moderately flawed, and 46.4% as clearly reliable. Most scores cluster around or 9, indicating strong confidence among annotators. Samples rated in the ambiguous 4-6 range are notably sparse. Additionally, it is clear that toxic data appears more frequently in scenes with higher complexity.
In addition, we examine the relationship between referring expression length and the quality of 3DVG data, as shown in Figure 7 Interestingly, short expressions (<15 tokens) and medium-lenge ones (16-30 tokens) exhibit similar error rates, likely due compensating effects between scene simplicity and annotation carelessness. Logically. descriptions with longer than 30 tokens show marked improvement reliability, suggesting that detailed language helps disambiguate targets in 3DVG. Nevertheless, longer descriptions may impose a cognitive burden on annotators. Leveraging LLMSs to provide adaptive, guided assistance during annotation may be a promising direction for future 3DVG dataset construction.
15

---

Table 6: Quantitative results on ScanRefer test set.
Unique
Multiple
Overall f
Method
Acc@0.25
A.c@0.5
Acc@0.25
Acc@0.5
Acc@0.25
Acc@0.5
InstanceRefer
77.82
66.69
34.57
26.88
44.27
35.80
DDVG-Tanslommer
51
75.76
$5.15
42.24
29.33
49.76
35.12
D3Net 8
79.23
68.43
39.05
30.74
48.06
39.19
D-LISA 46
81.95
69.00
49.75
39.67
56.97
46.25
SeanRefer 7
68,59
43.53
34.88
20.97
42.44
26.03
+ Refer-Judge
(3.45))
47.16(+3.63)
35.36:-0.48)
21,25(+0.28)
43.90:+1.46)
27 27.06(-1.03
3DVLP (4
78.24
62.98
45.32
34.05
52.70
40.54
+ Refer-Judge
79.39
65.46(+2.48)
(6.51
34.91
53.88/+1.18)
41.76(-1.22)
B Details of Metric Calculations
To evaluate the ability of the proposed method identify toxic samples in 3DVG datasets, first formulate the task as binary classification problem. Notably, since we are more concerned with the identification of toxic samples, toxic samples are considered the positive class, and valid samples as the negative class. We use TP, FP, TN, and FN to represent the true positive, false positive, true negative, and false negative, respectively.
We define toxic samples as those with numan-annnotttrr scores between 1-6, including both obviously untrustworthy (score between 1-3) and partially flawed samples (score between 4-6). Valid samples are those with scores in the range of 7-9. The calculation of agreement, precision, recall, and Fi-score follows:
Agreement (TP+TN) FT
Precision TTP (TP FP)
Recall TP/ (TP+FN)
Fl-score = (2 TP) / (2 TP + FP FP+FN FN)
To further evaluate the alignment between our method and human assessments on a finer scale, we additionally treat this task as regression problem and report the following metrics:
N N
N
RMSE
&
MAE
8;
Si1
N
where s; the human-annotated score and &; is the predicted score for the i-th sample.
C Additional Experiments
Comparison on the ScanRefer Test Benchmark. To further validate the effectiveness of filtering toxic data, we report method performance on the ScanRefer test set. As shown in Table after training on purified data organized via Refer-Judge, both baseline models exhibit consistent performance gains, with +1.03% and +1.22% in Overall Acc@ 0.5, confirming the stability of our method. Notably, combined with the validation results in Table 2 the performance improvements are more pronounced in scenes where only object of the target category exists Unique, compared to Multiple samples (with multiple similar objects). This is likely because the original training data contains distributional biases that help models overfit toxic samples. Correspondingly, as shown in Figure 6 toxic samples are more concentrated in complex scenes, allowing baseline models to gain an extra advantage in Multiple-type evaluations.
Comparison on the Purified Validation Set. To better analyze the impact of toxic data on algorithm evaluation, extend the analysis in Table 2 by progressively decreasing the threshold in toxicity judgment, thereby including broader range of potentially toxic samples. As presented Table 7 the proportion of identified toxic data rises from 7.6% to 40.6% when the Refer-Judge threshold increases
16

---

Table 7: Quantitative results on toxic and purified data under different thresholds. The 'Thr'' indicates the score threshold and the resulting proportion of toxic data under that threshold.
Toxic data 1
Unique (purifted) 1
Multiple (purified) Acc@0.25
Overall (purified) f Acc@0.25
Method
Thr.
A.c@0.25
Acc@0.5
Acc@0.25
Acc@0.5
Acc@0.5
Acc@0.5
ScanRefer
20.44
13.12
76.91
50.57
34.77
21.78
43,60
27.81
+ Refer-hidge
17.96.248
12.44.220
79.52.220
55.400483
35.730009)
24.27
30.79 44.91.130
3DVLP
7..6%
22.69
17.36
84,68
68.27
44.58
34,38
52.97
42.17
+ Refer-Jidge
22.41:038)
14.43:299)
86.70 2.00
70.420215)
46.49
36.01
54.91
43.22: 1.05
ScanRefer
21.69
14.58
76.89
50.57
34,96
21.8
43,91
27.94
+ Refor-Judge
$ -9.3%
18.76
13.67:499
79.50-240
55.41
36.07/41119
24.44:2260
45.33:-1 im
31.04-5
3DVLP
22.91
17.75
85.97
70.02
46.44
36.02
54.86
43.01
+ Refor-Judge
22.1 so
13.46-2.29
86.41c-0
72.27
47. 146000
37.43:+1 41)
55 35.50.0060
44.85:-180
22.58
16.03
50.66
37.34
+ Refer-Juuce Judge
< 3 -21.1%
21.53 21.53:1009
15.60-044)
79.61c-299
45.59
38.59;-129)
25.91:-30)
48.56
33.12:-5.49)
3DVLP
27.57
70.53
Refer-Judge
18.83.2.77)
86.52--030)
72.42
50.84(1.19)
40.63:+215)
19.51
48, 38.36--2.09
ScanReler
24.23
17.00
77.12
50.71
38.97
23.39
49.41
30.87
Refer Judge
S -40.%%
43,46,407
17.28.a0:0
79.65-2259
55.72
40.51:-150
26.73:-350)
51.22:-181
34.66--379
3DVLP
29.32
22.91
86.3
70.71
$2.34
40.42
61.63
48.70
Refor-Judge
28.36,4000
21.85.1000
86. 86.64c-1111)
72.55t+136
53.77(-141
42.69.2221)
62.76:1119
50.86-2210)
Table 8: Comprehensive ablation study of module design. The Cost shows the token required to complete the ScanRefer-Justice dataset (Million). Bold denotes best performance.
Refer-Judge
Jr
Je
Ja
Ja
3
Jmae
Agreement
Precision
Recall
RMSE
MAE |
Cost
82.77
82,95
85.77
84.33
2.69
1.71
112.79
82.53
83.87
83.81
83.84
1.77
111.36
X
81.75
82.92
83.44
83.18
2,88
1.87
110.21
X
X
80.86
85.28
78.09
81.53
2.97
1.89
50.83
X
62.41
64.00
69.65
66.71
3.74
2.68
78.78
X
81.61
83.33
82.49
82.91
2.82
1.81
85.13
7
X
82.67
83.44
84,76
84.01
2.73
1.74
110.07
X
X
81.55
83.44
82.18
82.80
2.91
1,93
108.78
X X
81.72
83.15
83,00
83.08
82.41
X
X
54.20
58.93
50,50
$4.39
5.10
3.72
9.41
from to 4. While a lenient threshold inevitably misjudges more valid samples, we ensure methods are evaluated the same purified validation set to keep fairness. The results demonstrate that across all toxicity thresholds, models trained with Refer-Judge consistently outperform the corresponding baselines, confirming the robustness of method. Interestingly, between purified and baseline results. For instance, at threshold I (strict), 3DVLP+Refer- Jude outperforms the baseline by +1.05% in Overall Acc@ 50, whereas the gain increases to +2.16% at threshold < 4 (lenient). This highlights that beyond the identification toxic data, establishing standardized evaluation framework is essential for enabling robust and comprehensive method assessment.
observe non-uniform margins
Comprehensive Ablation of Module Design. We then expand the model design shown in Table 5 to investigate the impact of each module in Refer-Judge framework. Experiments 2-5 Table examine the removal of each individual juror within the Jury system. The results show a consistent performance drop, confirming all four perspectives contribute indispensably to toxic data identifi- cation. Among them, the Ambiguity Juror proves most impactful, highlighting prevalent issue in 3DVG annotation: the failure to verify the presence of similar misleading objects in the scene. Meanwhile, comparing experiments and 6, the introduction of Corroborative Refinement in Jmma improves the Agreement score by +1.16%, demonstrating the benefit of using auxiliary information to guide and correct flawed judgments arising from incomplete observation.
D Case Study
Finally, we present some representative cases intuitively illustrate the decision-making process of Refer-Judge on 3DVG samples. On the one hand, we show some cases from the ScanRefer dataset (7). which respectively correspond Normal, Hard, and Complex scenarios.
17

---

Logical Juror
Consistency Juror
There is black square tv. placed on wall kitchen
small table.
Score:
A.R
arfiod
Rationale: descijtton kuckkes placed the kitchnn
D
reforence another sieder
1.R
jotroduces Rigiiall paradox
Score:
itationlle The deveriptioon in D mostl comsittent with both cieeerbo bleck mounted on and The only miner snver are the shape ('square" n. "ractanguler" ? and the location ('kitchen which not mentioeed G. These are not direct contruulitions. just unvenilable or algndy ambipucus
3D Scene
Ambiguity Juror
messages
Magistrute Judge of Ambiguity
Distinguishabillty Juror
Magistrate Judge of Distinuiishahiiit
A R
Corroborative Pair:
Suspition 1. Assestance
D
Snone:
Ratiorale: The bey abject the TV, muteche color and placement mectmnuuar insteed descrbed or opuaen The incetion the description khohees
A
R
Score: Rationall: description accurately metcfex imuge al objects thee redetionn correctlyy depicted
Score 0
Ratlonane:: the akoonbed black square TV the aua of the kitchen doer not appoor in the provided image, as the anvlronment and objectss ahown not matco the description
A.R
A.R.
A.R.
Assigaanc
D
Enauge
A
R
A.R
decrription uniqueness
shope defisttiely
only
diaryy cstalliiter lrugt
souare
A,
Scove
Radionsle There onl ane black TV on woll nanching the absonjettoe though the shape discrepancy evirts
**JSON MESSÁGE
'JUDGE MESSAGES") I
Judge
"Ambiguty"
'Logica":
Score 9
"Consistenc":
Ratlonsie: The presevee of muitipll ponitive cistiguuehabbilty samples and no minleacling or serious logical errora nupprter high acone. fhe consietency le sound minor shape miterntc eightly redvcve uniqueneer but does not algnificantly harm overal evalartion.
"RATIONALE MESSAGES":
"Uniquereos" [Rationale my.
"Loge": "Ratinallc
"Consisency" Rationale -.. 13:
Figure A normal case where only one TV is involved in the scene.
Detailed Analysis. Although this is a relatively simple case, several interesting details emerge. For instance, when analyzing the second view, Ja fails to detect a matching object because the TV only partially visible at the edge of the frame. This viewpoint-induee induucd failure is subsequently corrected during the refinement process provided by Jmmag Moreover, we also observe errors caused by ballucination. For example, its second step. J julágg incorrectly rejects the TV in the image for not satisfying the specified "square" shape.
18

---

Logical Jurur
Consistency Jurur
gray
Description Groupe
identical
There a gray is
oéher facing
A.R.)
other chair, noxt each
Ratiovale
Scene
focter edendiall "crearing circolar idennty the intended ieenical chaii others the next-othhr ambigunty
D
reforence iegossible
Rationale: mostly comvintent with one Group's
descriptions andigaity ariuss
Distinguishabiiit Juror
identical denall strengly
Grous Ovanal captures essence
Magistrate Judge of Distinguishabiiity
discription aclegoolad."
A...R
A.R
Corroboration Pain:
Ambiguity Juror
Score imuge gayy
setisfing elements descrjation
However the araanement specificdion
perfectly
7.00
Scove
Emuage wtitching
Ratlovail: The descnption of color and placemate
matcher avel with the Image The uncersiiity about the materall (Barber) its minor tonce the drage shus not cleary demorstzate teutions
Score: 9
R
description gray
Rationsil The coler and anarggmmnn detsvli of the chavee the image mustch the description perficclly The only axpeet shar's unverfabble a the materiar baing louther cuee to the imagee quality and anglee (
placed
R
image, expilcitly arrangeennt gely
Á.R
described colors ahthougg identical
A.R.
Score: 2
Nowe
?
Rationale: There are maiiijule gray and lakee chairs alocee i row, shontics ahhich atfich renders the descrjption now wrique.
A..
A.. R,
A,
R
Magistrate Judge of Ambiguity
District Judge
A, R
"JOO MISSAGE* needs anahpas
"Sstrpsiisbbll [5:0]]
MESSAGIS":
Rationale Despito accoptable concitonncy presence anbigulyy
Suspicious 12 S. 61
"Amiiqulty" 5.444.
"HAIIOHALE NESSAGES":
'Comesinno/:8)
"Lagiaa"s 3,
unique presenting Dirtinguldbablitt
ligical
Rationale:
A".RR"
Rationale descrioninn partielly chali vibbie, but the with identiccal chaire
"Exstence" Rorionale:
"Uniqurness" Faaiooaaee in,
muteriat artrisscss blck differ significantly from the destrijtionn
"Lage" "Rat oralec
"Comasienoy"
Figure 9: A hard case where Refer-Judge provides correct assessment in both paradox and ambiguity identification.
Detailed Analysis. As case exhibiting both paradox and ambiguity, multiple components contribute to the final judgment. For instance, Ji provides direct evaluation of the textual paradox, while Ja assesses the object's uniqueness. This demonstrates the advantage of Refer-Judge analyzing problematic samples from multiple complementary perspectives.
19

---

Logical Juror
Consistency Juror
3D Scene
grey yellow pattern, away
Description Groupt [
The char the lowest one from the right ace the u-shaped table, the chair has curoed bacicdde and best seat)
A.. Racionall: The descrition dtentilie the chall by ihe posiiion rederive to speettic "nolled out chal." The count dectig pittee anic chail being refenred to resolving ambiguity
Score 10
D
A.R
Score: (Deisult)
Rationale: Not enough information to analyz
Magistrate Judge of Distinguishuiiiity
Amhiguity Juror
Distinguishability Juror
A.R.
Corroborative Pair Suapltisee Q Amsrrtannee
Rationale: The spatial arrangement of chaiss nurches the descriation, and thee color gray consórevi newe,, the yelloe pattonn nut vostil lealig to dicreponcy
Score: 7
Score: 5 $
A R
Rationaler Wile chairs present, specitic chanacteripics Mle being grny with yellow portern are not visblle and spocol pocement cannot be verifled with certalory.
Score 4
Racionaler The deserttedd color and patter nutch the chairs suugn However spucial desesppliss is andigoouu makking atticcuft confivm which specitic chale belng referwnced.
chasr slleeed inectiiiattin
Scone:
A.R.
A,R
Retionalle The imsage partilly matches the deseriptton The spatiol arrangement accurate, but the pattent's cobr deneel canvor be fully discorned ciee to image clenty.
Corraborative Palr
Score: T
Rationale: Bath views to prosiée sufficient context clarity to identify the chelr pulled
A R.
Scone:
Rationale: The choice match the described partens, but the spadiel rebationahip deveriied connet connfornee dis to the lekk information about the 'Brrt puslled out chai..
unique may The image dieiierr comber nenoler artigssiin
R.R.
Score 2
Rationabl Several could fe patteen dencipptio, image gualtt makes sllettifiing jelleer trugh positoning relectve puilec- out it anclaar.
A.R
R
R.
Magistrate Judge of Ambiguity
District Judge
The *JSON MESSAGE" needs 3 "JUDSE MESSAGEST:
Score:
A.R
Suspicius aD (4)
Rationale: Although there it logical arror, and high connisrency helpr ritigate the dhcrage there are no posltive distinglebabbllly asmpplas, and the sample is algndficantly ambiiuuity with threr high ambigutty valowe (10, & 1)) mhich oxlcation pair scove Naddgin it somard pour but unttinll brokin sumples.
"Deengutshbiity"100 1.6 21 21,7:4II
"Ambuuutt" 14,6, 5, 6,3,,,3"" 6. 'Lagical" 30,
A
Rationale: The description accorately correspands with the visible Image Noth the design pottern and spontal arrangemenr of the chaits.
Score 10 ?
"Consieenyy"
THANIONALL MESSAGES:|
R
(Rationale:
A
"Uniqueness" |Raiionler
"Lage": prale:
"Conastency: Nationale:...
18.
Figure 10: A complex case where over 20 similar objects introduce significant visual ambiguity.
Detailed Analysis. In this more complex scene, Ja processes larger batch of images and thus plays a more significant role in the overall decision-making process. However, the analyses from Ja and Jung also reveal that even for two highly similar images (T3, and 7²,), their corresponding evaluations may differ substantially. This further highlights the importance maintaining evaluation consistency within such agentic system.
20

---

Logical Juror
Consistency Jurur
The brown amm chair below the painting close the purple couch.
Description
1:
left, thi
The armchair
A.R.
God midill the the
Score 10
pairting
Retionaric The deserigtton inchalo mudiple spatial and color cuss ('drown arm chai," "balow the painting "slose to the purple couch 1) which togedtor provide clear and unique identification with no cinaler recipenzal ambauuity
D
3D Scene
description clesely alipe spocial anaigement object tycc differing onl percolved rone explaviable dicrepancy.
Distingaishablity Juror
Magistrate Judge of Distinguishubiitt
A.,
A,
Ambiguity Juror
No conflicting messages
Score: 7
Rationale: The arnicaa' ensts and rouply nurches the descried color howevve, die painting above it and the couch proninity cannor be fully veriffed from this esagl.
Mugistrate Judge of Ambiguity
R
Suspickous I0:
!11
objects apatial Rationale: present The disiupanccs rolationsliu
Rationale: descriptinn
('romm "sorplic" marvon//oa)) are considored sudjective differences.
seeutilles object (arechial) location
Score:
Rationale:: The discreponcy in color ("brown," "purple") is minor issuc. The description correctly identifled the arnchair' location under the painting, but the mahh issus was that incornectly described inother anmohair next to as "sofa," a typolog call error for key object,
A..
describes "coued" aroccaii which
when chartly
major type
2 Ratiosaale The desor pstorition accurately identifles kee objects michal,, painting couch) and their crucial spotial relationships l"below," "close to').
Score 9
Score: 10
A.R.
.R.
Rationale: The deseripcion contails key spetial rebitinship ('oclwr the paintiog)) that successfully isolites a single, specilic annchai from avother simier one in the scume.
A. R
District Judgee
The *ISO MUSSGGE needs b
"RATIONALE MESSAGES":
"AUDGE M
"Dsthggreabbih""i | 10), (8: 2013
Tisstence" |Rationale::m
Rationale: Although slowce muttjale pasitive samples disigussiuallyy Eagitan abvious
5
"Amniguty" 14.9)
"Logica" 10.
"Consstency" 8
"comustincy" Ratiorade:
"Lape": "Brllnns
17.
ontering
final
Figure 11: case passes the inspection of Nr3D but is actually ambiguous.
On the other hand, we further extend Refer-Judge to the Nr3D dataset 2 which additionally provides examiner (listener) annotations assessing the quality of the original labels. Despite this stricter verification setting, Refer-Judge still uncovers toxic data within Nr3D, including cases where samples consistently judged as correct by annotators are actually problematic, and instances flagged erroneous by examiners in fact solid. These extended case studies further demonstrate the effectiveness and generalizability our proposed framework.
Detailed Analysis. Although this is relatively simple scene, the evident ambiguity was not detected by annotators, revealing the widespread presence toxic data in manually labeled 3DVG datasets. Notably, during the ambiguity analysis, I, captures only partial view of the painting, leading to a low score from Ja, which Jmag also fails to correct. This observation underscores the current limitations of our system.
21

---

3D Scene
Logical Juror
Consistency Juror
it a shelf with poster on it touching the comerr the roon behind the green chair
Description Groupe !
This shelf la carner.
The shelf in the coenar nast the angal staticas
Radionater The desciptton provides a cohereit special clasi ('selt mith poster ic" "Gosciligg the carner. "bekknd the green chall ") that unigonly focalizes the target object without reciprocal oofoenioooeniil andigily
Store 10
A..R.
Facing the shelves, the shall on the right.
Shorter sheff the coreer.
The sholf that is tucied inshe conner]
Rationali:: fully with spatiel depiction a while providing details
Magistrate Judge
"Bebood
Distinguishubilkty Juror
Distinguishubiiity
A.R R.R.
Stspption Asserance 1
Corroborativ Pali:
2B
Ambiguity Juror
Image wiitehiig
Rationalls: some
shoel'
AL
location are correct are elecrspancios, impraciie
inclooing incarrect color the poster.
Rarionater abjects their key specia! rebriónnsips accurately regresented
slews. The miner
Score: 9
"agalur" nptreviiial
Rationale: All mantioned objects the attributes their spatial relebtorniip accuretaly deseribed The
with
A.R
primary sutject, image appear
strang rutah for deseription
R.
A.R.
Corroborative Pair:
A.
A.R R.
Novs
District Jungee
Ay. R.
"ISON MESSAGE" anelysia
MESSAGES":
"Daingckhuuiltt" st, [o:10]]
Mugistrate Judge of Ambiguity
"Ambruuty
ceessage
R.R.
"Logical: 30,
excollent quality mutiile
Scorw:
"Comiiteera"
strong postiive samples
Rationale: desrriptionn uniguuely Idennlibble bndlwark chair specily locarion shell succesofully beataring slogle
"BATIONAL MMSSAGGS
Disdigiinnabblyy signs
No ralated messagge
"Esoitenca" (hation
"unqueress" |Fationate: ".,
Lugical Cousisanncy
Figure 12: A case marked as failed Nr3D (due speaker-listener mismatch) but turned out be valid on re-check.
Detailed Analysis. This is an interesting case where the examiner assigned a low-confidence mark, though the referring expression already quite detailed. Through our Refer-Judge framework. we observe that the high score given by Ja to 1 leads to a final positive evaluation for this sample.
22

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction of the paper clearly outline the main contributions and the scope of the research.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper adequately discusses the limitations of the work in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

This paper focuses on the practical application of large language models in assessing the quality of 3D visual grounding. and does not introduce new theoretical results. Guidelines:
Justification:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper provides a clear and comprehensive explanation of the proposed method, Key setups critical for reproducing the results are meticulously detailed in Sec-
4..
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
24

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code and the proposed dataset provided in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ ccippi.cc/pulli///gr. guwdes/ddeeeee CooeeSumeeieeeeeeeeeee ssi onyylicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
The experiment setting and details are presented in Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We keep the same basic method settings across all the experiments reported in Section and Appendix [[to produce meaningful ablation study. However, do not report eтror bars due to the significant cost of continuously invoking the MLLMs. In fact, we have spent nearly $10,000 on API calls to obtain the current experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
25

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The information on the computer resources can be found Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
The research conform the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The societal impact described in Section 5
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
26

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our method is based on original proprietary or open-sourced MILMs, the procedure does not introduce additional risks. Users are encouraged to adhere to the safeguards provided by the corresponding MLLM.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators of the used datasets and models in our paper are properly cited. The proposed dataset is constructed based on ScanRefer and adheres to its license agreement (CC BY-NC-SA 3.0).
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
27

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
latasets.papersstt
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification: The code/dataset provided in the supplementary material is documented with detailed instructions.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: We thank the contributors involved in data annotation and provided compensa- tion well above the local minimum wage.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
28

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: As a core component this work, we provide detailed description of how the proposed method leverages large models.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurips htct:::://eerips..cc//ooeeeeeeee cc///oofeeeenees/2225//.... for what should or should not be described.
29


Reviews:
[{"review_id": "G8iZGGpiPG", "reviewer": "Reviewer_hBqW", "summary": "This paper proposes Refer-Judge, an agentic system to detect toxic data in 3D visual grounding. The framework employs four jurors to assess different aspects of the data quality in 3DVG, followed by a judge refinement and aggregation procedure.\nThe proposed framework achieves an accurate assessment that is close to human performance. \nExperiments show existing 3DVG approaches can benefit from fine-tuning using the purified data.", "questions": "- What is the typical process time of one scene?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n- The idea and the framework design are insightful.\n- The paper is well-written and easy to follow.\n- The experiments are comprehensive.\n\nWeakness:\n- In Tab. 2 (a), the performances of ScanRefer [1] and 3DVLP [2] show impressive performance gains after using the Refer-Judge framework. However, it will be more persuasive if the framework can consistently boost at least one more recent work (given ScanRefer [1] was published in 2020).\n- How is the misleading view set $I_{mis}$ described in L. 188 constructed.\n\n[1] Chen, Dave Zhenyu, Angel X. Chang, and Matthias Nießner. \"Scanrefer: 3d object localization in rgb-d scans using natural language.\" European conference on computer vision. Cham: Springer International Publishing, 2020.\n[2] Zhang, Taolin, et al. \"Vision-language pre-training with object contrastive learning for 3d scene understanding.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 7. 2024.", "was_revised": true, "final_justification": "The authors addressed my questions about the baseline, details of the misleading set, and the running time.", "rebuttal": "We thank the reviewer for the encouraging comments and recognition of the insights, clarity, and comprehensiveness of our work. We appreciate the helpful suggestions and provide detailed responses below.\n\n**Q1: Additional recent baselines beyond ScanRefer/3DVLP (Tab. 2(a))**\n\nWe agree with the reviewer that demonstrating broader applicability across more recent 3DVG baselines would strengthen the generality of our framework. In our revision, we have extended the Refer-Judge experiment to include ConcreteNet [ECCV 2024], a recently published and strong 3DVG baseline.\n\nDue to the relatively heavy architecture of ConcreteNet, reproducing full training following the official configuration required approximately 4–5 days on our hardware setup. As of submission, we have completed 235 out of 400 training epochs on our Refer-Judge-purified ScanRefer dataset. Encouragingly, we already observe an improvement of +0.55 in Overall Acc@0.5 over the original ConcreteNet baseline, and we expect further gains once full training is complete. These updated results will be included in our revised manuscript.\n\nIn addition, we emphasize that our original choice of ScanRefer and 3DVLP as testbeds was deliberate. These models represent different levels of model complexity and data fitting capacity, allowing us to assess whether data toxic impacts both lightweight and powerful 3DVG systems. The consistent performance gains across these diverse architectures support our broader claim: toxic data negatively affects a wide range of 3DVG models, making data auditing an essential step in trustworthy model training.\n\n**Q2: Construction of the misleading view set (Line 188)**\n\nThank you for pointing out this potential ambiguity. We realize our description of the positive set and the misleading set construction may not have been sufficiently clear in the original manuscript.\n\nTo clarify, the generation of these two image sets is grounded in the existing annotations of the ScanRefer and ScanNet datasets. Specifically:\n\n- For each referring expression in ScanRefer, the corresponding ground truth object is identified. Leveraging the alignment between ScanNet's RGB frames and object-level 3D annotations, we determine which images in the scene contain visible projections of the target object. These images comprise the positive set, ensuring that each view indeed depicts the intended referent.\n- In contrast, the misleading set is formed by selecting images from the same scene that do not contain the ground truth object, but do contain other instances of the same object category. These views are intentionally constructed to be partially correct yet referentially incorrect, serving as distractors that may tempt a model to violate exclusivity constraints.\n\nThis construction protocol ensures that images in the I_mis set provide both visual and semantic ambiguity, which is essential for evaluating whether the referring expression is uniquely resolvable. \n\nWe will revise the manuscript to clarify this process in more detail.\n\n**Q3: Inference time per scene**\n\nWe appreciate the reviewer’s interest in the runtime characteristics of Refer-Judge. However, we note that per-scene inference time is not a meaningful performance bottleneck in our setting, as the framework is designed for offline data curation, not real-time inference.\n\nIn practice, thanks to the independent nature of each sample’s evaluation, Refer-Judge naturally supports batch-mode parallelization. With a 32-thread setup, the average end-to-end time (including all four juror evaluations and judge refinement) is approximately 6 seconds per sample.\n\nGiven that Refer-Judge is intended for one-time dataset purification, whether for improving 3DVG training reliability or constructing instruction-following benchmarks for MLLMs, throughput is not a limiting factor, and quality is prioritized over latency.\n\n**Closing remarks**\n\nWe hope these clarifications and additional experiments address your concerns. Thank you again for your constructive and supportive review."}, {"review_id": "gvE5p8q4zU", "reviewer": "Reviewer_63DY", "summary": "This paper introduces a novel  Refer-Judge framework designed to remove ambiguous or paradoxical annotations in 3D visual grounding (3DVG) datasets. Inspired by judicial deliberation, multiple jurors analyze each sample from four complementary perspectives, and their outputs are then refined and consolidated by a panel of judges. Experiments on the ScanRefer-Justice dataset show that Refer-Judge achieves human-comparable performance in detecting problematic samples and that filtering toxic data leads to improvements in downstream 3DVG tasks.", "questions": "1. The author should discuss the relation between computation cost (e.g., by token or flops) and the refinement gain, which is critical for the scalability of the proposed method.\n2. Potential bias analysis would also be helpful since existing MLLMs exhibit a lot of hallucination.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength\n\n1. The method closely mimics human deliberative processes and yields reasonable performance improvement.\n2. The ablation study comprehensively demonstrates that the designed modules are important.\n\nWeakness\n1. The proposed framework involves a large amount of computation by querying strong and expensive MLLMs multiple times, while the performance improvement on downstream tasks seems limited.\n2. The quality of refinement seems to largely rely on the MLLM choice, while there is little discussion of failure cases or model bias.", "was_revised": true, "final_justification": "I appreciate the authors' effort and detail response. Thanks for pointing out that GPT-4.1-mini could achieve comparable results with lower price. The performance analysis is reasonable and I will raise my score.", "rebuttal": "We sincerely thank the reviewer for the thoughtful feedback and detailed comments. Below, we address your concerns regarding scalability, computational cost, and model dependency.\n\n**Q1: Computation cost vs. refinement gain (scalability concern)**\n\nWe would like to clarify a potential misunderstanding regarding computational cost. While Refer-Judge involves multiple structured MLLM queries per scene, its overall cost remains manageable and significantly lower than human annotation for comparable tasks.\n\nAs shown in Table 5 (ID 4), a full Refer-Judge pass on the ScanRefer-Justice dataset (3,001 samples) consumes approximately 112.8M tokens, with 97.16M input tokens and 15.64M output tokens. Based on current GPT-4o pricing (USD 1.25/M input, USD 5/M output), this translates to 200 USD total. Scaling up to the full ScanRefer dataset (51,583 expressions), the estimated cost would be 2,500 USD, as the original dataset contains simpler cases, which require fewer visual and ambiguity evaluations.\n\nImportantly, this cost is a one-time, offline expenditure. Refer-Judge is designed for data curation or pretraining supervision, where runtime latency is less critical than quality assurance. In contrast, human annotation of ScanRefer-Justice required over 200 human-hours by 5 trained annotators, which is not only more expensive, but also less scalable or reproducible.\n\nTo further reduce the running cost of Refer-Judge, we also recommend using a more cost-effective MLLM. As we reported in Table 1, using GPT-4.1-mini as a replacement also produces comparable results, and the inference overhead on the ScanRefer-Justice dataset can be further reduced to a negligible $32, demonstrating the scalability potential of our work.\n\nMeanwhile, the discussion on the correlation between computation cost and the quality of toxic data evaluation has been fully discussed in Appendix C. The ablation experiment in Table 8 clearly shows the impact of each component in Refer-Judge on token cost and its correlation with the refinement gain.\n\nWe hope this clarifies that Refer-Judge is both practically affordable and structurally scalable for real-world deployment in offline data refinement.\n\n**Q2: Limited improvement on downstream tasks**\n\nWe respectfully clarify that Refer-Judge leads to consistent and non-trivial improvements on downstream 3DVG models, without modifying their architectures. As shown in Table 2(a), filtering the training data with Refer-Judge improves Overall Acc@0.5 by +1.43% (ScanRefer) and +1.6% (3DVLP).\n\nAdditionally, we believe that baseline models trained on the original ScanRefer dataset may implicitly learn annotation priors embedded in toxic samples. In cases involving paradoxes or ambiguities, human annotators may exhibit consistent biases in selecting certain objects as ground truth (e.g., favoring closer or more salient targets). This leads to models overfitting to these spurious but learnable annotation patterns, rather than truly reasoning over spatial and semantic consistency.\n\nSpecifically, our analysis in Appendix C (Table 7) reveals that baseline models may implicitly overfit to annotation biases in toxic samples. For example, the baseline model outperforms 3DVLP+Refer-Judge on toxic data, even though the referring expressions in those samples may be ambiguous, paradoxical, or logically invalid. For example, at a toxic threshold of ≤2, the baseline even achieves +2.29% higher Acc@0.5 on toxic samples, indicating inflated scores due to spurious correlations. In contrast, Refer-Judge-filtered models achieve more reliable performance on valid data and show greater gains as the toxic threshold expands (e.g., +2.16% at threshold ≤4), further supporting the value of data purification.\n\nWe would also like to emphasize that data quality remains the cornerstone of deep learning. While we are deeply grateful for foundational contributions such as ScanRefer in advancing 3D visual grounding, we believe that identifying and addressing latent flaws in these datasets is equally essential for the long-term progress of 3DVG and spatial intelligence.\n\nAs stated in our conclusion, “*We hope this work offers a new lens on dataset reliability in 3D scene understanding and serves as a foundation for more trustworthy training and evaluation practices.*” Beyond improving model performance, our goal is to spark broader reflection on the reliability of mainstream 3DVG datasets, especially given the inherent challenges of annotation in such complex, multimodal settings.\n\n**Q3: Model bias and failure case discussion**\n\nThank you for raising this important concern.\n\nTo evaluate the generalizability and robustness of the Refer-Judge framework, we tested it using a diverse set of MLLMs, including GPT-4o, GPT-4.1-mini, Gemini 2.5, Grok-3, and LLaMA-3.2 (11B). These models differ significantly in architecture, training corpus, and reasoning capabilities. Naturally, we observed performance variance across models. For example, LLaMA-3.2, a much smaller open-source model, achieved a lower agreement score (67.88%), underscoring the difficulty of toxic data identification in complex, multi-view 3D settings.\n\nHowever, we would like to clarify that these discrepancies reflect model capability gaps, rather than indicating a lack of robustness in the Refer-Judge framework itself. In fact, the component ablation experiments based on GPT-4o (see Table 8) consistently demonstrate the contribution of each module to final prediction quality. We welcome future work that leverages our proposed ScanRefer-Justice benchmark to further evaluate and enhance these findings across broader model families.\n\nOn the other hand, we acknowledge that our current manuscript lacks a detailed discussion of failure cases, particularly those arising from MLLM hallucinations. While Appendix D offers brief visualizations for normal, hard, and complex scenes, the error analyses are not sufficiently comprehensive. In our revision, we will include a deeper case study section focusing on these issues.\n\nFor example, in the reported *normal case*, the uniqueness refinement module hallucinated that a TV *\"fails to match the specified `square` shape,\"* which distorted the final judgment. This failure is traceable in the District Judge’s score breakdown, where the corresponding distinguishability rank was marked as [10: 2]. Nevertheless, because the other visual views were accurately interpreted, the final aggregated confidence score remained high, demonstrating Refer-Judge's resilience to localized hallucinations.\n\nWe will also correct the mismatched image ordering in the previously reported case studies to avoid confusion.\n\n\n**Closing remarks**\n\nWe thank the reviewer again for raising these important points. We hope our clarifications and additional analysis have addressed your concerns and demonstrated the practicality, scalability, and reliability of Refer-Judge. We respectfully ask the reviewer to reconsider the overall evaluation."}, {"review_id": "yiuLBAYQpj", "reviewer": "Reviewer_Udtb", "summary": "This paper addresses the issue of toxic data in 3D Visual Grounding (3DVG) datasets, which contain paradoxical or ambiguous annotations that hinder model performance. The authors propose Refer-Judge, a framework inspired by the judicial system’s deliberative process, using Multimodal Large Language Models (MLLMs) to detect and filter such toxic data.\n\nThe framework employs a Jury-and-Judge Chain-of-Thought paradigm:\n- Jurors evaluate data from four perspectives (Logic, Consistency, Distinguishability, Ambiguity), identifying logical contradictions and ambiguities.\n- Judges refine these evaluations using a Corroborative Refinement strategy, reorganizing visual/textual evidence to resolve uncertainties.\n\nExperiments on the ScanRefer-Justice dataset show Refer-Judge achieves human-level discrimination and improves baseline model performance when toxic data is removed. The core contribution is a principled, multi-perspective reasoning system for 3DVG data quality assessment.", "questions": "- Reproducibility Crisis: Why does the paper omit code for the \"stitching operation\" and Jury-Judge prompting templates? Without these, results cannot be verified.\n- Biased Dataset Analysis: How does Refer-Judge perform on ScanRefer's original test set, which has a different complexity distribution? The reported gains on ScanRefer-Justice may not generalize.\n- Cost-Effectiveness Failure: A $10,000 budget for GPT-4o API calls is prohibitive for most labs. Why not compare against cheaper alternatives (e.g., Llama-3.2 with fine-tuning)?.\n- Theoretical Vacuity: What is the formal relationship between the judicial analogy and the proposed reasoning stages? The metaphor lacks predictive power for model design.", "limitations": "- Computational Infeasibility: The paper downplays high costs, failing to compare against lightweight methods (e.g., rule-based filters) for toxic data detection.\n- Dataset Bias: ScanRefer-Justice overrepresents complex scenes (20% complex samples vs. 6.3% in ScanRefer), skewing performance metrics.\n- MLLM Dependency: Performance collapses with open-source LLMs (e.g., LLAMA-3.2 achieves 67.88% agreement), yet the paper frames GPT-4o results as a universal solution.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n- Original Framework Design: The judicial-inspired Jury-Judge paradigm is novel, providing structured multi-perspective reasoning absent in prior work. This decomposes complex scene-level evaluation into manageable dimensions.\n- Comprehensive Evaluation: By integrating logical, consistency, and visual distinguishability checks, Refer-Judge addresses multiple sources of toxic data (e.g., internal contradictions, referential ambiguity).\n- Empirical Validation: Experiments on ScanRefer-Justice and ScanRefer demonstrate significant improvements in data purification and model performance, with GPT-4o achieving near-human accuracy.\n- Practical Impact: The framework directly addresses a critical challenge in 3DVG, where dataset quality has been overlooked. Filtering toxic data leads to tangible gains in downstream tasks.\n\nWeakness\n- Lack of Theoretical Innovation: The judicial analogy is superficial, merely renaming standard multi-agent reasoning (jurors) and refinement (judges). Concepts like \"twin thought\" replicate existing CoT strategies (e.g., parallel reasoning in GoT [3]) without novel theoretical grounding.\n- Inadequate Evaluation Scope: Experiments exclusively use ScanRefer-Justice, a dataset curated to emphasize complex scenes.\n- Performance on real-world 3DVG datasets (e.g., SUNCG) or outdoor environments is untested, undermining generalizability.\n- Questionable Computational Efficiency: The framework incurs exorbitant costs ($10,000 in API fees) for marginal gains (e.g., +1.6% Acc@0.5 for 3DVLP). This makes it impractical for large-scale datasets, a critical limitation unaddressed in the paper.\n- Flawed Baseline Comparisons: Baseline models (e.g., ScanRefer, 3DVLP) are evaluated on purified data without controlling for data splitting biases. The reported improvements may stem from dataset curation rather than the framework itself.\n- Ambiguous Refinement Mechanisms: The Corroborative Refinement strategy lacks clear definitions. For example, \"stitching operation\" (sti·) is undefined, and the replacement of evaluations (replace()) lacks transparency, making reproducibility impossible.", "was_revised": true, "final_justification": "Based on the reviews and rebuttals, i've updated my review to boarderline accept.", "rebuttal": "We sincerely thank the reviewer for the detailed feedback and the opportunity to clarify and strengthen our submission. Below, we address each of the raised concerns.\n\n**Q1: Theoretical framing and judicial analogy**\n\nWe respectfully disagree that the proposed framework merely renames standard reasoning mechanisms. While Refer-Judge is rooted in CoT-based LLM reasoning, it is, to the best of our knowledge, the first agentic, deliberation-based system aimed specifically at evaluating complex dataset quality in 3DVG. We believe this Jury-and-Judge CoT design represents a structured and extensible paradigm for LLM-as-a-Judge applications. Its value lies in a novel and structured formulation of LLM-based data auditing—a dimension increasingly relevant in the era of foundation models.\n\nWe appreciate prior efforts such as GoT, which explore topological variations of CoT graphs. However, as we stated in our related works, *most CoT-based methods are tailored for narrow-scope tasks, including arithmetic reasoning (like GoT) or short image-text QA (like CoCoT, DCoT)*. These methods demonstrate the existence of complex CoT topologies, but not their utility in executing open-ended, multi-perspective tasks like data auditing in 3DVG.\n\nOur framework introduces:\n\n- Modular Role-Driven Reasoning: Jurors are semantically differentiated agents, each targeting a key perspective: logical, consistency, distinguishability, and ambiguity. This design goes beyond ensemble prompting, aligning with how real-world juries solicit diverse perspectives. Unlike GoT-style parallel agents, Refer-Judge enforces cross-perspective reasoning and inter-agent role separation.\n- Corroborative Refinement and Decision Aggregation: The judge layer is not a majority vote but a structured aggregation then re-evaluation mechanism, including conflicts recognition, information reorganizing, and corroborative scoring. This mitigates the effects of unreliable observations. As shown in Table 4, our refinement significantly outperforms both GoT and CoT-SC, underscoring its effectiveness.\n- Twin Thought Reuse Across Roles: While structurally parallel, our Twin Thought concept reflects multi-role reuse of reasoning traces. For example, existence judgments are shared across J_a and J_d but differ in their inputs (positive vs. misleading view sets), resulting in divergent semantic functions.\n\nWhile we acknowledge that Refer-Judge is not focused on in-depth exploration of CoT topology, we believe this framework represents methodological innovations in LLM-as-a-Judge field.\n\n**Q2: Evaluation Scope and Generalizability (Dataset Bias)**\n\nWe acknowledge that ScanRefer-Justice emphasizes more complex and ambiguous cases, and we explicitly stated this design choice in the paper. Our goal was to create a stress-test benchmark that evaluates grounding robustness under challenging conditions.\n\nAs noted in Appendix A.2, manual validation of toxic samples requires significantly more effort than initial annotation. Consequently, it was infeasible for us to manually validate the full ScanRefer dataset. Even for the ScanRefer-Justice subset, five trained annotators spent over 200 human-hours.\n\nMoreover, we believe that concentrating on these “pressure samples” is critical for revealing the limits of current MLLMs and understanding the comparative strengths of different Refer-Judge components. Particularly in scenarios where full human verification is impractical, ensuring sufficient coverage of complex edge cases is more impactful than uniformly sampling the entire dataset. \n\n**Q3: Expensive Computational cost**\n\nThank you for raising the cost concern. We would like to clarify that the $10,000 figure reflects our cumulative expenditure across all stages during the writing and experimentation phase. The actual cost of running Refer-Judge for data filtering is significantly lower and well within a practical range.\n\nAs shown in Table 5, a complete Refer-Judge pass over the ScanRefer-Justice dataset involves approximately 112.8M tokens (97.16M input, 15.64M output). Based on current API pricing  (USD 1.25/M input, USD 5/M output), this equates to a total cost of 200 USD. Scaling up to the full ScanRefer dataset, the estimated cost would be 2,500 USD, for the original dataset contains simpler cases, which require fewer visual and ambiguity evaluations.\n\nFurthermore, as demonstrated in Table 1, substituting GPT-4o with GPT-4.1-mini achieves comparable performance at a much lower cost, ~$32 for the ScanRefer-Justice dataset. This highlights the scalability and adaptability of Refer-Judge.\n\nCompared to manual annotation or the development of handcrafted rules for scene-specific ambiguity detection, we believe Refer-Judge offers a more cost-effective, reusable, and modular solution for dataset purification in 3DVG and beyond.\n\n**Q4: Baseline comparison and dataset split bias**\n\nWe clarify that in Table 2(a) and Table 6, both ScanRefer and 3DVLP models are evaluated on the original ScanRefer test set, using identical train/val/test splits for fair comparison. This ensures our data purification impacts training quality.\n\nAt the same time, we also clearly explained in the original manuscript that the experiments in Table 2 (b) and Table 7 further evaluated the performance of the model on toxic data and purified data. As we mentioned in the response of Q4, this move was to analyze whether the model would overfit the toxic samples.\n\n**Q5: Ambiguous Mechanism Descriptions and Reproducibility**\n\nWe appreciate the reviewer’s comment regarding the clarity of terminology and operations in the manuscript. We acknowledge that the “stitching operation” and “replacement operation” were insufficiently explained in the original text, and we will provide clearer definitions in the revised version.\n\nConcretely, the stitching operation refers to a simple visual composition procedure applied to Corroborative Pairs: two images, one being the suspicious view and the other the assistance view. These two images are horizontally concatenated with aligned edges, as illustrated by the thumbnail in Figure 3, to form a richer and more context-aware visual prompt. This stitched image provides the judge with a more complete reference for final assessment.\n\nThe replacement operation substitutes the original (possibly erroneous or low-confidence) score and rationale produced by individual jurors with the refined outputs from the Judge module, which integrates broader evidence and resolves conflicts. This enables more accurate and globally consistent evaluations.\n\nRegarding reproducibility, we note that the main paper presents a simplified view of the prompting templates and data flow operations for the sake of clarity. However, to support reproducibility, we have included all implementation details in the supplementary materials. These include:\n\n- Preprocessing scripts for both ScanRefer and ScanRefer-Justice\n- The end-to-end pipeline for Refer-Judge\n- The used prompting templates\n- The ScanRefer-Justice benchmark\n\nWe hope this clarifies the implementation of our proposed method.\n\n**Q6: MLLM Dependency and Lack of Fine-Tuning on Open-Source LLMs**\n\nIndeed, performance varies by model. LLaMA-3.2 achieves lower agreement, showing current limitations of open models. However, Refer-Judge remains functional across all tested LLMs. The modular design and corroboration mechanisms mitigate individual hallucinations, as shown in Appendix D.\n\nAs for fine-tuning LLaMA or other open models, we considered this, but such tuning requires substantial task-specific annotations (e.g., annotating ambiguity types, visual correspondence under multiple views, and instance-level scene justifications). This would require significant annotation effort, defeating the purpose of scalable data auditing.\n\nMoreover, as stated in Q3, cost is already manageable with GPT-4.1-mini, which offers excellent cost-performance trade-offs.\n\n**Q7: Broader Evaluation and Generalization**\n\nWe thank the reviewer for highlighting the importance of broader evaluation. We agree that validation beyond a single dataset improves the perceived generalizability of Refer-Judge. \nIn response to your suggestion and that of Reviewer # KTYR in limited time, we first conducted a small-scale study on the Nr3D dataset, which provides viewpoint splits and distractor-based setups. Manual review by human experts confirmed that ambiguous and paradoxical expressions also exist in Nr3D.\n\nEncouragingly, Refer-Judge aligned well with expert judgment in these cases. For example, for sample #23273 (*The brown arm chair below the painting close to the purple couch*), although the original annotation marked the listener’s pick as correct, the scene (scene0520_00) actually contains two valid armchairs below different paintings, making the expression ambiguous. Refer-Judge successfully identified this.\n\nInterestingly, we also observed cases where annotations were marked as failed in Nr3D (due to speaker-listener mismatch) but turned out to be valid on re-check. For instance, sample #39432 (*It is a shelf with a poster on it touching the corner of the room behind the green chair*), though judged incorrect in Nr3D, can be clearly grounded to a unique shelf (seen in 000500.jpg of scene0565_00) by both experts and Refer-Judge.\n\nWe will include these findings in the revised version, and we welcome future work extending Refer-Judge to other datasets such as Sr3D and SUNCG.\n\n**Closing remarks**\n\nWe thank the reviewer again for their rigorous critique. We will revise the paper to clarify system design, expand evaluation scope, and emphasize reproducibility and cost-efficiency. While Refer-Judge may not introduce new theoretical formalisms, we believe it makes a methodologically novel contribution to dataset quality assessment in 3DVG, offering an effective, interpretable, and modular solution to a widely underexplored challenge. We respectfully ask the reviewer to reconsider the overall evaluation."}, {"review_id": "AwUW2dmQSs", "reviewer": "Reviewer_KTYR", "summary": "This paper proposes Refer-Judge, a reasoning-based framework for improving data quality in 3D visual grounding tasks. The core idea is to apply a two-stage Jury-and-Judge Chain-of-Thought (CoT) mechanism using Multimodal Large Language Models (MLLMs) to detect and refine “toxic” (logically inconsistent or ambiguous) data. Jurors provide diverse judgments, which are consolidated by a Judge module using a \"Corroborative Refinement\" step. The refined dataset is then shown to improve model performance across various evaluation splits of the ScanRefer dataset.", "questions": "See Weaknesses\n\n**6. Computation and Cost Concerns**\nRunning MLLMs over large-scale 3D datasets (like ScanRefer) can be computationally expensive. How many total model calls are required?\nHave the authors estimated the cost in terms of GPU-hours or dollar cost for filtering ScanRefer?\nWould it be more efficient or cost-effective to rely on human annotators instead? At the very least, a comparison of cost and scalability between human and MLLM-based evaluation would add realism to the proposed solution.", "limitations": "Yes, however, it is a significant limitation, specifically with marginal gain.", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "---\n**Strengths**\n\n1- Targeted Problem: The paper addresses a real and often overlooked challenge in 3D visual grounding—noisy and ambiguous annotations that arise from scene complexity and limited field of view.\n\n2- High-Level Conceptual Framing: The use of a jury-and-judge metaphor combined with Chain-of-Thought reasoning is intuitive and aligns well with recent trends in using LLMs for structured evaluation.\n\n3- Empirical Signals: The purified dataset shows measurable improvements across multiple performance metrics and split types (unique, multiple, and toxic data), suggesting some practical benefit.\n\n---\n**Weaknesses and Suggestions for Improvement**\n\n**1. Limited Methodological Novelty:**\nThe proposed pipeline essentially repurposes existing MLLMs for multi-perspective reasoning. While practically useful, the approach lacks significant algorithmic novelty. It primarily orchestrates structured prompting with strong pre-trained models.\n\nThe reliance on MLLMs without any architectural or methodological contribution makes it unclear how much of the performance gain comes from the framework itself versus the capabilities of the underlying model.\n\n\n**2. Overclaims and Writing Style:**\nThe writing often overstates the technical depth (e.g., referring to structured prompting as a “framework” with “corroborative refinement”) and lacks precision in describing concrete steps.\nThe abstract and introduction are heavy on conceptual metaphors but thin on details, which hinders readability and makes the contributions harder to parse.\n\n\n**3. Missing or Incomplete Related Work**\nThe paper omits several key and recent works in 3D visual grounding [1,2,3]\nThese works provide strong baselines and theoretical frameworks that should have been acknowledged and compared against.\n\n\n**4. Experimental Limitations**\nSingle-Dataset Evaluation: The framework is only evaluated on ScanRefer, while Nr3D and Sr3D are more controlled benchmarks for reasoning and ambiguity. Particularly, Nr3D’s viewpoint splits and distractor-based setups align well with the paper’s motivation and should not be omitted.\n\n\nTable 1 Clarity:\nThe input modality is unclear—are models using raw point clouds, RGB images, or both?\nHow are Precision, Recall, and RMSE computed? What is the ground truth (e.g., bounding box centers or full masks)?\nThe table appears to mix classification-style and regression-style metrics without a proper explanation.\n\nTable 2a:\nOnly two baselines are fine-tuned using purified data. Given the broad landscape of 3DVG models, this is insufficient. The omission of ConcreteNet, a strong recent model, is especially concerning.\n\nTable 2b:\nThe results on toxic data are confusing. For instance, purified models sometimes perform worse than their baseline counterparts. Why is that the case?\nA key point is missing: if the evaluation set is toxic, then models trained on purified data may appear worse due to mismatched assumptions. A random baseline or distractor-aware baseline could help contextualize the drop.\n\n**5. Ambiguity in the Dataset Filtration Process**\nThe paper mentions \"positive sets\" (e.g., Line 172) without clarifying how they are constructed. Are they human-verified, heuristically generated, or filtered by some confidence threshold?\nThe filtration process should be clearly specified: how many MLLM calls per sample? Are jurors prompted independently or jointly?\nIf MLLMs are trusted enough to curate data, why not simply rely on them during inference as well? This tradeoff deserves discussion.\n\n[1] 3DRP-Net: 3D Relative Position-aware Network for 3D Visual Grounding \n[2] Viewpoint-Aware Visual Grounding in 3D Scenes \n[3] CoT3DRef: Chain-of-Thoughts Data-Efficient 3D Visual Grounding", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the detailed feedback and the opportunity to clarify and strengthen our submission. Below, we address each of the raised concerns.\n\n**Q1: Limited Methodological Novelty**\n\nWe respectfully disagree that the proposed method merely renames standard reasoning mechanisms. While Refer-Judge is rooted in CoT-based LLM reasoning, it is, to the best of our knowledge, the first agentic, deliberation-based system aimed specifically at evaluating complex dataset quality in 3DVG. We believe this Jury-and-Judge CoT represents a structured and extensible paradigm for LLM-as-a-Judge applications. Its value lies in a novel and structured formulation of LLM-based data auditing—a dimension increasingly relevant in the era of foundation models.\n\nWe appreciate prior efforts such as GoT, which explore topological variations of CoT graphs. However, as we stated in our related works, *most CoT-based methods are tailored for narrow-scope tasks, including arithmetic reasoning (like GoT) or short image-text QA (like CoCoT, DCoT)*. These methods demonstrate the existence of complex CoT topologies, but not their utility in executing open-ended, multi-perspective tasks like data auditing in 3DVG.\n\nOur framework introduces:\n\n- Modular Role-Driven Reasoning: Jurors are semantically differentiated agents, each targeting a key perspective. This design goes beyond ensemble prompting, aligning with how real-world juries solicit diverse perspectives. Unlike GoT-style parallel agents, Refer-Judge enforces cross-perspective reasoning and inter-agent role separation.\n- Corroborative Refinement: The judge layer is not a majority vote but a structured aggregation then re-evaluation mechanism, including conflicts recognition, information reorganizing, and corroborative scoring. This mitigates the effects of unreliable observations. As shown in Table 4, our refinement significantly outperforms both GoT and CoT-SC, underscoring its effectiveness.\n\nWhile we acknowledge that Refer-Judge is not focused on in-depth exploration of CoT topology, we believe this method represents methodological innovations in LLM-as-a-Judge field.\n\nMeanwhile, to verify the role of each component in Refer-Judge, we also conducted extensive ablation experiments. As shown in Table 8, the contribution of each module to toxic data uncovering is clearly distinguishable.\n\n**Q2: Overclaims and Writing Style**\n\nWe appreciate this point and will revise the paper to: (1) reduce metaphorical overstatements, and (2) clarify all modules with more technical precision.\n\nThat said, we would like to clarify our intent behind the “Jury-and-Judge” analogy. We believe this is not a superficial metaphor, but a natural and useful abstraction for the emerging MLLM-as-a-Judge field. It captures the essence of multi-perspective deliberation and structured refinement, echoing real-world adjudication processes that aggregate diverse evidence and viewpoints.\n\nMoreover, as detailed in our response to Q1, the Corroborative Refinement process is not merely “structured prompting,” but a new refinement method that performs inter-judgment consistency checking.\n\n**Q3: Missing or Incomplete Related Work**\n\nThank you for highlighting this. These are indeed important and highly relevant theoretical precedents. We will revise the Related Work section to explicitly incorporate and discuss the following recent advances:\n\n- 3DRP-Net: Enhances 3DVG by modeling relative spatial relations\n- VPP-Net: Emphasizes viewpoint positioning\n- CoT3DRef: Introduces CoT into 3DVG with a Seq2Seq approach\n\n**Q4: Experimental Limitations**\n\n**Part 1: Dataset Scope:**\n\nWe acknowledge that evaluating solely on ScanRefer may limit the perceived generalizability of Refer-Judge. Therefore, due to time constraints, we first conducted a small-scale study on the Nr3D dataset. Manual review by human experts confirmed that toxic samples also exist.\n\nEncouragingly, Refer-Judge aligned well with expert judgment in these cases. For example, for sample #23273 (*The brown armchair below the painting close to the purple couch*), although the original annotation marked the listener’s pick as correct, the scene (scene0520_00) actually contains two valid armchairs below different paintings, making the expression ambiguous. Refer-Judge successfully identified this.\n\nInterestingly, we also observed cases where annotations were marked as failed in Nr3D (due to speaker-listener mismatch) but turned out to be valid on re-check. For instance, sample #39432 (*It is a shelf with a poster on it touching the corner of the room behind the green chair*) can be clearly grounded to a unique shelf (seen in 000500.jpg of scene0565_00) by both experts and Refer-Judge.\n\nWe will include these findings in the revised version, and we welcome future work extending Refer-Judge to other datasets.\n\n**Part 2: Clarification on Table 1**\n\nWe thank the reviewer for requesting clarification. As described in Section 3.1, Refer-Judge uses RGB images as the primary observation modality throughout the reasoning process. \n\nRegarding the evaluation metrics, we provide a detailed definition in Appendix B. Specifically:\n- For classification-style metrics, we adopt human expert annotations as binary ground truth: we define toxic samples as those scoring 1–6, while valid samples are those scoring 7–9. This binary formulation enables us to assess whether Refer-Judge can reliably match human-level identification of flawed annotations.\n- For regression-style metrics, since Refer-Judge produces scores in the same 1–9 range as the ScanRefer-Justice expert annotations, RMSE offers a fine-grained view of numerical alignment.\n\n**Part 3: Table 2a Coverage**\n\nWe agree with the reviewer that demonstrating broader applicability across more recent 3DVG baselines would strengthen the generality of our framework. In our revision, we have extended the Refer-Judge experiment to include ConcreteNet.\n\nDue to the relatively heavy architecture of ConcreteNet, reproducing full training following the official configuration required approximately 4–5 days. As of submission, we have completed 235/400 training epochs. Encouragingly, we already observe an improvement of +0.55 in Overall Acc@0.5 over the original ConcreteNet, and we expect further gains once full training is complete. These updated results will be included in our revised manuscript.\n\nWe are confident that completing the training will yield further gains, and we will include the results in the revised version of the paper.\n\n**Part 4: Table 2b Interpretation**\n\nThank you for highlighting this. This observation indeed reveals a critical property we discovered during the evaluation of Refer-Judge's impact on baseline models.\n\nAs reported in Table 2b and further elaborated in Appendix C (Table 7), baseline models often exhibit higher performance on toxic data, even though the referring expressions in those samples may be ambiguous, paradoxical, or logically invalid. For instance, at a toxic threshold of ≤2, the baseline model achieves +2.29% higher Acc@0.5.\n\nWe believe this phenomenon stems from implicit overfitting to annotation artifacts in flawed samples, where the model may learn dataset-specific priors rather than generalizable grounding capabilities. This raises concerns about evaluation reliability when training and test data contain similar toxic patterns.\n\nFurthermore, the experiments in Appendix C show that as the toxic threshold becomes more inclusive (i.e., including moderately flawed samples), Refer-Judge-filtered models achieve increasingly stronger gains (e.g., +2.16% at threshold ≤4), reinforcing the benefit of data purification.\n\n**Q5: Ambiguity in Dataset Filtration Process**\n\nThank you for pointing this out. We acknowledge that our original explanation of the positive set and the misleading set construction may have been unclear. We clarify that these sets are not derived via MLLM predictions or heuristic rules.\n\nSpecifically, for a given referring expression in ScanRefer, which targets a unique ground-truth object, we identify ScanNet frames in which this object is visible, constituting the positive set. In contrast, the misleading set consists of frames from the same scene that contain objects of the same semantic category but do not include the ground-truth target object. These misleading views simulate plausible yet incorrect matches and are essential for enabling ambiguity-aware reasoning.\n\nWe will revise the manuscript to clarify this preprocessing process in detail.\n\n**Q6: Computation and Cost Concerns**\n\nThank you for raising this important point. We clarify that the actual cost of using Refer-Judge for dataset auditing is entirely affordable and practical.\n\nAs reported in Table 5, a full Refer-Judge evaluation pass over the ScanRefer-Justice dataset consumes approximately 112.8M tokens (97.16M input, 15.64M output). Based on the current GPT-4o API pricing (USD 1.25/M input tokens, USD 5/M output tokens), this results in a total cost of 200 USD. Scaling to the full ScanRefer dataset, the total estimated cost would be 2,500 USD.\nMore importantly, we also evaluated a cost-efficient variant of our system using GPT-4.1-mini, which completes the Refer-Judge process over ScanRefer-Justice for 32 USD. This highlights that Refer-Judge is highly scalable, modular, and not tied to expensive APIs.\n\nCompared to manual annotation, which required over 200 human hours for 3,001 samples (with an equivalent market cost likely exceeding several thousand USD), our approach offers a significantly more cost-effective and labor-saving alternative.\n\n**Closing Remarks**\n\nWe appreciate the reviewer’s thoughtful critique. While we acknowledge our method is not architecturally novel, we believe Refer-Judge offers a practical and extensible solution to a critical and underexplored problem in dataset reliability. We respectfully ask the reviewer to reconsider their evaluation in light of these clarifications and updates."}]

Author rebuttal:

Meta-review: This paper presents Refer-Judge, a Jury-and-Judge Chain-of-Thought framework that leverages MLLMs to uncover toxic data in 3D visual grounding. The approach is conceptually novel, framing data auditing as a structured deliberative process, and shows human-level discrimination on ScanRefer-Justice along with measurable gains for downstream baselines. Reviewer hBqW found the idea insightful and the experiments comprehensive, recommending acceptance. Reviewer 63DY initially raised concerns about computational overhead and MLLM dependency, but after rebuttal and evidence of affordable cost-effective variants (e.g., GPT-4.1-mini), raised their score. Reviewer Udtb and KTYR emphasized limited methodological novelty and dataset scope, but the rebuttal provided clarifications, additional results on ConcreteNet and Nr3D, and clearer explanations of the filtration process. While not algorithmically deep, the work makes a timely and important contribution to improving dataset reliability in 3D vision-language tasks. Given the strong motivation, effective rebuttal, and potential impact, I recommend acceptance.
Decision: Accept (poster)