Title: Jury-and-Judge Chain-of-Thought for Uncovering Toxic Data in 3D Visual Grounding
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM

Chain-of-Thought

3D Vision Grounding

LLM-as-a-Judge
Abstract: 3D Visual Grounding (3DVG) faces persistent challenges due to coarse scene-level observations and logically inconsistent annotations, which introduce ambiguities that compromise data quality and hinder effective model supervision. To address these challenges, we introduce Refer-Judge, a novel framework that harnesses the reasoning capabilities of Multimodal Large Language Models (MLLMs) to identify and mitigate toxic data. At the core of Refer-Judge is a Jury-and-Judge Chain-of-Thought paradigm, inspired by the deliberative process of the judicial system. This framework targets the root causes of annotation noise: jurors collaboratively assess 3DVG samples from diverse perspectives, providing structured, multi-faceted evaluations. Judges then consolidate these insights using a Corroborative Refinement strategy, which adaptively reorganizes information to correct ambiguities arising from biased or incomplete observations. Through this two-stage deliberation, Refer-Judge significantly enhances the reliability of data judgments. Extensive experiments demonstrate that our framework not only achieves human-level discrimination at the scene level but also improves the performance of baseline algorithms via data purification. Code is available at https://github.com/Hermione-HKX/Refer_Judge.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Refer-Judge, a reasoning-based framework for improving data quality in 3D visual grounding tasks. The core idea is to apply a two-stage Jury-and-Judge Chain-of-Thought (CoT) mechanism using Multimodal Large Language Models (MLLMs) to detect and refine “toxic” (logically inconsistent or ambiguous) data. Jurors provide diverse judgments, which are consolidated by a Judge module using a \"Corroborative Refinement\" step. The refined dataset is then shown to improve model performance across various evaluation splits of the ScanRefer dataset.\n\nSTRENGTHS AND WEAKNESSES:\n---\n**Strengths**\n\n1- Targeted Problem: The paper addresses a real and often overlooked challenge in 3D visual grounding—noisy and ambiguous annotations that arise from scene complexity and limited field of view.\n\n2- High-Level Conceptual Framing: The use of a jury-and-judge metaphor combined with Chain-of-Thought reasoning is intuitive and aligns well with recent trends in using LLMs for structured evaluation.\n\n3- Empirical Signals: The purified dataset shows measurable improvements across multiple performance metrics and split types (unique, multiple, and toxic data), suggesting some practical benefit.\n\n---\n**Weaknesses and Suggestions for Improvement**\n\n**1. Limited Methodological Novelty:**\nThe proposed pipeline essentially repurposes existing MLLMs for multi-perspective reasoning. While practically useful, the approach lacks significant algorithmic novelty. It primarily orchestrates structured prompting with strong pre-trained models.\n\nThe reliance on MLLMs without any architectural or methodological contribution makes it unclear how much of the performance gain comes from the framework itself versus the capabilities of the underlying model.\n\n\n**2. Overclaims and Writing Style:**\nThe writing often overstates the technical depth (e.g., referring to structured prompting as a “framework” with “corroborative refinement”) and lacks precision in describing concrete steps.\nThe abstract and introduction are heavy on conceptual metaphors but thin on details, which hinders readability and makes the contributions harder to parse.\n\n\n**3. Missing or Incomplete Related Work**\nThe paper omits several key and recent works in 3D visual grounding [1,2,3]\nThese works provide strong baselines and theoretical frameworks that should have been acknowledged and compared against.\n\n\n**4. Experimental Limitations**\nSingle-Dataset Evaluation: The framework is only evaluated on ScanRefer, while Nr3D and Sr3D are more controlled benchmarks for reasoning and ambiguity. Particularly, Nr3D’s viewpoint splits and distractor-based setups align well with the paper’s motivation and should not be omitted.\n\n\nTable 1 Clarity:\nThe input modality is unclear—are models using raw point clouds, RGB images, or both?\nHow are Precision, Recall, and RMSE computed? What is the ground truth (e.g., bounding box centers or full masks)?\nThe table appears to mix classification-style and regression-style metrics without a proper explanation.\n\nTable 2a:\nOnly two baselines are fine-tuned using purified data. Given the broad landscape of 3DVG models, this is insufficient. The omission of ConcreteNet, a strong recent model, is especially concerning.\n\nTable 2b:\nThe results on toxic data are confusing. For instance, purified models sometimes perform worse than their baseline counterparts. Why is that the case?\nA key point is missing: if the evaluation set is toxic, then models trained on purified data may appear worse due to mismatched assumptions. A random baseline or distractor-aware baseline could help contextualize the drop.\n\n**5. Ambiguity in the Dataset Filtration Process**\nThe paper mentions \"positive sets\" (e.g., Line 172) without clarifying how they are constructed. Are they human-verified, heuristically generated, or filtered by some confidence threshold?\nThe filtration process should be clearly specified: how many MLLM calls per sample? Are jurors prompted independently or jointly?\nIf MLLMs are trusted enough to curate data, why not simply rely on them during inference as well? This tradeoff deserves discussion.\n\n[1] 3DRP-Net: 3D Relative Position-aware Network for 3D Visual Grounding \n[2] Viewpoint-Aware Visual Grounding in 3D Scenes \n[3] CoT3DRef: Chain-of-Thoughts Data-Efficient 3D Visual Grounding\n\nQUESTIONS:\nSee Weaknesses\n\n**6. Computation and Cost Concerns**\nRunning MLLMs over large-scale 3D datasets (like ScanRefer) can be computationally expensive. How many total model calls are required?\nHave the authors estimated the cost in terms of GPU-hours or dollar cost for filtering ScanRefer?\nWould it be more efficient or cost-effective to rely on human annotators instead? At the very least, a comparison of cost and scalability between human and MLLM-based evaluation would add realism to the proposed solution.\n\nLIMITATIONS:\nYes, however, it is a significant limitation, specifically with marginal gain.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the issue of toxic data in 3D Visual Grounding (3DVG) datasets, which contain paradoxical or ambiguous annotations that hinder model performance. The authors propose Refer-Judge, a framework inspired by the judicial system’s deliberative process, using Multimodal Large Language Models (MLLMs) to detect and filter such toxic data.\n\nThe framework employs a Jury-and-Judge Chain-of-Thought paradigm:\n- Jurors evaluate data from four perspectives (Logic, Consistency, Distinguishability, Ambiguity), identifying logical contradictions and ambiguities.\n- Judges refine these evaluations using a Corroborative Refinement strategy, reorganizing visual/textual evidence to resolve uncertainties.\n\nExperiments on the ScanRefer-Justice dataset show Refer-Judge achieves human-level discrimination and improves baseline model performance when toxic data is removed. The core contribution is a principled, multi-perspective reasoning system for 3DVG data quality assessment.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Original Framework Design: The judicial-inspired Jury-Judge paradigm is novel, providing structured multi-perspective reasoning absent in prior work. This decomposes complex scene-level evaluation into manageable dimensions.\n- Comprehensive Evaluation: By integrating logical, consistency, and visual distinguishability checks, Refer-Judge addresses multiple sources of toxic data (e.g., internal contradictions, referential ambiguity).\n- Empirical Validation: Experiments on ScanRefer-Justice and ScanRefer demonstrate significant improvements in data purification and model performance, with GPT-4o achieving near-human accuracy.\n- Practical Impact: The framework directly addresses a critical challenge in 3DVG, where dataset quality has been overlooked. Filtering toxic data leads to tangible gains in downstream tasks.\n\nWeakness\n- Lack of Theoretical Innovation: The judicial analogy is superficial, merely renaming standard multi-agent reasoning (jurors) and refinement (judges). Concepts like \"twin thought\" replicate existing CoT strategies (e.g., parallel reasoning in GoT [3]) without novel theoretical grounding.\n- Inadequate Evaluation Scope: Experiments exclusively use ScanRefer-Justice, a dataset curated to emphasize complex scenes.\n- Performance on real-world 3DVG datasets (e.g., SUNCG) or outdoor environments is untested, undermining generalizability.\n- Questionable Computational Efficiency: The framework incurs exorbitant costs ($10,000 in API fees) for marginal gains (e.g., +1.6% Acc@0.5 for 3DVLP). This makes it impractical for large-scale datasets, a critical limitation unaddressed in the paper.\n- Flawed Baseline Comparisons: Baseline models (e.g., ScanRefer, 3DVLP) are evaluated on purified data without controlling for data splitting biases. The reported improvements may stem from dataset curation rather than the framework itself.\n- Ambiguous Refinement Mechanisms: The Corroborative Refinement strategy lacks clear definitions. For example, \"stitching operation\" (sti·) is undefined, and the replacement of evaluations (replace()) lacks transparency, making reproducibility impossible.\n\nQUESTIONS:\n- Reproducibility Crisis: Why does the paper omit code for the \"stitching operation\" and Jury-Judge prompting templates? Without these, results cannot be verified.\n- Biased Dataset Analysis: How does Refer-Judge perform on ScanRefer's original test set, which has a different complexity distribution? The reported gains on ScanRefer-Justice may not generalize.\n- Cost-Effectiveness Failure: A $10,000 budget for GPT-4o API calls is prohibitive for most labs. Why not compare against cheaper alternatives (e.g., Llama-3.2 with fine-tuning)?.\n- Theoretical Vacuity: What is the formal relationship between the judicial analogy and the proposed reasoning stages? The metaphor lacks predictive power for model design.\n\nLIMITATIONS:\n- Computational Infeasibility: The paper downplays high costs, failing to compare against lightweight methods (e.g., rule-based filters) for toxic data detection.\n- Dataset Bias: ScanRefer-Justice overrepresents complex scenes (20% complex samples vs. 6.3% in ScanRefer), skewing performance metrics.\n- MLLM Dependency: Performance collapses with open-source LLMs (e.g., LLAMA-3.2 achieves 67.88% agreement), yet the paper frames GPT-4o results as a universal solution.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-05"}

comments: SUMMARY:
This paper introduces a novel  Refer-Judge framework designed to remove ambiguous or paradoxical annotations in 3D visual grounding (3DVG) datasets. Inspired by judicial deliberation, multiple jurors analyze each sample from four complementary perspectives, and their outputs are then refined and consolidated by a panel of judges. Experiments on the ScanRefer-Justice dataset show that Refer-Judge achieves human-comparable performance in detecting problematic samples and that filtering toxic data leads to improvements in downstream 3DVG tasks.

STRENGTHS AND WEAKNESSES:
Strength

1. The method closely mimics human deliberative processes and yields reasonable performance improvement.
2. The ablation study comprehensively demonstrates that the designed modules are important.

Weakness
1. The proposed framework involves a large amount of computation by querying strong and expensive MLLMs multiple times, while the performance improvement on downstream tasks seems limited.
2. The quality of refinement seems to largely rely on the MLLM choice, while there is little discussion of failure cases or model bias.

QUESTIONS:
1. The author should discuss the relation between computation cost (e.g., by token or flops) and the refinement gain, which is critical for the scalability of the proposed method.
2. Potential bias analysis would also be helpful since existing MLLMs exhibit a lot of hallucination.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-05

comments: SUMMARY:
This paper proposes Refer-Judge, an agentic system to detect toxic data in 3D visual grounding. The framework employs four jurors to assess different aspects of the data quality in 3DVG, followed by a judge refinement and aggregation procedure.
The proposed framework achieves an accurate assessment that is close to human performance. 
Experiments show existing 3DVG approaches can benefit from fine-tuning using the purified data.

STRENGTHS AND WEAKNESSES:
Strength:
- The idea and the framework design are insightful.
- The paper is well-written and easy to follow.
- The experiments are comprehensive.

Weakness:
- In Tab. 2 (a), the performances of ScanRefer [1] and 3DVLP [2] show impressive performance gains after using the Refer-Judge framework. However, it will be more persuasive if the framework can consistently boost at least one more recent work (given ScanRefer [1] was published in 2020).
- How is the misleading view set $I_{mis}$ described in L. 188 constructed.

[1] Chen, Dave Zhenyu, Angel X. Chang, and Matthias Nießner. "Scanrefer: 3d object localization in rgb-d scans using natural language." European conference on computer vision. Cham: Springer International Publishing, 2020.
[2] Zhang, Taolin, et al. "Vision-language pre-training with object contrastive learning for 3d scene understanding." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 7. 2024.

QUESTIONS:
- What is the typical process time of one scene?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-07