Do 3D Large Language Models Really Understand 3D Spatial Relationships?
Authors: Xianzheng Ma
TAO SUN
Shuai Chen
Yash Sanjay Bhalgat
Jindong Gu
Angel X Chang
Iro Armeni
Iro Laina
Songyou Peng
Victor Adrian Prisacariu
Venue: ICLR 2026 Poster (2026)
Keywords: 3D-LLM
3D spatial reasoning
3D Situated QA
3D-QA
Primary area: datasets and benchmarks
TL;DR: Existing 3D-QA benchmarks overestimate progress due to textual shortcuts, and our Real-3DQA benchmark plus 3D-reweighted fine-tuning enable more faithful evaluation and stronger 3D reasoning.

Abstract: Recent 3D Large-Language Models (3D-LLMs) claim to understand 3D worlds, especially spatial relationships among objects. Yet, we find that simply fine-tuning a language model on text-only question-answer pairs can perform comparably or even surpass these methods on the SQA3D benchmark without using any 3D input. This indicates that the SQA3D benchmark may not able to detect if the model exploits textual shortcuts rather than engages in 3D-aware reasoning. To address this issue, we introduce Real-3DQA, a more rigorous evaluation benchmark that filters out easy-to-guess questions and introduces a structured taxonomy to assess various aspects of 3D reasoning. Experiments on Real-3DQA confirm that existing 3D-LLMs struggle with spatial relationships once simple cues are removed. We further propose a 3D-reweighted training objective that leverages negative samples via explicit 3D-relation alignment, substantially enhancing 3D-LLMs’ performance in spatial reasoning tasks. Our findings underscore the need for robust benchmarks and tailored training strategies to advance genuine 3D vision-language understanding.

Published as a conference paper at ICLR 2026
Do 3D LARGE LANGUAGE MODELS REALLY UNDER- STAND 3D SPATIAL RELATIONSHIPS?
Xianzheng 22 Tao Sun Shuai Chen Yash Bhalgat' Jindong Gu
Angel X Chang Iro Armeni Tro Laina' Songyou Peng* $ Victor Adrian Prisacariu
VGG, University of Oxford 2 AVL,, University of Oxford Stanford University
Simon Fraser University Google DeepMind
Code & Dataset:
https://real -dt...dq............... wwthub. github.io/
ABSTRACT
Recent 3D Large-Language Models (3D-LLMs) claim to understand 3D worlds, especially spatial relationships among objects. Yet, find that simply fine-tuning a language model on text-only question-answer pairs can perform comparably or even surpass these methods on the SQA3D benchmark without using any 3D input. This indicates that the SQA3D benchmark may not be able to detect if the model exploits textual shorteuts rather than engages in 3D-aware reasoning. To address this issue, we introduce Real-3DQA, a more rigorous evaluation benchmark that filters out casy-to-guess questions and introduces a structured taxonomy to assess various aspects of 3D reasoning, Experiments on Real-3DQA confirm that existing 3D-LLMs struggle with spatial relationships once simple cues are removed. We further propose a 3D-reweighted training objective that guides model to rely more on 3D visual clues, substantially enhancing 3D-LLMs' performance spatial reasoning tasks. Our findings underscore the need for robust benchmarks and tailored training strategies to advance genuine 3D vision-language understanding.
Scene
& tonutreninnge stsamnn unme ohhs hnams
3i iis beliigth with owecitie
Sil ai wetwiing fhw toam lactnnggi printer no rnyright.
Si ern pretangties
Real-3DQA Benchmark Solution
mylat.
More 3D dependent Questions
New Viewpolnt Rotation Score
3D-Reweighted Fine-tuning
Q /Nluniiommmee tormatyo enchory?
Bi Whar the fine black elgectypu
Q: #ih ane the
Weich sirecr de nou hare to walk. couch??
SQA3D
Real-3DQA Perfermence (h)
15
3D-LLM (with 3D input)
piano
armchair
in front me
Blind (ne 3D imput)
piano
couch
bohind
right
Ours (3D-awarm)
piano
backpack
behind
leitt
Figure 1: Our Real-3DQA Benchmark and Solution. We demonstrate the answer differences between 3D-LLM (LEO) and its blind-finetuned version on various questions, with correct answers highlighted in green frames and incorrect ones in red frames. We discover that some questions can be answered correctly regardless of whether 3D information is used, which we consider to be 3D-independent questions. By filtering out these 3D-independent questions and introducing new viewpoint rotation score, the original models' performance drops significantly on Real-3DQA. Finally, using our proposed 3D-aware Reweighted Finetuning strategy, performance improves again.
" 'Equal contribution
Cooresonneence author
I Equal supervision

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Understanding and reasoning about 3I) environments is fundamental for embodied AI, AR/VR applications, autonomous driving, etc. Recent work has proposed 3D Large Language Models (3D-LLMs) Hong et al. (2023); Chen al. (2024); Guo et al. (2023); Huang et al. (2024a), aiming to empower language models with spatial awareness. Early evaluations focused on tasks such as 3D captioning Chen et al. (2020) and object grounding Yang et al. (2024); Achhioptas et al. (2020). while 3D question answering (3D-QA) Azuma et al. (2022): Ye et al. (2021); Etesam et al. (2022) soon emerged as a more flexible benchmark for assessing diverse aspects of spatial understanding. More recently, 3D Situated QA Ma et al.] (2023); (Zhang et al. (2025c); [Lnghu et al. (2025) extended this line of evaluation by adopting an egocentric, first-person perspective, simulating how humans perceive the world and challenging models to reason continuously about objects and relations in context. While these benchmarks are intended to measure progress toward robust embodied intelligence, we question whether the reported improvements truly reflect genuine 3D spatial reasoning. This paper investigates this gap between perceived progress and actual spatial understanding.
We find that much of this apparent progress is illusory. On the most-widely used sit- uated QA benchmarks, i.c. SQA3D Ma
Original m Finetuned
60
60
54.4
ctt al. 2023), where accuracy has increased
55
66
from 30% to over 50% in recent years, 3D- LLMs can achieve competitive scores with- out actually processing any 3D data. A sim- ply fine-tuned blind mode?" finetuned on text-only question-answer pairs can match or surpass the performance of recent 3D- LLMs, as illustrated in Fig. This reveals that many SQA3D questions can be solved solely by linguistic priors, rather than genuine spatial reasoning. Importantly, we observe similar vulnerabilities beyond SQA3D, including on ScanQA and the more recent MSR3D benchmark. Mean- while, our findings echo those of [Huang] et al. ) Li et al.) (2025); /Zhang et al. (2024)) Li et al. (2025) and indicate that even the latest 3D-LLMM are not immune to this gap.
50
-47.8
-47.8
49.2
49.4
50.9
50
49.450.2
45
45
39.940.3
40
40
35
35
-32.3 38
30
30
3DLLMM NeurIPS23
LED ICML
Chat-Soone NeuriP5
Scan(JA CVPR/22
MSR3D
ICLA223
N4urPP924
Various Models SQA3D, [96] on Benchmarks, [%)
Figure 2: Our finding: A language model fine-tuned only text QA pairs without any 3D inputs (Blind Finetuned) can match or even surpass state-of-the-ar 3D-LLMs (Original)
multiple 3D-QA benchmarks. This exposes a critical weakness in current benchmark design and calls into question their ability to assess genuine 3D reasoning despite linguistic shortcuts.
What causes existing benchmarks to fall short in accurately evaluating 3D reasoning? One hypothesis is the presence of bias Vo et al. (2025) in 3D-QA datasets. These biases can stem from both manual annotation Azuma et al. 2022); Ye et al. (202T); Zhang et al. (2025b); Huang et al. (2025) and automatic cemi-automatic LLM assistee generation processes Linghu et al. (2025); Zhang et al. (2025c). where linguistic or commonsense priors are embedded in the data. For example, "What is the black rectangular object on the wall?" almost always yields "TV" for indoor scenes regardless of spatial context. These questions allow models to succeed via shortcut learning instead of true 3I) understanding. While SQA3D Ma al./ (2023) attempt to mitigate this issue by balancing answer distributions, such surface-level fixes cannot eliminate deeper biases, like preferences for salient objects, canonical configurations, or easily guessable answers. These dataset biases are multi-faceted and difficult to eliminate through either manual correction or distribution balancing.
To address this, we introduce Real- -DQA. a benchmark that filters out questions solvable by language priors alone, as summarized in Table [ We compare each 3D-LLM's accuracy to its "blind" counterpart-finetuure
on text-only QA pairs without any 3D inputs. Questions answered correctly by both models are considered low in 3D dependency and are removed, as the answer can be inferred without spatial understanding. By filtering out such trivial questions, we implicitly mitigate a broad range of biases without requiring detailed manual intervention. This model-based filtering allows our Real-3DQA more robustly evaluate genuine 3D reasoning.
We refer readers section 4.1 for details how the blind finetuned model is obtained.
2

---

Published as a conference paper at ICLR 2026
Although removing low-3D- depeendeyyy questions prevents linguistic shorteuts, it does not guarantee that a model truly understands spatial structure. To further assess genuine 3D understanding, we propose cross-question consistency test under viewpoint changes. The key idea is that if model truly grasps the 3D scene, should answer the same question correctly even when the observer's orientation changes. Concretely, we rotate the viewpoint while keeping the position and question fixed, and adjust any directional terms in both the situation description and the expected answer. This results in logically equivalent QA pairs across different reference frames, forcing the model to interpret spatial relations consistently. To quantify this, we further introduce the Viewpoint- -otpttnooooooooonnn Score, which measures whether the model maintains spatial consistency across varied perspectives.
While our proposed benchmark and consistency test offer more rigorous evaluation, they do not by themselves prevent models from learning shortcuts during training. We find that standard supervised fine-tuning still encourages models to learn linguistic shortcuts, limiting the benefit of 3D information. To address this limitation from the training perspective, we introduce a 31D-aware Reweighted Fine- tuning (3DR-FT) strategy. Our key idea is to quantify the 3D dependency of each question by measuring the performance gap between a blind (text-only) model versus a full 3D-LLM. Based on this gap. downweight questions that can be guessed easily from text alone, and upweight those requiring genuine spatial reasoning. This adaptive reweighting encourages models to incorporate 3D cues, and experiments confirm consistent performance gains across multiple 3D-LLMs our revised benchmark.
Overall, our contributions can be summarized below:
We introduce new diagnostic benchmark to measure genuine 3D reasoning ability of 3D-LLMs. It has filtered out questions solvable by linguistic shortcuts and uses a viewpoint-rotatio score to provide stricter and more reliable evaluation.
We reveal that modern 3D-LLMs are surprisingly brittle. On our benchmark, their performance drops by over 60%, and they almost completely fail our viewpoint consistency tests, a critical flaw overlooked by prior benchmarks.
To fix this, we propose a 3D-aware reweighted fine-tuning strategy that forces models to focus more on spatially complex data, consistently boosting the 3D reasoning capabilities of LLMs.
2 RELATED WORK
3D-LLMs. 3D-LLMs integrate spatial data with LLMs for scene understanding Ma et al. (2024b)
including 3D grounding, captioning, and question answering (QA). Prevalent approaches typically follow a formulation similar to 2D vision-
models (VLMs), by concatenating 3D tokens from point-cloud encoders with text tokens before being processed by an LLM Hong al. (2023) ] Yang ; Fu et al. (2024); Xu et al. (2024): Yang et al. (2024); Chen et al. (0024) Li et al. (2024); |uung et al. (2024b); Guo et al. (2023). Others incorporate additional visual signals, such as multi-view images, into point-cloud-based 3D-LLMs. This hybrid design, exemplified by LEO Huang et al. (024bb) and Chat-Scene Huang et al. (2024a). has been shown to improve 3D captioning, grounding, QA, and Situated QA (SQA). More recently, methods such as GPT4Scene Qi et al. (2025), LLaVA-3D Zhu et al. (2024), and Video-3D LLM[Zheng et al.) (2024) sidestep explicit 3D encoders and instead rely on 2D VLMs for 3D awareness. While these works primarily aim to boost model accuracy through architectural and training innovations, we take a different perspective: questioning whether current 3D-LLMs truly acquire consistent understanding of 3D spaces. Our findings suggest that the answer is far from affirmative.
et al. (2023): Wang et al. 2023); Huang et al. (2023):
Linguistic Biases in Multi-modality Models. The issues related to the exploitation of linguistic bias have been extensively studied in 2D-VLMs Aggawal et al. (2018); Niu et al. (202T); Ouyang et al. (22T)) Ma et al. (2024a), leading to fairer benchmarks Agrawal et al. (2018) and mitigation strategics Ma et al. (2024a), such as ensemble learning, data augmentation, self-supervised contrastive learning, and response re-ranking. However, the challenges of mitigating similar exploitation in 3D-QA benchmarks remain largely unexplored. Many existing 3D-QA datasets rely on automatic or semi-automatically generated QA pairs Yan et al. (2023); Li et al. (203);; Qian et al. (2024), which often include language priors in their questions. Although et al. (2023) balances answer categories, it cannot remove deeper annotation artifacts (e.g., saliency preferences, canonical layouts,
3

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of 3D Question Answering Benchmarks. Our Real-3DQA benchmark features situated questions, uses LLM-assisted text collection with human verification, applies debiasing techniques, evaluates robustness across different viewpoints, and provides diverse question categories assess true 3D spatial reasoning ability.
Dataset
Situated
Text collection
Debiased
Robustness evaluation Quality check
Question categories
ScanQA
Human
3D-QA
Human
X
4
SQA3D
Human
/
5
MSQA
L.M
X
ScanReQA
Human
SpartunD
LLM
X
Beacon31)
X
Human
X
V
Real-3DQA
LLM
/
11
or guessable questions). We instead reduce linguistic bias via model-level comparison: by contrasting full model with its blind (text-only) fine-tuned variant, we filter items that are solvable through language priors alone. This consistency-bant -based procedure offers more robust alternative to purely rule-based or distributional balancing.
3D-QA Diagnostic Benchmarks. Existing 3D-QA benchmarks largely reuse the 2D-QA protocol of per-item accuracy Rajpurkar et al. (2016), which overlooks whether predictions self-consistent in 3D spaces. To overcome such issues, the recent diagnostic benchmark Beacon3D Huang et al. (2025), instead assesses 3D understanding through cross-task consistency: whether the model gives consistent answers across both QA and grounding tasks. In contrast, we evaluate 3D understanding ability through cross- gooss-quetioon consistency, where we test if the model consistently solves the question under different viewpoint augmentations.
3 REAL-3DQA BENCHMARK DESIGN
Situated Question Answering (SQA) challenges Al to comprehend its position and surroundings within an environment before responding to queries. The first benchmark for this task, SQA3D Ma et al. (022), has been widely adopted by recent research to evaluate the spatial awareness and 3D reasoning capabilities of 3D-LLMs. However, as shown earlier Figure we discovered a critical flaw in the current evaluation benchmark, which reveals that current 3D-LLMs obtain comparable or even worse results than its blind counterpart trained with text input only.
This observation underscores the need for more rigorous evaluation framework. To address this gap, we introduce Real-3DQA, a benchmark derived from SQA3D to more accurately assess 3D spatial reasoning in 3D-L.LMs. The key innovations in constructing Real-3DQA consist of two main steps: (I) Filtering 3D-independent Questions Section 3.1 we identify and filter out those questions that can be answered by text context alone with high probability, minimizing the model's reliance on dataset biases and ensuring a more genuine assessment of 3D reasoning capabilities. (2) Viewpoint Rotation Score Section 3.2), we introduce a viewpoint rotation evaluation metric that complements existing SQA3D metrics, encouraging models to rely on genuine 3D understanding.
3.1 FILTERING 3D-INDEPENDENT QUESTIONS
In this section, we describe how we remove questions that are "easy-to- coasyy-to-guue to-guess"" without 3D spatial information. Let Q be the original set of test questions from Ma al. 2023). We evaluate three different 3D-LLMs Huang et al. (2024b a): Hong et al. (2023) denoted as MA. Mn. and Mc. For each specific model X, we begin with training the original model Mx. where X B, and its blind counterpart Miiid which is the same model finetuned without 3D inpu
Next, we utilize both Mx and Miidd models to identify questions that are independent of 3D input. Based on our definition, question q Q is considered independent of 3D imput both the original
and finetundd models predict the correct answer: Mx (q) = A hiiind (q) correct. For any given
model Mx- The set of such questions for each model is: Qx = {g E Q Mx = Mblind (g)
We refer readers Section 4.1 for details how Blind Finetuned model M is obtained
4

---

Published as a conference paper at ICLR 2026
Filtering 3D-independent Questions (Sec 3.1)
Rotational Viewpoint Augmentation (Sec 3.2)
927
Both Corrent
High Quality
Rew Dastrs
CPT-40
Expert
Real-3DQA
Fittrred
27tr
Figure 3: Overview of Real-3DQA Construction Process. Real-3DQA provides fair and rigorous evaluation framework for 3D spatial reasoning in 3D-LLMs. The construction process begins with Filtering 3D-independent Questions, which removes questions that can be correctly answered by both the 3D-LLM model M, and its text-only M counterpart, as well as those answerable by the GPT model without 3D input. The remaining high-quality questions Qrinal are then augmented using GPT, generating spatially consistent variations through viewpoint rotations while preserving the underlying 3D relationships. Finally, expert reviews eliminate redundancy and invalid data, ensuring the highest dataset quality.
correct]. To ensure robustness, we repeat this for all three models and take the union of questions that are independent of 3D inpul: Q3D-fltered = Qa U QB U Qc. The underlying hypothesis for this approach is that if a of questions can be answered correctly by both the original and blind- finetuned models, it means the question set is both less relevant to 3D spatial understanding and can be overfitted by data priors. Thus, we obtain more rigorous remaining question set by removing those 3D-independent and trivial questions: Q' Q Q3D filtered- We refer the reader to Figure 3 to illustrate the procedure up to this stage.
To ensure the final benchmark exclusively assesses true 3D understanding, we further refine Q' by removing questions that a general large language model (i.e., GPT-4o-mini OpenAl (2024) can correctly answer based solely on the textual content of the situation and question without 3D input. Assuming the set of questions, GPT can answer using only textual input as: QUPT {g Q' (g) correct). The final step to obtain the filtered test set is then: Qtiaal Q' QGPT- In Section D.I of Appendix, we present the change in the number of questions at each step of the filtering (Table 7 We also provide illustrative examples of 3D-independent questions that were removed.
3.2 VIEWPOINT ROTATION SCORE
To comprehensively assess the effectiveness of the Real-3DQA test set, we employ a combination of standard evaluation metrics and viewpoint rotation metric specifically designed measure true 3D understanding. First, we adopt the same evaluation metrics used in SQA3D, such as refined exact match (EM_R) RR) Huang et al. (2024b), ensuring comparability with existing benchmarks. For a detailed analysis of their impact on model performance, we refer the readers to Section which presents comprehensive experimental results.
To reduce the influence of superficial textual cues on model performance, we propose the Viewpoint Rotation Score (VRS) assess whether 3D-LLMs genuinely comprehend spatial relationships. The core idea is to generate rephrased test questions by rotating the situation descriptions and corresponding answers from the original seed question while preserving underlying 3D relationships. Figure illustrates this process: in the original scenario, an agent faces a trash can with table behind and asks, "What is on my right?" The correct answer is "whiteboard".. By rotating the agent's 's viewpoint (e.g., 90%, 180", 270%), we create alternative descriptions where the agent faces a different object, yet spatial relationships of scene objects remain unchanged. The ground-tru answer dynamically updates based on the new viewpoint. We employ GPT to generate augmented SQA questions by processing pre-designed templates and scene graphs from Ma et al. (2023) from the original question. We provide prompt template in Figure 9 of the Appendix D
5

---

Published as a conference paper at ICLR 2026
2807
Track Dan
ZFV
Tou Car
beart
[Tebl]
Sttutioo: im tecing trahh oin, with a tible behiid me.
Situatoo: im opening the windows with table on my left.
Shoodion: lan facing a caliii, with
Stuation: om trving ciein the whitte bearts with a trush can on left.
wn boars on left.
right?
right"
dinhttigh??
right?
Figure 4: Viewpoint Rotation Augmentation. Real-3DQA generates viewpoint-augmente SQA instances to enforce spatial reasoning. The left panel shows the original room layout, where an agent asks "What is on my right?" with the correct answer "white board." The right panels illustrate the SQA examples of the original viewpoint and rotated viewpoint (90", 180", 270%). where the agent's s perspective shifts and the correct answers dynamically adjust while preserving spatial consistence.
Quality Control. To ensure the reliability of our augmented dataset, adopt multi- quality control process combining GPT hallucination mitigation and structured expert review. We guide GPT generation with in-context examples, apply manual validation enforce rotation sensitivity and QA validity, and conduct expert reviews with qualification checks and agreement metrics. We also use multiple vision-kanguage-modde
cross-validate the augmented QA pairs, confirming that they faithful to 3D rotations and grounded in genuine spatial reasoning. Full details of this process are provided in Appendix D.2
VRS evaluation metric design. VRS measures the robustness of a model to spatial variations on scale of to 100%, where higher values indicate better performance. During the evaluation, each batch consists of four related questions: the original and its three rotated variants (90%, 180 and 270"). After processing the entire data set, we calculate the percentage of instances in which the model correctly answers at least k questions, where h E {1, 3, 4]. Formally, let N₂. be the number of instances where at least i questions in a batch are answered correctly, and Noa be the total number of instances. The percentage for each given by: Ps: Nues
No. x 100 ,here P& represents the accuracy percentage for answering at least questions correctly. The final VRS metric is computed as the mean of these percentages: VRS = Et=1 Ps.
To achieve high performance in VRS, the 3D-LLM are expected to correctly answer all variations of the same question, demonstrating a true understanding of 3D spatial relationships rather than relying on textual patterns. Intuitively, responses inconsistently across rephrased questions reveal weaknesses in the model's 3D reasoning capabilities. By design, VRS is a stricter evaluation metric than the standard ones mentioned previously, it penalizes models that fail to provide consistent answers across multiple question variants. In addition, taking the mean of Ps ensures that the metric does not disproportionately favor cases where the model succeeds on only a subset of viewpoints.
3D-AWARE FINE-TUNING STRATEGY
4.1 ORIGINAL FINE-TUNING
Let the 3D-QA dataset D consist of triplets (T 3D y ()) D, where Drext represents a textual prompt that includes situational description and a corresponding question, æ3D denotes the associated 3D context (e.g., point clouds), and is the ground truth answer.
(8)
Supervised Fine-tuning. In a typical 3D-LLM Supervised Fine-Tuning (SFT) setting Huang et al. (2024b): Wang et al. (2023); Huang et al. (2023) Chen et al. (2024). we fine-tune the model on both text and 3D data to learn mappings from 3D representations
the correct answers. However, the SFT process doesn't it consider how the model uses the data. Empirically, we observe that models still tend to learn over-rely on textual cues rather than learning from 3D context, what we refer to as a textual shortcut problem.
To highlight this potential textual shortcut issue, we next consider Blind Fine- tuning (BF), where we completely ignore the 3D context æap and train the model only on textual inputs. Formally, we fine-tune the model parameters 9 using the standard next-token cross-entropy loss, but conditioned only on the textual prompt T Using BF, we optimize blind model, which reveals how much of the 3D question-ansee answering task can be solved (or guessed) by language context alone. Furthermore, we leverage the BF model reference for assessing the true benefit of 3D data.
Blind Fine-tuning.

---

Published as a conference paper at ICLR 2026
4.2 3D-AWARE REWEIGHTED FINE-TUNING
Motivated by the observation of SFT and BF models, we propose 3D Reweighted Fine-tuning (3DR- FT). Our goal is explicitly encourage the model to use the 3D context, rather than "shortcutting" through language patterns. We begin with the BF model Ps and upweight samples adaptively for which the BF model finds more difficult guess compared with current model, ensuring that the final model is pushed to 3D dependency.
Reweighting function. We define a reweighting function wy (y, Frea) based the ratio of surprise. It measures how surprised the blind model pe is when it finds out about the ground truth token yj. compared to the current training model ps. More formally, we define
Sy Frext)
log Po Øсj Frext
Wi Wy Tres)
So (y. Fuxx)
log リı Frext)
where Se (3,, is the surprise function Ash 20122) of predicting the last token in y given æ. Higher values of W/ indicate that the BF model places a higher surprise on ground truth token y/ than the current model, indicating the token is harder guess by textual context alone. Naturally, the weights to those tokens in the loss function. We then define the 3DR-FT loss function as:
increase
(TD) ED Σ
wy Tuax) log polys Vсj Frext: E3D)
In Appendix we provide theoretical insights on how this objective can promote the model's 3D-context dependency.
EXPERIMENTS AND ANALYSIS
Our experiments try to answer the following questions: (1) How does benchmark differ from SQA3D in evaluating true 3D understanding? (2) Do existing 3D-LLMs exhibit rotation robustness when evaluated with proposed Viewpoint-Rotation Score? (3) Does ReDeehhee Fine- tuning strategy enhance 3D dependency and improve performance on Real-3DQA?
5.1 EXPERIMENTAL SETUP
We evaluate five representative 3D-LLM models (Hong al. 2023 Huang et al. 2024b Qi
et al. 2025: Huang et al. 2023) across ten reasoning abilities on both the SQA3I and Real-3DQA benchmarks. For each model, we test their released pretrained model weights. To demonstrate the effectiveness of our training strategies, select more advanced models LEO and Chat-Scene as baselines, then apply our proposed training strategy.
5.2 BENCHMARK EPFECTIVENESS
Performance comparison with SQA3D. To sess the increased difficulty of Real-3DQA, we compare model performance before and after the benchmark update. As shown in Tab. 2 the up- dated benchmark leads to a substantial drop in the Exact Match (EM) metric: 40.3 for 3D-LLM Hong (2023), , 41.6 for Chat-3D v2 Huang et al. (2023). ,, 35.1 for LEO Huang et al. (2024b), 37.4 for Chat-Scene Huang et al. (0024)), , and 27.5 for GPT4Scene Qi al. (2025), ). relative the SQA3D benchmark. Refined EM metrics show similar de- clines. These results illustrate that benchmark poses greater challenges and better exposes differences in 3D understanding, as methods with similar SQA3D performance (e.g., 3D-LLM, Chat-3D v2, and LEO) diverge more clearly here.
Table 2: Performance comparison on SQA3D and our Real-3DQA. We see a significant per- formance drop in our new benchmark.
SQA3D
Real-3DQA
3D-LLMs
Venue
EM
EM R
EM
EM R
3D-LLM
NeurlPS 23
47.8
49,6
7.5
10.4
Chat-3D v2
Arxiv 24
45.0
48.1
3.4
9.7
LEO
ICML 24
49.4
52.2
14.3
19.1
Chat-Scene GPT4Seene
NeurTPS 24
54.4
57.2
17.0
22.1
ICLR 26
60.6
63.3
33.1
36.9
To verify the effectiveness of our proposed viewpoint rotation evaluation, we conduct a probing experiment on five 3D-LLMs; results of the correct times under four rotations, the overall VRS, and
7

---

Published as a conference paper at ICLR 2026
Table Rotation Robustness Comparison on Real-3DQA. The table shows the performance of different 3D-LLMs (using refined exact match metric) when tested with varying numbers of correct rotations. All models demonstrate a clear performance degradation as the required number of correct rotations increases, highlighting the challenge of rotation robustness in 3D understanding.
Correct times
VRS
Question Types
3D-LLMs
one
two
three
four
distance
direction counting
existence
3D-LLM
33.2
4.1
1.1
0.1
9.6
5.5
24.6
24.2
44.2
Chat-3D v2
23.2
2.7
0.5
0.0
6.6
4.3
14.6
16.7
21.2
LEO
46.9
8.1
1.6
0.4
14.3
11.8
23.2
20.0
36,5
Chat-Scene
43.3
7.1
1.2
0.1
12.9
10.1
23.7
16.7
40.4
GPT4Scene
55.5
14.3
2.5
0.5
18.2
15.8
25.2
31.7
36.5
the per-type breakdown are summarized in Table To better diagnose which spatial skills are most sensitive to viewpoint changes, we further categorize questions into four types based on the reasoning skill required: Distance (nearest/farthest object), Direction (relative spatial orientation), Counting (enumerating objects a region), and Existence (verifying object presence).
3D-LLMs struggle with viewpoint rotation. All models show steep decline in EM Refined the number of required correct rotations increases. Even GPT4Scene, the best best- perffrmnnnn model, drops from 55.5% one rotation to merely 0.5% four rotations, while the remaining models collapse near zero: LEO (0.4%). 3D-LLM (0),, Chat-Scene (0.1%). and Chat-3D v2 (0.0%). Overall VRS range from 6.6% 18.2%, underscoring the models' inability to maintain consistent spatial reasoning across viewpoints. We hypothesize that this arises because existing 3D-LLMs have never been explicitly evaluated on rotation robustness, leading current designs overlook rotation -ivvarannn feature encoding, cross-view alignment, and fine- fine-grained spatial relationship modeling.
Rotation failure is architecture--anostic. This collapse is universal regardless of whether models adopt point-wise feature encoding (3D-LLM) or object-centric representations (Chat-3D v2, LEO, Chat-Scene, GPT4Scene), indicating systemic limitation rather than a design-specific issue. Inter- estingly, 3D-LLM, despite having the second-lowest VRS (9.6%), achieves the highest existence-type score (44.2%), outperforming GPT4Scene (36.5%) and Chat-3D v2 (21.2%). We attribute this to its point-wise encoding, which preserves richer local geometric cues beneficial for object pres- ence detection, whereas object-centric approaches may lose such fine-grained information through feature aggregation. In contrast, distance and direction questions remain consistently challenging across all architectures, confirming that current 3D-LLMS broadly lack viewpoint-invariant invariant spatial representations.
5.3 How CAN WE IMPROVE 3D-DEPENDENCY IN REAL-3DQA?
Comparison of three training strategies. We compare the effectiveness of all three training strategies in Tab. evaluated with the EM Refined metric. Initially, we find that the Blind FT models yields notably lower scores for both LEO and Chat-Scene on Real-3DQA. Again, we con-
LEO
Chat-Scene
Average Attention Score
firm that Real-3DOA questions are heavily 3D 3D-dependent. However, on our Real-3DQA benchmark. 3D-reweighted fine-tuning demonstrates significant improvements over
0.04
traditional methods. For LEO, achieves 29.3 (vs. 19.1
0.02
for SFT and 13.6 for Blind FT). while for Chat-Scene, it reaches 33.9 (vs. 22.1 and 14.4). These results confirm that our method successfully encourages models to lever- age 3D context for spatial reasoning tasks. To further test generalizability of the finding beyond the SQA3D dataset, we additionally construct another filtered bench- mark, Real-Scant QA, from the ScanQA test set using the same pipeline described Sec. 5.2 On Real-ScanQA, 3DR-FT improves LEO to 13.9 EM Refine (vs. 6.1 for Supervised FT and 5.9 for Blind FT), mirroring the gains observed Real-3DQA and highlighting the benefit of 3w-awe training when questions require genuine 3D evidence.
0.00
SFT
3DR-FT (27.7)
3DR-FT (31.6)
(18.3)
(20.9)
Figure 5: 3D tokens attention score af- ter 3DR-FT.
8

---

Published as a conference paper at ICLR 2026
Table 4: Ablation Study on Training Strategies. Columns group results by model and dataset: LEO on ScanQA/Real-Scann)) (left) and /QaADD/Raa (center), and Chat-Scene on SQA3D/Real- 3DQA (right). 3D-rewhighted fine-tuning (3DR-FT) delivers consistent gains across both datasets and models, with the largest improvements on the 3D-dependent sets Lets-Real-3DQQ and Real- ScanQA-wiin Supervised remains strongest on SQA3D.
LEO
LEO
Chat-Scene
Training Strategy
ScunQA
Real-ScanQA
SQA3D
Real-3DOA
SQA3D
Real-3DQA
Supervised FT
32.3
6.1
52.2
19.1
57.2
22.1
Blind FT
33.0
5.9
50.6
13.6
51.4
14.4
3D-reweighted FT
31.3
13.9
48.2
29.3
48.9
33.9
3D reweighttt finetuning does help 3D-dependency. As shown in Figure we provide analysis of attention scores for 3D tokens, following ZZhngg et al. (2025a), which demonstrates the models' overall dependency on 3D tokens when answering questions. Please refer the appendix for more implementation details. The results demonstrate that after applying 3D-Reweighted Fine-tuning (3DR-FT), models exhibit significantly higher average attention scores to 3D tokens for both 3D- LLMs. These findings align with the performance gains observed on Real-3DQA. The increase in attention scores confirms that our 3DR-FT strategy not only enhances model performance but, more importantly, successfully guides the models to rely more on 3D visual information rather than textual shortcuts when making spatial reasoning.
Analysis on Failure Cases. Table shows a seemingly coumterintuitive result: while 3DR-F FT substantially im-
SQA3D
Real -Dal
Fittered Set
proves performance on the 3D-dependent Real-3DQA test set, the overall performance test set decreases; for example, Chat-Scene drops from 57.2 48.9 after 3DR-FT, At first glance, this seems puzzling-WW why would emphasizing 3D information hurt performance?
the original SQA3D
Blind FT-
SFT
A breakdown of failure cases reveals that this decrease is mainly driven by the 3D-independent portion of SQA3D, as shown in Figure [ After FDR-FT. 591 questions flipped from correct to incorrect, and 441 of them were from the filtered set. Notably, nearly 70% (413/591) of these degraded questions were also answered correctly by the text-only (blind) model, suggesting they were chortcut-solvable when using language priors. Many of these questions fall into reasoning types, such count- ing (92). navigation (86), visibility (94), and spatial rela- tions (93). We hypothesize that these questions can often be guessed from surface language and answer priors, but become harder when answers must be grounded in gen- uine 3D input. Thus, the observed drop does not indicate weaker 3D understanding; rather, it mainly reveals how inflated scores prior benchmarks were driven by lin- guistic shorteuts, further validating the necessity of our debiased dataset filtering and the proposed VRS metric score.
3DR FT
1000 1500 2000
2500
3000
3506
Wrong
Change
Figure 6: Why 3DR-FT reduces SQA3D performance? For Chat-Scene, 591 ques- tions flip from correct to wrong after 3DR-FT; 441 of these come from the Filtered Set (green box with diagonal hatching). Because SQA3D mixes 3D- dependent questions (Real-3DQA) with 3D 3D-independent ones (Filtered Set), em- phasizing 3D evidence via 3DR-FT can hurt the latter set, lowering the overall SQA3D score.
CONCLUSION
This work examines the 3D reasoning capabilities 3D-LLMs, revealing that state-of-the-art models can rely textual shortcuts rather than true 3D spatial understanding on existing SQA3D dataset. To address this, we introduce Real-3DQA, benchmark designed to minimize textual shortcuts and enforce genuine 3D reasoning. Through rigorous filtering and the Viewpoint Rotation Score, benchmark provides a more robust evaluation framework. Additionally, we propose a 3D-reweighted training strategy that enhances models' reliance on 3D context, significantly improving the spatial

---

Published as a conference paper at ICLR 2026
reasoning performance for existing 3D-LLM models. Our findings underscore the need for stronger evaluation benchmarks and training methodologies to advance 3D-LLMs.
FUTURE WORK
For rotation augmentation, we currently use four common angles to generate 3,000 questions. We believe that considering more rotation angles (such as the c'oook direction) could be valuable. However, this would require more complex rules to define these directions and angles, well as precise language descriptions to match them, ultimately necessitating re-annotation and significantly increasing manual effort. Nevertheless, from the perspective of expanding rotation robustness, consider this promising direction worth exploring. Additionally, investigating model architectures and training schemes that are inherently robust to viewpoint rotation remains an open and equally important direction.
Beyond question answering, have observed similar issues related shorteut learning and lack of spatial consistency in 3D grounding and captioning tasks. However, current benchmarks in these areas do not incorporate situated information; all existing datasets adopt an allocentric (scene-centric) perspective rather than egocentric (observer-centric) one. To enable a meaningful evaluation of spatial understanding grounding and captioning. future work will require redefining these tasks with egocentric situation descriptions and constructing new QA caption variants under viewpoint changes. This would involve significant annotation efforts, which we leave as promising future directions broaden the scope of spatial reasoning evaluation and inspire further attention in the community.
ACKNOWLEDGEMENT
We would like to thank Xiongkun Linghu and Jiangyong Huang for generously providing object labeling and detailed scene graph annotations of ScanNet scenes, which were instrumental in the construction of our benchmark. We are grateful to Brandon Smart for carefully proof-reading the manuscript. We also sincerely appreciate Lixiong Chen, Zhenyu Chen, Renrui Zhang, Ian Reid, Guanqi Zhan, Alexei Efros, David Forsyth, Ingmar Posner, and Sijin Chen for their insightful discussions and constructive feedback throughout the development of this work.
ETHICS STATEMENT
This work adheres the ICLR Code of Ethics. In this study, human subjects animal experimen- tation was involved. All datasets used, including SQA3D, ScanQA, were sourced in compliance with relevant usage guidelines, ensuring no violation of privacy. We have taken care avoid any biases or discriminatory outcomes in our research process. No personally identifiable information was used, and no experiments were conducted that could raise privacy or security concerns. We are committed to maintaining transparency and integrity throughout the research process.
10 REPRODUCIBILITT STATEMENT
We have made every effort to ensure that the results presented this paper are reproducible. All code and datasets have been made publicly available in the repository to facilitate replication and verification. We have also provided a full description of dataset construction and reweighted finetuning strategy, to assist others in reproducing our experiments. Additionally, all pretrained models we evaluate, such as 3D-LLM, Chat-3D v2, LEO, Chat-Scene and GPT4Scene, are publicly available, ensuring consistent and reproducible evaluation results. We believe these measures will enable other researchers to reproduce our work and further advance the field.
REFERENCES
Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3d: Neural listeners for fine-gri iin-ngriined 3d object identification real-world scenes. In ECCV, 2020.
10

---

Published as a conference paper at ICLR 2026
Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. Don't just assume; look and answer: Overcoming priors for visual question answering. In CVPR, pp. 4971 4971-4980, 2018.
Robert B Ash. Information theory. Courier Corporation, 2012.
Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question
answering for spatial scene understanding. In CVPR, 2022.
Dave Zhenyu Chen, Ali Gholami, Matthias Nießner, and Angel X. Chang. Scan2cap: Context-aware dense captioning in rgb-d scans. In CVPR, 2020.
Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. LI3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning. In CVPR, pp. 26428-26438, 2024.
Yasaman Etesam, Leon Kochiev, and Angel X. Chang. 3dvqa: Visual question answering for 3d environments. In Conference on Robots and Vision (CRV), 2022.
Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wenhan Xiong. Scene- Ilm: Extending language model for 3d visual understanding and reasoning. arXiv preprint arXiv:2403.1140, 2024.
Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, et al. Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following. arXiv preprint 2:Xi::2309:00615. 2023.
Yining Hong. Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d- Injecting the 3d world into large language models. NeurIPS, 2023.
Haifeng Huang, Zehan Wang, Rongjie Huang, Luping Liu, Xize Cheng, Yang Zhao, Tao Jin, and Zhou Zhao. Chat-3d v2: Bridging 3d scene and large language models with object identifiers. arXiv preprint ar arXiv:2312.08166 2023.
Haifeng Huang, Yilun Chen, Zehan Wang, Rongjie Huang, Runsen Xu, Tai Wang, Luping Liu, Xize Cheng, Yang Zhao, Jiangmiao Pang, et al. Chat-scene: Bridging 3d scene and large language models with object identifiers. NeurlPS, 2024a.
Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang, An embodied generalist agent in 3d world. In ICML, 2024b.
Jiangyong Huang, Baoxiong Jia, Yan Wang, Ziyu Zhu, Xiongkun Linghu, Qing Li, Song- Zhu, and Siyuan Huang. Unveiling the mist over 3d vision-language understanding: Object-centric evaluation with chain-of--anllssi arXiv preprint 2:xXx::2503.222000 2025.
Haoyuan Li, Yanpeng Zhou, Yufei Gao, Tao Tang, Jianhua Han, Yujie Yuan, Dave Zhenyu Chen, Jiawang Bian, Hang Xu, and Xiaodan Liang. Does your 3d encoder really work? when pretrain-sft from 2d vlms meets 3d vlms. arXiv preprini arXi::2506.053/8 2025.
Mingsheng Li, Xin Chen, Chi Zhang, Sijin Chen, Hongyuan Zhu, Fukun Yin, Gang Yu, and Tao Chen. M3dbench: Let's instruct large models with multi-modal 3d prompts. arXiv preprint arXiv:2312.10763. 2023.
Zeju Li, Chao Zhang, Xiaoyan Wang, Ruilong Ren, Yifan Xu, Ruifei Ma, Xiangde Liu, and Rong Wei. 3dmit: 3d multi-modal instruction tuning for scene understanding. 2024 IEEE International Conference Multimedia and Expo Workshops (ICMEW), pp. 1-5. IEEE, 2024.
Xiongkun Linghu, Jiangyong Huang, Xuesong Niu, Xiaojian Shawn Ma, Baoxiong Jia, and Siyuan Huang. Multi-modal situated reasoning in 3d scenes. NeurlPS, 37: 37:144003--40936 2025.
Jie Ma, Pinghui Wang, Dechen Kong, Zewei Wang, Jun Liu, Hongbin Pei, and Junzhou Zhao. Robust visual question answering: Datasets, methods, and future challenges. IEEE TPAMI, 2024a.
11

---

Published as a conference paper at ICLR 2026
Xianzheng Ma, Yash Bhalgat, Brandon Smart, Shuai Chen, Xinghui Li, Jian Ding, Jindong Gu, Dave Zhenyu Chen, Songyou Peng, Jia- Wang Bian, et al. When Ilms step into the 3d world: A survey and meta-analysis of 3d tasks via multi-modal large language models. arXiv preprint arXiv:2405. 10255, 2024b.
Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes. ICLR, 2023.
Yulei Niu, Kaihua Tang, Hanwang Zhang. Zhiwu Lu, Xian-Sheng Hua, and Ji-R Ji-Rong Wen. Counter- factual vqa: A cause-effect look at language bias. In CVPR, pp. 12700-12710, 2021.
OpenAl. Gpt-4o, 2024. URL
Ninglin Ouyang, Qingbao Huang, Pijian Li, Yi Cai, Bin Liu, Ho-fung Leung, and Qing Li. Suppress- ing biased samples for robust vqa. IEEE Transactions on Multimedia, 24:3405-344 33155, 2021.
Zhangyang Qi. Zhixiong Zhang, Ye Fang. Jiaqi Wang, and Hengshuang Zhao. Gpt4scene: Understand 3d scenes from videos with vision-language models. arXiv preprint arXiv:2501.01428, 2025.
Tianwen Qian, Jingjing Chen, Linhai Zhuo, Yang Jiao, and Yu-Gang Jiang. Nuscenes-qa: A multi- modal visual question answering benchmark for autonomous driving scenario. In AAAI, volume 38, pp. 4542 4550, 2024.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang, Squad: 100,000+ questions for machine comprehension of text. EMNLP, 2016.
An Vo, Khai-Nguyen Nguyen, Mohammad Reza Taesiri, Vy Tuong Dang, Anh Totti Nguyen, and Daeyoung Kim. Vision language models are biased. arXiv preprint arXiv:2505.2394 2025.
Zehan Wang, Haifeng Huang, Yang Zhao, Ziang Zhang, and Zhou Zhao. Chat-3d: Data-efficiently tuning large language model for universal dialogue of 3d scenes. arXiv preprint arXiv:2308.0876) 2023.
Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang. and Dahua Lin. Pointllm: Empowering large language models to understand point clouds. ECCV, 2024.
Xu Yan, Zhihao Yuan, Yuhao Du, Yinghong Liao, Yao Guo. Shuguang Cui, and Zhen Li. Comprehen- sive visual question answering on point clouds through compositional scene manipulation. TVCG, 2023.
Jaaning Yang, Xuweiyi Chen, Shengyi Qian, Nikhil Madaan, Madhavan Iyengar, David F Fouhey, and Joyce Chai. Lim-grounder: Open-vocabulary 3d visual grounding with large language model as an agent. ICRA, 2024.
Senqiao Yang. Jiaming Liu, Ray Zhang, Mingjie Pan, Zoey Guo, Xiaoqi Li, Zehui Chen, Peng Gao, Yandong Guo, and Shanghang Zhang. Lidar-Ilm: Exploring the potential of large language models for 3d lidar understanding. arXiv preprint arXiv:2312.
2023.
Shuquan Ye, Dongdong Chen, Songfang Han, and Jing Liao. 3d question answering, 2021.
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun I Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Wei Chang. Yu Qiao, al. Mathverse: Does your multi-modal Il truly see the diagrams in visual math problems? In ECCV, 2024.
Shaolei Zhang, Qingkai Fang, Zhe Yang, and Yang Feng. Llava-mini: Efficient image and video large multimodal models with one vision token. arXiv preprint arXiv:2501.0389) 2025a.
Weichen Zhang, Ruiying Peng, Chen Gao, Jianjie Fang, Xin Zeng, Kaiyuan Li, Ziyou Wang, Jinqiang Cui, Xin Wang, Xinlei Chen, et al. The point, the vision and the text: Does point cloud boost spatial reasoning of large language models? arXiv preprint arXiv:2504.044 0445400 2025b.
Yue Zhang, Zhiyang Xu, Ying Shen, Parisa Kordjamshidi, and Lifu Huang. Spartun3d: Situated spatial understanding of 3d world in large language models. In ICLR, 2025c.
12

---

Published as a conference paper at ICLR 2026
Duo Zheng, Shijia Huang, and Liwei Wang. Video-3d Ilm: Learning position-aware video represen- tation for 3d scene understanding. arXiv preprint arXivv2412.00493, 2024.
Chenming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang, and Xihui Liu, Llava-3d: A simple yet effective pathway to empowering Imms with 3d-awareness, arXiv preprint 2xix:2449.11125. 2024.
13

---

Published as a conference paper at ICLR 2026
A
OUTLINE OF THE APPENDIX
LLM usage(Section
More experiments and analysis
Details of how we calculate 3D token attention score (Section C.2
Ablation on LEO Inference (Table 6
Details of benchmark construction
GPT prompt template for rotation-augmenntalin (Figure
Statistics and examples of filtering 3D-independent questions (Table 7)
Data quality control (Section D.2)
Theoretical insights of 3D-reweighted Fine-tuning (Section B
B LLM USAGE
Large Language Models (LLMs) were used to aid in the writing and polishing of the manuscript. Specifically, used an LLM to assist in refining the language, improving readability, and ensuring clarity in various sections the paper. The model helped with tasks such as sentence rephrasing, grammar checking, and enhancing the overall flow of the text.
important to note that the LLM was not involved in the ideation, research methodology. or experimental design. All research concepts, ideas, and analyses were developed and conducted by the authors. The contributions of the LLM were soiely focused on improving the linguistic quality of the paper, with no involvement in the scientific content or data analysis.
The authors take full responsibility for the content of the manuscript, including any text generated or polished by the LLM. We have ensured that the LLM-generated text adheres to ethical guidelines and does not contribute to plagiarism or scientific misconduct.
C MORE EXPERIMENTS AND ANALYSIS
C.I PERFORMANCE DROP ACROSS QUESTION TYPES.
To analyze model performance differences across question types, we provide a detailed breakdown in Tab. ( and visualization in the radar chart in Fig. 7 First, we observe that all the question types undergo significant performance drop, further confirming that Real-3DQA 's more rigorous nature over SQA3D. Particularly, observe that most 3D-LLMs consistently struggle with spatial relation- ships, shape and state recognition, and reasoning tasks. Two the largest drops in EM Refine appear for the shape and state recognition questions, where five models (3D-LLL.MChha--3) v2/LEO/Chatt Geene/GPT4Scene drop 55. //66///51111. 2..3333. respectively for shape and 0...5/577///999 1/1.8///9444...111 4/52.7/26.2 respectively on state. Moreover, find that reasoning questions already had lower scores in SQA3D, particularly for 3D-LLM at 40.0, LEO at 35.0, and Chat-3D v2 at 50.0, despite GPT4Scene achiev- ing 52.5. The multi-hop 3D reasoning is considered the most challenging problem for all current 3D-LLMs. When evaluated on Real-3DQA, these scores plummet further where all five models (3D-LLM/Chat-3) v2/LEO/Chat- hwwp//Caaa--enGGPPaa.... Scene/GPT4Scene) drop 40. wtt.0/55.4/11.0 2221.4/21.//2 8/25.2 respectively. Finally, we find that even some basic question types, including color and object identification, become substantially harder in Real-3DQA, with even the best-performing GPT4Scene experiencing a 18.6 point drop in color recognition and a 23.0 point drop in object identification.
Together, these findings reinforce that once 3D-independent and guessable questions are filtered out, current 3D-LLMs routinely fail to demonstrate a robust 3D understanding capabilities and highlight the necessity of the Real-3DQA. We hope our new benchmark will draw the community' 's attention to the "3D shorteut" issue and help the development of methods that genuinely comprehend 3D context. This multi-dine dimensional analysis enables more comprehensive comparison of model differences and strengths, highlighting each model's capability preferences and common weaknesses.
14

---

Published as a conference paper at ICLR 2026
3D-LLMs Performance Comparison on Real-3DQA
Spatial Relation
Shape
Number
160
Staté
50
40
30
30
10
Color
Object
Reasoning
LEO
Chat-Scene
Visibility
3D-LLM
Chat-3D
Navigation
GPT4Scene
Figure 7: Comparison of 3D-LLMs capabilities across different question types Real-3DQA. The radar chart visualizes the performance of five models (LEO, Chat-Sceng 3D-LLM, Chat-3D v2, GPT4Scene) distance from the center indicates the model's performance in that category. The area covered by each model's 's polygon reflects its overall capability across all question types. The chart highlights that GPT4Scene demonstrates superior performance in all categories.
ten question categories. Each axis represents specific question type, and the
Rotation Robustness Comparison on Real-3DQA
At Least Correct
At Least 3 Correct
46.9
Score (%)
20
7.1
0.5
3DMLLM
Chat:30 v2
LEO
Chat-Scene
GPT4Scene
Model
Figure Rotation robustness comparison. All models show a consistently increasing decline in EM Refined the viewpoint changes from to four, underscoring the models' difficulty in maintaining consistency across multiple viewpoints.
15

---

Published as a conference paper at ICLR 2026
Table Model ability across different question types. We compare model performance on both the original SQA3D benchmark and our more challenging Real-3DQA. On Real-3DQA, we find most 3D-LLMs have consistent deficiencies in spatial relationships, shape recognition and reasoning tasks. We report refined exact match for detailed question types.
3D-LLMs
EM EM
color number spatial relation shape object visibility navigation reasoning other
Refoned EM SQ43D
SD-LLM
47.8
49.6
69.2
45.7
$2.4
35.7
59.2
64.7
42.4
63.5
41.5
40.0
39.5
Chat-3D v2
45.0
48.1
36.1
57.0
56.0
43.2
67.8
71.5
60.4
71.1
45.7
50.0
88.4
49.6
52.2
73.1
58.1
52.8
59.9
69.9
52.8
66.4
41.5
35.0
76.7
Chat Sceme
54.7
57.4
80.8
58.1
57.2
46.8
66.5
76.0
59.2
70.0
48.1
40.0
81.4
GPT4Soene
60.6
63,3
76.9
70.8
65.8
52.6
73.7
83.7
68.8
73.3
43.3
52.5
79.1
Refined EM Real-3DQA
3D-LLM Chat-3D v2
7.5
10.4
0.0
10.1
9.8
9.3
3.5
8.2
10.6
4.0
18.2
0.0
20.0
20.0
5.1
13.5
13.7
1.9
17.1
12.2
4.6
0.0
Chat-Scone GPT4Scene
17.0
22.1
60.0
28.3
15.2
22.3
19.3
23.3
33.0
15.8
20.6
18.
20.0
33.1
36.9
40.0
$2.2
34.7
31.1
40.4
57.5
45.8
30.3
29.6
27.3
60.0
Table 6: Where does the 3D prior come from in LEO Huang et al. (2024b) on SQA3D? We conduct this ablation during only the inference time, using checkpoints provided by the authors. Results show that 3D information and questions are crucial for performance, while situation descriptions have minimal impact. We report both exact match (EM) and refined exact match Refined) metrics.
Situation
Question
3D
EM (%)
(%)
49.4
52.2
shuffled
44.2
46.7
V
X
32.4
43.3
X
49.3
51.7
V
X
0.2
10.4
X
x
18.6
30.1
C.2 DETAILS OF 3D TOKEN ATTENTION SCORE CALCULATION.
We randomly selected 100 questions from Real-3DQA, and used both original and 3D- 3D-Rewiighted fine-tuned versions of LEO and Chat-Scene models for inference. Specifically, we followed the approach in Zhang et al. (2025a), which quantifies models' overall dependency 3D point cloud tokens when answering questions. By calculating the average attention scores from 3D point cloud tokens answer tokens, responding to questions. To ensure statistical significance, we performed three separate sampling runs, collected three sets of data, and calculated standard deviation error bars.
can determine how much the model relies 3D information when
C.3 ABLATION ON LEO INFERENCE
Table presents ablation study on LEO to evaluate the impact of different input components on inference performance using the SQA3D benchmark. The study examines how 3D information, situational descriptions, and question relevance influence exact match (EM) and refined exact match (EM_R) scores. shows by removing 3D input alone, the model can still obtain 32.4% on EM and 43.3% EM.R, indicating that part of the model prediction is not rely 3D information. On the other hand, by removing situation description, the model's 's EM% and EM R% metrics barely dropped.
D DETAILS OF BENCHMARK CONSTRUCTION
D.1 STATISTICS AND EXAMPLES ON FILTERING 3D-INDEPENDEN QUESTIONS
Table 7 shows filtering statistics from SQA3D using our proposed methods in Section 3.1 of the main paper. The table compares the number of filtered and remaining questions across different filtering
16

---

Published as a conference paper at ICLR 2026
methods. Specifically, GPT-4o-mini is used for GPT-based filtering, while LEO, Chat-Scene, and 3D-LLM are employed for SFT and Blind Fine-tuned model comparison.
We find that GPT-based filtering removed only 1 12 questions, leaving 3407 out of the original 3519. This indicates that GPT filtering alone is relatively conservative, missing many questions that might be easy-to-guess non-3D-dependent. In contrast, model comparison approach filters substantially more questions.
Table 7: Statisties of filtered questions for each filtering step. The table reports the number of questions filtered and those remaining for the original test set, GPTT-based filtering, and model-based comparisons filtering across LEO, Chat-Scene, and 3D-LLM.
Filter Type
Original Test
GPT
Model Comparison
LEO
Chat-Scene
3D-LLM
Filtered
112
1197
459
232
Remaining
3519
3407
2176
1717
1485
We also show three representative examples of filtered questions to highlights the effectiveness of our filtering approach through model comparison. These filtered questions are so directional that the answer is easily guessable.
Question: What my left that gives me natural light in the room?
Answer: window
Question: What is to my right that can use washable marker to write with? Answer: whiteboard
Question: What instrument in front of you is ebony and ivory?
Answer: piano
D.2 QUALITY CONTROL
To ensure the reliability of our rotation-augne augmented dataset, we adopt a multi-stage quality control pipeline that combines GPT prompting strategies with structured expert review. This process ad- dresses both the risk of hallucination in GPT outputs and the need for consistent, rotation-sensitive annotations.
D.2.1 HALLUCINATION MITIGATION
A key concern when leveraging GPT for question augmentation is the potential issue of hallucination. To mitigate this, we employ a two-pronged strategy.
instruction-folloiing enhancement via ICL examples. We designed structured in-context learning (ICL) examples (Figure 9) to guide GPT's generation. These examples reinforced the desired prrduucin faithful situation texts that simulate specific 3D rotations-ann helped GPT adhere to the intended transformation format.
Manual verification with dual criteria. We further applied two-stage manual validation process. Specifically, we retained only those QA pairs that satisfied both: (i) the situation text is rotation- sensitive and accurately reflects the intended 3D rotation; (ii) the original question remains meaningful after rotation, with the updated answer remaining correct in the rotated context. Examples failing either criterion were discarded corrected. These measures minimize hallucination and ensure that the augmented QA pairs are semantically coherent and grounded in 3D reasoning.
D.2.2 EXPERT REVIEW
Beyond hallucination control, we implemented a structured expert review protocol to validate annotation quality.
Initial filtering of GPT outputs. Despite careful prompting, about 5% of GPT outputs proved invalid (e.g., incorrect answers after rotation inconsistent situation descriptions, Figure T0p. To detect
17

---

Published as a conference paper at ICLR 2026
system pronpt ('role' 'systen', 'ccontnt'' "ourr job augment testing by rotations for 3D VLM evaluation. The rotation is 98, 188, 278 degree along the gravity-axis. You are given ego-centric scene description far all objects in room with distances, orientations, heights from the floor, attributes, and relationships between other objects. You answer. 1. Read the scene description and understand the spatial relationships between the observer and the objects in the room.
given situation, question,
tasks:
change question. answer based the situation. Make sure the new answer is still valid based on the scene description.
keep question exactly
3. Generate the
4a. Output the new situation, question (should be the same as the original), and the new answer, Try find situation
DIFFERENT from the original answer but still VALID and explicitly mentioned in the scene description in the correct direction after rotation.
4c. If you cannot find valid answer under current rotation, output "skip" and do not generate any new situation.
5. If the situation cannot rotated meaningfully (no explicit direction), please output directly "skip" and not generate any situation question.
6. You need to keep the style of the original situation. Keep the length of the answer same the original. For most answers, a single word or two is enough. Uncapitalized unless it is an acronyn or proper noun
TV. 7. In scene description, the object named with type and number, e.g., chair-13. You only refer to the objects by names in situation and question. The number is to distinguish between objects. Do not leak the number.
[Example Input]
Scene:
bed-1: dist=êcm; orient=front; obj height=43cn;; colorswhite; shape-rectangular:;
naterial=wood; usage=sleeping; texture=smooth; structureaframe with mattress; state=made; the top of floor-2;
window-2: dist=103cm; orient-right; obj jbi_height=15Amm; color-white; material-glass;
usage=lighting; texture=smooth; structure=frame with glass; state=clean; embedded into wall-3;
door-3: dist-200cm; orient-left; obj_ heightt2etee:m color-white; material-wood;
usage-entrance; texture=smooth; structure-frane with handle; state=closed; enbedded into wall-3; Situation: am standing facing the bed in the bedroom. Question: What can see on my left?
Answer: door
Rotation: 270
[Example Output]
1. Situation: standing facing the window in the bedroom.
2. Question: What can see on my left?
3. Answer: bed
[Explanation] If rotate 270 degree (-rotate 98 degree to the right), the bed will be left. The answer should be bed.""">
for angle in [90, 180, 270]: messages [oystem_prompt] nessages.append((
'role': :'ser'',
'content' secescecetoocccceee/sstttttttt (s)uuusss: {q} (g)\nnnser:: (a))nRotation: (angle)' )) output (uery(messages)
Figure 9: Prompt Template for Rotation-Augmenteer Question Generation. We design this template to guide GPT-4o-mini in creating perspective-rotated variants of the original questions. The template instructs the model to preserve the original question's intent while altering the spatial reference frame according to specified viewpoint changes.
18

---

Published as a conference paper at ICLR 2026
doof dod 13 walfo trash can
Situation:
am leaning on an end of the table and there is trash can behind me.
windou-9
Question: What color are the window frames to my right? Answer: light brown
chair-6 their-7 table-p
ceiling-13 15
whitebolskd11e
beckpedk-22 -020 chair-
window-19
person
front
right
tabnraa017
whitWalhird-18
left
Figure Example view-specific question that cannot be rotated. This question demonstrates case where rotation is invalid because it references objects (window frames) that only exist in the current viewpoint (to my right). When the perspective rotated, these referenced objects are no longer visible or identifiable, rendering the question meaningless in the new viewpoint.
these, visualized the observer's position and orientation, determined object quadrants (front, back, left, right) from bounding box centers, and checked whether GPT's rotated situations and answers aligned with the new viewpoint. Inaccurate answers were corrected, while fundamentally invalid situations were discarded. Ultimately, we retained 750 question-ani answer pairs valid across all four viewpoints, yielding 750 4 3000 high-quality pairs.
Structured protocol. We further enforced three- prree-phase expert review protocol: Pre-review: annotators received explicit evaluation criteria and passed qualification test before large-scale scale review. During review: experts iteratively tracked errors in a shared log (e.g., mismatched rotations, mishandled states, object counting errors). Random spot checks and annotation speed monitoring ensured consistency. Post-review: annotation reliability was measured with agreement metrics, yielding Cohen's Kappa of 92% (strong inter-rater agreement) and self-consistency 97%.
Together, these steps guarantee that the final augmented dataset is rotation-sensitive, reliable, and well-suited for benchmarking genuine 3D reasoning.
E THEORETICAL INSIGHTS OF 3D-REWEIGHTED FINE-TUNING
To quantify the model's 3D-context dependency, we introduce the conditional-nndepedeeet gap between the ground truth token y, and the 3D input Ø3D,
6, po(y, Mej. Brxx FzD) Po(yily< Frex))
pe Mz Fxex))
When 6, = 0, the token yj is conditionally independent of Z&D given the text input Texti in other words, the 3D context is not altering the distribution of yy- This precisely what we aim to avoid in the Real-3DQA benchmark, since it implies that the model is ignoring the 3D information.
Below, we link the conditional-indeeeddeee gap to our loss function. For simplicity, we denote
Po We.. Drea) = 85, and thus
Fiext, Pap) == s;(1 + 8,).
(4)
Inserting Eq. (1) in main paper and Eq. into Eq. (2) in main paper leads
19

---

Published as a conference paper at ICLR 2026
FTDR-FT(0)
ED
Σ wy wy(y Traul (log3, + log + 6;))
= DD
log Polys
yes Biext) +
j=1
Perplexity without 3D input
T
E
Wj wy(y Frext (rex) log(1 + +5,)
1=1
Weighted conditional inddpeieaal--deeeeeeenee gap
Here, the first term corresponds the BF model's perplexity the ground truth. has zero derivative w.r.t. 0 and, thus, does not affect & during training.
Essentially, the 3DR-FT loss function increases log(1 + ōy) during training and leads non-zero 6, > whenever yj is correctly dependent on the 3D context. Consequently, 3DR-FT pushes the model rely on 3D information for improved predictions, mitigating the textual shorteut problem observed in standard fine-tuning.
20


Reviews:
[{"review_id": "4qjoRMMGhy", "reviewer": "Reviewer_L57o", "summary": "This paper investigates whether current 3D large language models (3D-LLMs) truly understand 3D spatial relationships, rather than exploiting language priors. The authors show that text-only finetuned LLMs can perform competitively on existing 3D QA benchmarks, suggesting strong linguistic bias. To address this, they propose Real-3DQA, a refined benchmark that filters out 3D-independent questions and introduces viewpoint-rotation consistency evaluation. Additionally, they introduce a simple fine-tuning strategy, 3D-Reweighted Fine-tuning (3DR-FT), which assigns higher weights to questions that are less answerable from text alone. Experiments demonstrate significant performance drops of existing 3D-LLMs on Real-3DQA, confirming the issue, and modest gains from 3DR-FT.", "questions": "See Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The finding that text-only models perform nearly as well as 3D-LLMs on standard benchmarks is valuable to the community.\n\n- Real-3DQA improves evaluation fairness by filtering 3D-independent samples and introducing rotation consistency, which is sound.\n\n- The authors evaluate multiple existing 3D-LLMs and provide both quantitative and qualitative analyses that convincingly demonstrate the identified problem.", "weaknesses": "Limited novelty in methodology: The paper diagnoses dataset bias effectively but does not propose new modeling architectures or mechanisms to fundamentally improve 3D reasoning.\n1. The Real-3DQA benchmark largely refines existing datasets via filtering and simple viewpoint augmentations; while useful, it feels more like an engineering refinement than a conceptual leap.\n2. The 3DR-FT method adds a weighting term to the loss based on text–3D prediction discrepancy; the idea is intuitive but technically minor.\n\nThe work’s value is primarily diagnostic rather than methodological.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C7XJfPKl6s", "reviewer": "Reviewer_SPo3", "summary": "The paper asks whether current 3D-LLMs truly understand 3D spatial relationships. The authors first show that a **text-only “blind” fine-tuned LLM** can match or surpass several 3D-LLMs on SQA3D, implying the benchmark allows shortcutting via linguistic priors rather than genuine 3D reasoning. They therefore construct Real-3DQA, filtering out questions that both a vision-conditioned model and its blind counterpart (and then a general LLM) can answer correctly without 3D input, yielding a set more dependent on 3D evidence. They further introduce a rotation-robustness metric, the Viewpoint Rotation Score (VRS), which evaluates consistency across rephrased, viewpoint-rotated variants of the same question. Experiments reveal sharp drops in accuracy when requiring correctness across all rotated views, highlighting poor rotation consistency in recent 3D-LLMs. Finally, the paper proposes 3D-aware Reweighted Fine-Tuning (3DR-FT), which uses a blind-model reference to upweight samples that are hard to guess from text alone, substantially improving performance on Real-3DQA (and a similarly filtered Real-ScanQA) while encouraging reliance on 3D cues.", "questions": "1. How sensitive is Real-3DQA to the choice of “blind” baseline? Would using different text-only LLMs (e.g., smaller or instruction-tuned ones) change which questions are filtered out?\n2. Could the authors validate that the GPT-generated rotated descriptions correspond to actual geometric viewpoint shifts rather than linguistic paraphrases? A small human-verified subset or rendered 3D examples would help.\n3. What are the main failure modes of 3DR-FT—does it hurt generalization to unseen object types or tasks beyond QA?\n4. The four fixed rotation angles cover only azimuth changes. Would adding elevation or roll perturbations further reduce accuracy?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The paper is original in framing “real” 3D understanding via the Real-3DQA pipeline and the rotation-consistency metric (VRS), moving beyond shortcut-prone benchmarks. Methodologically it’s solid: the blind-vs-vision contrast, rotated rephrasings, and multi-stage QC make the evidence credible, and 3DR-FT is a clear, effective objective. The writing and figures communicate the pipeline and metrics cleanly. Empirically, the work exposes meaningful brittleness in 3D-LLMs and shows a practical path (3DR-FT) to increase true 3D reliance—making the contribution significant.", "weaknesses": "Rotation robustness is evaluated using GPT-generated viewpoint texts. Although quality control is thorough, this approach still relies on linguistic rather than geometric variation and limits the test’s realism. Using scene-graph or pose-based rotations with automatic answer recomputation would more directly assess spatial consistency. Coverage is limited to four fixed views; expanding to denser yaw and pitch/roll, plus reporting uncertainty would strengthen claims. The Real-3DQA set is much smaller post-filtering, so testing robustness to alternative filtering unions would address potential distributional shifts. 3DR-FT is demonstrated on two backbones and trades off SQA3D performance; broader ablations and a multi-objective/curriculum variant could maintain gains on Real-3DQA without degrading shortcut-heavy sets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CFwsjgyTsA", "reviewer": "Reviewer_vJtJ", "summary": "This paper presents three main contributions to the development and evaluation of 3D large language models (3D-LLMs). 1) it introduces Real-3DQA, a benchmark that filters out questions answerable through textual shortcuts, thereby providing a more rigorous assessment of true 3D reasoning abilities. 2) it proposes the Viewpoint Rotation Score (VRS), a metric that evaluates a model’s spatial consistency under viewpoint changes, reducing the influence of superficial linguistic cues. 3) it develops a 3D-aware Reweighted Fine-Tuning (3DR-FT) method that re-emphasizes 3D-dependent information to enhance spatial understanding. Extensive experiments show that existing 3D-LLMs still exhibit limited real 3D reasoning capabilities on Real-3DQA, while the proposed 3DR-FT significantly improves their 3D comprehension and robustness.", "questions": "Minor typo: line 226 \"Mx. The ...\" -> \"Mx, the ...\"", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is clearly written and easy to follow, with a well-structured presentation of ideas. The authors’ approach to analyzing the limitations of current 3D-LLMs is insightful and well-motivated, making the paper engaging to read.\n- It provides a comprehensive and well-organized overview of existing 3D-LLMs and their evaluation benchmarks.\n- The proposed Real-3DQA dataset and VRS metric are meaningful and valuable additions to the 3D-LLM research community. The comparative evaluation of existing 3D-LLMs on both the original and filtered benchmarks is thorough and revealing.\n- The ablation studies on the proposed 3DR-FT method are well-designed and conducted across multiple models and datasets. The results are consistent and convincingly demonstrate the effectiveness of the proposed fine-tuning strategy and dataset.", "weaknesses": "- The proposed 3D-aware Reweighted Fine-Tuning (3DR-FT) method improves performance on the Real-3DQA benchmark but results in degraded performance on the original datasets. This suggests that the approach may either reduce the model’s generalization to language-prior-heavy questions or bias it toward producing answers that deviate from the dominant linguistic mode when 3D cues are absent. However, in real-world scenarios, many questions that 3D-LLMs encounter are likely to rely heavily on linguistic priors. Therefore, a more detailed discussion on how to balance 3D dependence and language adaptability would strengthen the paper.\n- The overall dataset size of Real-3DQA is relatively small, and the variation in scene types, spatial relations, and question templates appears limited. This may restrict the benchmark’s ability to comprehensively evaluate diverse aspects of 3D reasoning and generalization across different environments. The authors are suggested to discuss how to further improve the 3D-LLM evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X7B5mtGYNI", "reviewer": "Reviewer_dfML", "summary": "This paper finds that in the current benchmarks for 3D-aware reasoning, as SQA3D, there are many easy-to-guess questions or questions that do not require any 3D input and information. Therefore, the paper introduces Real-3DQA, which is a more rigorous benchmark that filter out the unproper questions for 3D understanding. Furthermore, the paper proposes a 3D-reweighted training objective to guide the model to focus on 3D features when performing 3D reasoning. Experimental results show that the previous high scores in 3D understanding benchmarks largely do not reflect the true 3D reasoning capability of the current 3D-LLMs. The performance on the proposed more rigorous benchmark can more truthfully reflect the real 3D understanding and reasoning capability of 3D-LLMs.", "questions": "-- For the viewpoint rotation score, how are the questions for creating the viewpoint variants getting chosen? From my understanding, it may require heavy manual work, as first we need to select questions that can have different answers when the viewpoints vary. Second, we need to manually label what the answers are for each directions. This is why I think the viewpoint rotation score part feels unnecessary, as it requires manual labor but is not quite justified to be useful.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "++ The paper spotted an interesting yet important issue in the current evaluation of 3D-LLMs, which can raise the awareness of the community on evaluating the true 3D understanding capability of 3D-LLMs.\n\n++ The way of filtering out the 3D-irrelavant questions by comparing the answers of with and without 3D input is reasonable and effective.\n\n++ The writing of this paper is mostly clear. The current issue is clearly stated, followed by the reasonable solutions to build a more reasonable benchmark, and finally shows the performance drop of current method on the more 3D-aware benchmark. Therefore, the overall paper flow is smooth, except for the viewpoint rotation score section which I have some confusions stated below.", "weaknesses": "-- Basically, 3D-LLMs are considered to be general purpose AI assistants. Therefore, in real applications, not all questions have to be related to 3D reasoning. In this case, the performance drop on the questions that can be shortcut-solvable by language priors becomes a real issue, because it means that the general usability of the system will get harmed, as there may only be a portion of tasks that truly require 3D reasoning and understanding. Therefore, the proposed 3DR-FT method has flaws that fails to preserve the original reasoning capabilities of LLMs.\n\n-- I do not quite get the meaning and motivation to design such a viewpoint rotation score in Section 3.2. Basically, I think from Section 3.1, the 3D-independent questions have already been filtered out when comparing the answers. Therefore, there seems to be no need to put the effort to create questions that rotate the viewpoints, as creating these questions for all the four viewpoints may need some manual labor.\n\n-- Typo: Line 461: \"3D-RFT\" should be \"3DR-FT\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)