3D Question Answering via only 2D Vision-Language Models
Authors: FENGYUN WANG
Sicheng Yu
Jiawei Wu
Jinhui Tang
Hanwang Zhang
Qianru Sun
Venue: ICML 2025 poster (2025)
Keywords: 3D Question Answering
2D Vision-Language Models
Primary area: applications->computer_vision

Abstract: Large vision-language models (LVLMs) have significantly advanced numerous fields. In this work, we explore how to harness their potential to address 3D scene understanding tasks, using 3D question answering (3D-QA) as a representative example. Due to the limited training data in 3D, we do not train LVLMs but infer in a zero-shot manner. Specifically, we sample 2D views from a 3D point cloud and feed them into 2D models to answer a given question. When the 2D model is chosen, e.g., LLAVA-OV, the quality of sampled views matters the most. We propose cdViews, a novel approach to automatically selecting critical and diverse Views for 3D-QA. cdViews consists of two key components: viewSelector prioritizing critical views based on their potential to provide answer-specific information, and viewNMS enhancing diversity by removing redundant views based on spatial overlap. We evaluate cdViews on the widely-used ScanQA and SQA benchmarks, demonstrating that it achieves state-of-the-art performance in 3D-QA while relying solely on 2D models without fine-tuning. These findings support our belief that 2D LVLMs are currently the most effective alternative (of the resource-intensive 3D LVLMs) for addressing 3D tasks.

3D Question Answering via only 2D Vision-Language Models
Fengyun Wang' Sicheng Yu² ? Jiawei Wu
Jinhui Tang*
Hanwang Zhang'
Qianru Sun
Abstract
<3D Scene>
Large vision-language models (LVLMs)) have sig- nificantly advanced numerous fields. In this work, we explore how to harness their potential to ad- dress 3D scene understanding tasks, using 3D question answering (3D-QA) as a representative example. Due to the limited training data in 3D, we do not train LVLMs but infer in zero-shot manner. Specifically, we sample 2D views from a 3D point cloud and feed them into 2D models answer given question. When the 2D) model is chosen, e.g.. LLAVA-OV, the quality of sampled views matters the most. We propose cdVies ews, a novel approach to automatically selecting critical and diverse Views for 3D-QA. cdViews con- sists of two key components: viewSelector prioritizing critical views based on their poten- tial to provide answer-specifie information, and viewNMS enhancing diversity by removing re- dundant views based on spatial overlap. We eval- uate cdViews on the widely-used ScanQA and SQA benchmarks, demonstrating that it achieves state-of-the performance in 3D-QA while re- lying solely 2D models without fine-tuning. These findings support our belief that 2D LVLMs are currently the most effective alternative (of the resource-intensive 3D LVLMs) for address- ing 3D tasks. The code is available at https: //git thit.ub. com/ fwwn///weeeeee//////mom cdViews.
<2D Views>
Tokenization
al. 3D-based method
a2. Hybrid method Daaoedly
(noo
<Question
What is the black
a3. Hybrid method 2
a4, 2D- method aligell)
couch facing?
aligned)
point token
text token
image token
point token reconstructed from multi-view tokens
Spatial information provided by: D point clouds
2D views
Illustration of Feature Alignment Issue
EMgl
BridgoQA
DDnFormenr3
LLAVA-OV
NaviLUM
3D-VLP
3D-based method
3D-VISTA
PQLD
Hybrid methed
Hyborid method 2
21>-hased method
CIDEr
(b) Performance on the test set (with objects) of ScanQA
Figure Comparison of 3D Question Answering meth- ods. (a): al for 3D-based methods; a2 and a3 for hybrid (2D+3D) methods. All of these methods require computa- tionally intensive 3ID-language alignment using point cloud data for spatial reasoning. a4 is our method that leverages pre- rre-trained LVLMs operating solely on 2D views. The well- aligned features between 2D visual features and language in 2D LVLMs enable zero-shot 3D-QA. (b): Model compari- son the test set (with objects) of ScanQA. The upper-right corner indicates the best performance. The circle rep- resents the size of training data required for aligning 3D and language. The "X" denotes zero-shot 3D-QA using 2D model LLAVA-OV (Li et al., 2024a). We respectively use @ uniform sampling, @ image retrieval, and our cdViews, to select views as input LLAVA-OV.
1. Introduction
The advancement of large vision-language models (LVLMs) has transformed the vision- lssoun-anguuae domain by jointly pro- cessing huge sets vision and text training data, leading to
"Equal contribution NNayynng Technological University, Sin- gapore 'Singapore Management University, Singapore 'National University of Singapore, Singapore "Nanjing University of Science Technology, Nanjing, China. Correspondence to: Qianru Sun <qianrusun smu.edu.sg>.
Proceedings of the 4dnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
significant breakthroughs addressing 2D visual question

---

3D Question Answering via only 2D Vision-Language Models
answering (2D-VQA) (Shao al., 2023: Guo et al., 2023; Lu al., 2023). However, extending these capabilities to 3D question answering (3D-QA) has unique challenges. Unlike 2D tasks, which benefit from abundant paired training data, the 3D domain lacks large-scale datasets to learn the align- ment between 3D (such as point clouds) and language (such as text descriptions of 3D scenes). Existing 3D-Jangaage models still fall short of serving as robust counterparts to the widely used 2D-language models such as LLaMA-3 (Dubey et al., 2024). Therefore, current 3D-QA methods often have to train from scratch on small-scale 3D datasets, resulting in poor model performance. In contrast, hybrid approaches leverage additional 2D information. One solution (Hong et al., 2023) is to reconstruct 3D features from the features of multiple 2D views (Figure (a2)). but its performance is poor due to the technical challenge of 3D reconstruction. Another solution (Mo & Liu, 2024) is to combine 2D and 3D features as input into the model (Figure 1 (a3)). 2D fea- tures extracted from LVLMs are already well-aligned with language, but further alignment with 3D features requires careful model design and advanced training techniques. Fig- ure (bb) shows that hybrid methods also require extensive amounts of training data (indicated by the large circle area),
<uestion>:
What is the black couch facing?
<Answer> Coffee table
Uniform Sampling ignores question context
Image Retrieval overlooks answer-related information
Ours "the black couch facing coffee lable" is included
Figure 2: Comparison of view selection methods.
(cdViews) and then use them to perform LVLMs-based 3D-QA in a zero-shot manner. cdViews is designed on two key principles. (1) Prioritize Critical Views: We aim for views that contain information crucial for an- swering questions, rather than merely finding views that match question texts. Thus, we develop a lightweight viewSelector module that prioritizes views most likely to contain answer-related information. To train this module, we design a viewAnnotator that automatically gener- ates training data in two steps. vi ewwnnotator firstly converts question-answer pairs into descriptive captions. It then leverages a pre- trained LVLM to identify the most in- formative views that match these captions. (2) Enhance View Diversity: The aim is to improve spatial diversity and minimize redundancy for the selected views. To this end, we develop view Non-Maximum Suppression method dubbed as vi viewNMS. This method uses camera parameters, including position and orientation, to filter out overlapping views while preserving spatial views as diverse as possi- ble. When viewSelectoo and viewNMS are ready, they will be plugged into pre-trained 2D LVLM for zero-shot 3D-QA in the inference stage.
which are not always available.
In this paper, we take completely different approach by avoiding direct alignment between 3D and language. In- stead, we rely solcly on 2D views and pre- pre-trained LVLMs for understanding 3D scenes. For implementation, we first select limited number of 2D views, and then take them as the only visual input to LVLMs to answer the input question.
During our preliminary trials, we identified several chal- lenges. First, all LVLMs have a token limit, restricting the number of 2D views they can process at once. This constraint makes it crucial to carefully select the most in- formative views. Second, given a fixed number of views, the quality of the selected views plays a critical role. Ex- isting methods for view selection fall into two categories: uniform sampling, which randomly selects views, and im- age retrieval, which selects views based on question-based based retrieval (Li et al., 2022). However, both approaches have significant limitations, either being inefficient or failing to capture critical views. Specifically, as shown in Figure 2, image retrieval outperforms uniform sampling but has two major limitations. First, it prioritizes question-related views over truly essential ones for answering the question. For example, when asked "What is the black couch facing?", the model retrieves images of the "couch" but overlooks the "coffee table", which is the answer-related object but in the opposite view of "couch". Second, it often selects redundant or overlapping views, causing inefficiency.
We evaluate the proposed cdViews on two widely used benchmarks of 3D-QA: ScanQA (Azuma et al., 2022) and SQA (Ma et al., 2022). Our experimental results demon- strate that cdViews 's view selection significantly outper- forms conventional approaches such as uniform sampling and image-text retrieval. Notably, edViews achieves supe- rior performance compared to models using 3D or hybrid input data. In summary, our contributions are three-fold. (1) We explore the use of 2D-only LVLM to address 3D- QA in a zero-shot manner, analyzing various view selec- tion methods. (2) We introduce edViews that integrates a viewSelector with a viewNMS to capture critical and diverse views. We design a vi viewAnnotator to generate
To tackle the challenges, we introduce a new frame- work cdViews to select critical and diverse Views
2

---

3D Question Answering via only 2D Vision-Language Models
training data for riewSelectoo automatically. (3) Our experiment results demonstrate that cdViews achieves state-of-the-art performance on two 3D-QA benchmarks, even surpassing the 3D or hybrid models.
lack of an efficient solution in prior works. To tackle this, we propose simple yet effective strategy for selecting criti- cal and diverse views (i.e., cdViews), thereby enhancing the utility of readily-trained 2D LVLMs for 3D-QA.
Hybrid Methods. Hybrid methods (Huang et al., 2024; Mo & Liu, 2024; Huang et al., 2023; Hong et al., 2023;
2. Related Works
Man et al., 2024b; Fu et al., 2024) leverage pre-trained 2D LVLMs to address 3D vision-language tasks in two main ways. The first approach involves mapping multi-view 2D image features (which are well-aligned with language due to 2D LVLMs) into the 3D feature space (Zhu et al., 2025; Hong et al., 2023). These mapped features can either replace original 3D features (Hong et al., 2023) or serve as comple- mentary inputs to enhance the alignment between language and hybrid (2D+3D) features (Zhu et al., 2025). The sec- ond approach processes 2D images and 3D point clouds as parallel inputs (Mo & Liu, 2024), using complementary strengths: 2D views provide fine-grained semantic details, while 3D point clouds capture spatial awareness. Although these methods improve 3D-QA performance, they rely on explicit 31) reconstruction, needing additional models and causing more processing steps. In contrast, our method uses 2D views and feeds them into unified 2D LVLM, which makes simpler pipeline.
Existing approaches to 3D-QU QA can be categorized into three folds based on the format of visual inputs: 3D-based, 2D- based, and hybrid (combining 3D and 2D).
3D-based Methods The 3D-based methods (Man et al., 2024a) use 3D point clouds as visual input, allowing direct processing of point cloud data to understand 3D environ- ments. However, these methods face two challenges. First, the scarcity of 3D-language training data limits its scala- bility. Efforts such as 3D-VLP (Yang et al., 2024) attempt to mitigate this issue by leveraging large-scale symthetic datasets, and recent works (Zhang et al., 2024; Jin et al., 2023b; Hong et al., 2023; Zhu et al.. 2023; Chen et al., 2024b) aim to unify multiple 3D tasks, such as caption- ing, question answering, and grounding, under a single framework. Second, using an entire 3D scene as input in- troduces unnecessary information for QA, distracting the model and reducing efficiency. To address this, methods such as SIG3D (Man et al., 2024a) incorporate situational awareness to focus on only relevant 3D regions guided by the language prompts (e.g., the input situation). Overall, 3D- based methods have constraints due to the lack of large-scale 3D language pretraining data. The resulting 3D-lanuuage alignment in the feature space is thus suboptimal. Besides, using entire scenes input to answer local questions is costly and inefficient.
Different from these hybrid methods, our approach relies solely on 2D views as input, without the need for map- ping between 3D and 2D. Our technical contribution is an efficient view selection strategy, edViews. Among the hybrid methods, the work most closely related to ours is BridgeQA (Mo & Liu, 2024), which selects views by first retrieving the top-1 question-related view and then combining it with 3D point clouds as input for a hybrid model. However, BridgeQA depends on 3D point clouds to extract spatial information for QA, requiring complex 3D-2D -languaee alignment. Additionally, its retrieval- based approach risks overlooking critical views (which we will show in the experimental sections). In contrast, our method leverages multiple 2D views to understand 3D, meanwhile utilizing the strong language alignment already achieved by pre-trained 2D LVLMs.
2D-based Methods. Recent 2D-based methods use uni- formly sampled 2D views as input to 2D LVLMs (Singh et al., 2024; Zheng et al., 2024; Liu et al., 2024b), primar- ily focusing on evaluating the performance of 2D LVLMs on 3D-QA. They focus more on evaluating pretrained 2D LVLMs on 3D-QA tasks, rather than developing approaches to adapt and improve their performance for spatial reason- ing. Some more recent works have attempted to utilize 2D views more effectively. OpenEQA (Majumdar et al., 2024), transforms visual information into textual context, such as frame-level or scene-graph captions, and then lever- ages LLMs to answer questions. This approach depends on whether the generated text description can accurately cap- ture the critical visual details, which may lead to incomplete or inaccurate information.
3. Preliminaries
Leveraging pre-trained 2D LVLMs in a zero-shot manner for 3D-QA tasks is promising yet underexplored. Since 2D LVLMs are fundamentally designed to process 2D im- ages as input, we propose cdViews to efficiently select the most informative 2D views of 3D scenes. To understand the complexities in view selection, we conduct a prelimi- nary study using intuitive view selection methods, taking LLAVA-OV (Li et al., 2024a) as the backbone and using the validation set of the ScanQA dataset (Azuma et al., 2022). Note that it requires no training data due to the zero-shot
Compared to the above methods, we make two key contri- butions. First, we are the first to leverage 2D LVLMs via zero-shot inference (or by plugging a lightweight module) to address 3D-QA tasks. Second, we identify view selection as critical factor in zero-shot 3D-QA, for which there is
3

---

3D Question Answering via only 2D Vision-Language Models
<Question>
<D Views>
What is the black couch facing?
Visual Encoder
Image Retrieval Model
sort
Tokenizer
Projection
<Question>>
IR(Q. M)
Top-k
viewAnnotator
(5
View Selection:
2
Linear
Transformer
Pooling
Cosine similarity
sort
3
Linear
Transformer
Pooling
<Answer>
LVLM
viewSelector
viewNMS
Coffee table
Figure 3: The pipeline of zero-shot 3D-QA using three different view selection methods: uniform sampling (option @), image retrieval (option @), and our cdViews (option @). The views marked with * are selected ones. As for inference, our cdViews has two modules to run: the viewSelector identifies critical views, and the viewNMS enhances view diversity and minimizes redundancy. The viewSelector is trained using automatically generated labels from the vi eww.notatos module, which is detailed in Figure 5.
EMgg
30.1
either be question-depenndntt (denoted as F(M, Q k:), or not (denoted as F(M, k)). Then, M' and the question Q are input into the model to produce the answer A:
29.5
20.1
28.8
28.3
37,5
27.9
A = LVLM(M', Q).
(2)
26.5
19.4
25.5
The same zero-shot inference process applied throughout all experiments in this work, with variations only in two key aspects: the view selection function F and the number of selected views & that determine the final set of views M'.
153
24.5
12.9
k: Number of Selected Views
11
13
19
21 Ours
28
uninform sampling
image retricval
Uniform Sampling vs. Image Retrieval. We show the zero- shot experimental results of these two methods in Figure 4. We also include the results of our cdViews for comparison.
Figure 4: Performance comparison of view selection
methods on the validation set of ScanQA (Azuma et al., 2022). can be observed that: 1) performance improves with an increasing number of views, peaks at certain point, and finally declines; and noticeable performance gaps arise from different view selection methods, highlighting the importance of effective view selection. An earlier peak 130.1) appears in cdVi iews thanks to vi viewNMS.
1) Uniform sampling randomly selects 2D views without considering the context of the question Q (option ① in Fig- ure 3), formulated as:
F Funiforn ( (M, (V,, J (niform(1,NN)
(3)
Uniform sampling is the most straightforward way to select 2D views as input into 2D LVLMs for 3D-QA, and the best achieved metric score of EM@1 28.3%.
approach. In the following, we first present problem for- mulation for zero-shot 3D-QA, followed by experiments using two intuitive view selection methods: uniform sam- pling and image retrieval.
2) Image retrieval has been used in BridgeQA (Mo & Liu, 2024). Following (Mo & Liu, 2024), use the BLIP's image-text retrieval model (Li et al., 2022) to select views that best match the question Q (option @ in Figure 3). This process can be represented as:
Problem Formulation. Given question Q and a 3D scene represented by set 2D views M (V,, V2...., VN), each associated with camera matrix containing the position and orientation. The view selection identifies subset of k views (that are useful to answer Q), denoted as M':
Freurival (M, k, (Vi, 1 E (IR(. (4)
M' = F(M.Q,k) = [V., Vy...., ...,. V.J.
(L)
where IR( denotes the semantic similarity scores be- tween Q and every view in M, i.e. identifying the views
where & S N, F a view selection function, which can

---

3D Question Answering via only 2D Vision-Language Models
Step 1: Caption Generation
Promptæ >: You are helpful assistant. For each QA pair, generate a caption that describes the visual scene, fully incorporating relevant information from the question and answer.
<Answer>: brown cabinet with tv sitting in it a brown cabinet with television inside is located the right comerr the room, near the curtains.
<Question>:
What is in the right corner of room by curtains?
Step 2:
View Matching
Promptu You are given an image and caption describing the visual content. Determine if the image matches the caption, and respond with one of the following options:
A. Yes, fully matches.
B. No, does not match. C. Uncertain, insufficient or unclear information.
Positive Negative
Figure 5: Our viewAnnotator module operates in two steps: Caption Generation and View Matching (illustrated by light green boxes indicating outputs at each step). In Step I, LVLMs processes question-answw answer pairs to produce detailed descriptive captions. Step 2, these captions are compared against sampled views to assess their relevance in answering the corresponding questions. For clarity, the figure depicts only positive (A) and negative (B) view matches, excluding uncertain (C) ones.
semantically aligned with the question. As shown in Fig- ure 4, the best EM@I score that this approach achieves is 29.1%, slightly outperforming uniform sampling.
viewAnnotator to label views as positive, negative, or uncertain based on their matching scores with the descrip- tive captions (generated from question-answer pairs). Then, we train wwwwweleeeeee lector with these labels in supervised manner. For the selected views, we introduce viewNMS to remove redundant ones and improve the view diversity.
Analysis. Overall, image retrieval shows modest improve- ments over uniform sampling. It relies on the semantic similarity between questions and views, which introduces two key limitations: 1) Missing Critical Views. While it ef- fectively identifies views containing objects explicitly men- tioned in the question, it frequently overlooks relational cues essential for answering the question. This limitation stems from the fundamental difference between object identifica- tion and relationship comprehension, and the latter requiring stronger understanding capabilities. 2) Redundancy. Our analysis shows that views from adjacent viewpoints typi- cally receive similar semantic similarity scores, resulting in the selection of overlapping views. This redundancy limits the diversity of visual information captured across multi- ple views, reducing the overall effectiveness of the image retrieval approach.
4.1. viewAnnotator
The implementation of vi vieeAnnotator has two steps: caption generation and view matching, shown in Figure 5. Both steps the same LVLM in the zero-shot 3D-QA (i.e., the final inference model). This process aims to identify the critical views that match mostly the content of both input questions and the corresponding answers. Please note that these data are from the training set where the answers are available for use.
Caption Generation. It begins by feeding question- answer pair A) and rephrasing prompt (Prompta) into the LVLM,, as in Step 1 of Figure 5. This prompt is fixed for every question-answe answer pair and instructs the model to rephrase the pair into an image caption C' which abridges the reasoning between the question and answer:
4. cdViews: Critical and Diverse Views
Based on the above analysis, we argue that effective zero-shot 3D-QA requires identifying views that are both critical to represent the key information in the scene and sufficiently diverse to cover the scene. To this end, we introduce edViews, i.e. the option @ in Fig- ure 3. In the inference stage of 3D-QA, cdViews loads two modules, viewSelector and viewNMS. The train- ing of viewSelector contains two steps: data anno- tation and model training. First, we propose an auto
C' = LVLM(Q A, Promptg).
(5)
Please note that caption generation is crucial prior step of view matching. Directly using the (Q, A) pair for matching causes the model to focus on answering the question rather than labeling the views. In other words, it encourages the model to take a shortcut by simply copying the answer A.
View Matching. For each view Vi in set of 2D views
5

---

3D Question Answering via only 2D Vision-Language Models
M. we evaluate its information relevance the generated caption using LVLM. Specifically, prompt the caption C and matching prompt Prompt M, in Step 2 of Figure 5, to LVLM. LVLM classifies V; into one of three categories, "positive", "negative", or "uncertain", respectively corre- sponding to the options A, B, and C in Promptar.
where N' < N is the number of views labeled ("posi- tive") or 0 ("negative").
During inference, viewSelector acts as scoring func- tion to evaluate each input view: a higher score S, indicates higher criticality of Vi.
S = LVLM(C, VA, ar)mplas);
(6)
4.3. viewNMS
The views selected by viewSe may introduce re- dundaney: overlapping views might all get high scores- similar to the problem of image-retrieval- We propose viewNMS to filter out redundant views. We leverage camera parameters, i.e., position and orientation, calculate distances between selected views, and discard views less distant than a predefined distance threshold.
where S; E {0. } is the classification label of the view V.. For example, in Figure 5, a view is classified as "positive" (S; - 1) because it contains the correct objects with spec- ified attributes and spatial relationships, such as a "brown cabinet" with a "television" inside and "curtains" nearby. Otherwise, views are labeled as "negative" (S, 0). Views classified as "uncertain" when the model chooses the option of UUncettain insufficient or unclear information" or outputs none of the given options, and these views are excluded from training.
methods.
Specifically, viewNMS operates in three steps: 1) Ranking views sorts all views {V.),= by their scores { in descending order, resulting in [V;,) where 4. is the highest-scoring view. 2) Initializing candidate views se- lects the highest-scoring view the initial set MM' { 3) Adding diverse views sequentially processes the remain- ing views in sorted order, adding view Vú, to the set if its distance from previously selected views exceeds a threshold T, formulated as:
4.2. viewSelector
As shown in Figure 3, viewSelector is plugged be- tween the visual encoder and LVLM to select "views" in the feature space. It takes the question embedding Q and the visual embedding set ( as input and outputs a binary label S; (0 or 1) for each visual embedding. Then, Š; is compared to the corresponding view label generated by the vi #ieAnnotatoz The mismatch loss is used to optimize the parameters of viewSelecton
(9) M' V.. if Vj) > T, E M..
Finally, viewNMS outputs new set of selected views M', which are both critical and spatially diverse. After that, M' and Q are fed into the 2D LVLM to generate an answer which is the final output of zero-shot 3D-QA.
Specifically, the question embedding Q is first passed through a linear layer. followed by a two-layer Transformer block, and a pooling layer. The output can be regarded as a compact summary of the question, producing question vector q. Similarly, for visual inputs, each visual embedding V; is processed through the same modules. We apply cross- attention in each transformer layer between the question embedding Q and the visual embeddings (V.) in order to enhance the model's ability to identify views containing critical content for QA. After pooling, the resulting set of vectors [v serve as compact summaries of question- aligned visual embeddings.
View Distance Calculation. The core of viewNMS lies in the calculation of the view distance, i.e., D(V, Vz), measuring the cameras' position and orientation distance between V, and Vy. For each view, the camera parame- ters [R]t] (we omit the subscript for simplicity) consist of a camera orientation R E R3x3 and a camera position t E R The distance is calculated by combining both the orientation distance and position distance. For the orien- tation R, we first convert into a quaternion representation p = [Pe: Py: Pz, Pm]] for more efficient distance calculations. Then, the orientation distance Dori Vj) is calculated by
N.
Finally, the outputs q and (v:), are used measure the criticality between the question and each view by cosine similarity:
Dori (V,, V2)) = 2 arccos( (Ip: p,]).
(10)
S = q V;
where arccos represents the inverse cosine function. This formula gives the angular distance in radians between the orientations of two views. Since arccos( (Ip. Pyy)) yields half the angle, the factor of restores the full angle difference.
(7)
||lll|aal|
The score S, is supervised with the corresponding label S; by binary cross-entropy loss:
The position distance Dpas (V,, V2)) between views V; and Vi is calculated using the Euclidean distance between their camera positions t, and ty:
N'
LBCE =
1
Σ E1
S lo.gll((s)) +(1 (1- 5.) log(1 - s.))
N
(8)
Dpos (V,, V2) = ||t, - -tyll.
(11)
6

---

3D Question Answering via only 2D Vision-Language Models
ScanQA
SQA
Method
Type
EM@1
BLEU-I
ROUGE
CIDEr
EM@1
ScanQA (Azuma et al., 2022)
23.5 20.9
31.6/ 30.7
34.3 / 31.1
67.3 60.2
45.3
SQA3D (Ma ct al., 2022)
3D
47.2
3D-LLM (Hong et al., 2023)
3D
19.1
38.3/
35.3
69.6/-
48.1
3D-VLP (Jin al., 2023a)
3D
21.6
33.2/31.5
36.0/31.8
70.2
3D-Vis VisVisTA (Zhu al., 2023)
27.0/ 23.0
38.6/ 32.8
76.6/ 62.6
48.5
SIG3D (Man et al., 2024a)
3D
52.6
SynFormer3D (Yang et al., 2024)
3D
27.6/ 24.1
39.2/
00.2762.7
LL3DA (Chen et al., 2024a)
3D+2D
38.2 1.2/335
78.2 70.3
PQ3D (Zhu et al., 2025)
3D+2D
26.1 20.0
43.0 36.1
87.8 65.2
47.1
BridgeQA (Mo & Liu, 2024)
3D+2D
31.3/ 30.8
34.5 34.4
43.3/41.2
83.8 79.3
52.9
I.LAVA-VV Funiform
2D
33.1 33.5
43.2 44.2
46.9 46.6
95.8 93.3
53.5
LLAVA-OV Fuasieal
2D
33.9 34.6
44.8 46.1
48.3/44..7
98.8 97.7
55.0
LLAVA-OV + Fesvieus
35.0 35.6
46.1 47.2
49.7
49.5
102.8 100.4
56.9
margin over the compared best
3.7 T 14.8
3.1
1/9.1
6.4
15.0 t 21.1
3.9 1
Performance comparisons with the state-of-the-ar methods on the test set of ScanQA (Azuma al., 2022) and SQA (Ma et al., 2022). For ScanQA, scores are presented in the format "with object test set" "without object test set". The best and second best results are in bold and underlined, and the last row shows the performance margins between LLAVA-OV + F.nDOVFFaavaiee Fedviews and the top-performing related methods.
Table I:
where ||
||
is the Euclidean norm.
Implementation Details. We utilize recent state-of-th art LVLM, i.e., LLAVA-OV-7B (Li et al., 2024a), as the 2D LVLM for all experiments, including viewAnnotatox and 3D-QA. The model remains frozen throughout all experiments. Analysis on more LVLM backbones is shown in Appendix C. The only trainable component is viewSelector, a lightweight module with total of 5.9M parameters. Training of the viewSelector is con- ducted with learning rate of 5 x 10 and a batch size of 8. Each training iteration samples positive and 5 negative views per instance generated by vie vieeAnnotator. Here the number of views, e.g., k= for cdView is selected the validation set (Figure 4).
The final camera distance D(V,, V2) is sum of the position and orientation distances,
DVV, V2) = Dpos VG, Vj)+ + Dori(Vi, Vz)-
(12)
Combining the camera's position and orientation, the dis- lance estimates the spatial overlap between the regions cap- tured by two views, with smaller values indicating greater overlap. An ablation study on threshold selection is pro- vided the experimental section.
5. Experiments
5.1. Comparisons with the State-of-the-Arts
Datasets. We use ScanQA (Azuma et al., 2022) and SQA (Ma et al., 2022) in our experiments, both constructed from ScanNet dataset (Dai et al., 2017). ScanQA contains over 41K question-anss answer annotations across 800 indoor 3D scenes, which are divided into train, val, and test (with without objects). SQA contains over 33K question- answer pairs derived from 650 indoor scenes. It encom- passes diverse range of question types, including object identification, spatial relationships, scene-level understand- ing, and general reasoning.
Table presents the quantitative results comparing 2D- only methods (uniform sampling, image retrieval, and cdViews) with other LLAVA-OV (Li et al., 2024a) with state-of-the-art 3D and hybrid methods. First, it is ob- served that 2D-only methods achieve superior performance, showing the advantage of applying 2D pre-trained models for 3D tasks. For example, compared to BridgeQA (Mo & Liu, 2024), our Faaviews achieves significant improve- ments of 15.0% and 21.1% CIDEr on the two test sets of ScanQA. Second, among the 2D-only methods, cdViews outperforms the others. For example, Foavi outper- forms F by 4.0% and 2.7% CIDEr on both test sets of ScanQA. The reason is that the uniform sampling method ig- nores the question and the image retrieval method often fails to capture critical views or introduces redundancy views. In contrast, cdView effectively identifies critical and diverse views for efficient 3D-QA. The qualitative comparison of
Evaluation Metrics. We adopt Exact Match (EM@) 1) for both datasets. EM@ measures the proportion of cases where the 1 predicted answers match any of the ground- truth answers. Furthermore, since the answers in ScanQA are often free-form, we use standard text similarity metrics, including BLEU-1 (Papineni et al., 2002), ROUGE-L (Lin, 2004), and CIDEr (Vedantam et al., 2015) to assess the quality of generated answers.

---

3D Question Answering via only 2D Vision-Language Models
<Question > What is to the right of the chair?
Answer >: desk
BridgeQA: ottoman
* LLAVA-OV + F retrievat couch
LLAVA-OV + F Faviiew
desk
<Question >; What is on table next to a bed?
<Answer >; fan
BridgeQA: lamp
* LLAVA-OV F retrieval pillow
LLAVA-OV + F caView
fan
Figure 6: Qualitative results for BridgeQA (Mo & Liu, 2024). LLAVA-OV F Freuieval and our final model LLAVA-OV + Fedviens- The marks A, x, and represents the selected views respectively by three methods. We can that cdV New captures the most critical and diverse views to answer the questions.
LVAVA-OV
view Selector
view NMS
Best EM@1
Optimal
EMg1
30
29.
30.1
+ best k
k=9
29.7
(K-11)
+ Faniform
28.3
29.6
Fretrieval
29.1
17
(k-17)
29.7
29.3 (k=7)
29
Fretrieval
/
29.2
9
29.4
+ Feavions
29.7
29.2
+ Feaviees
30.1
T
Table 2: An ablation study performed on ScanQA. We show the best EM@ I scores with the corresponding (optimal) k.
Figure The results of EM@ using two configurations: optimal & (blue) vs. fixed 9 (green). X-axis is the thresh- old T of viewNMS. T= 0 means disabling viewNMS.
selected views is shown in Figure and Figure 6. More comparisons are provided in the Appendix Section B.
implementation of Foavi where viewNMS reduces the imput to just 9 views-almost half the visual token length- without reducing the performance, but further boosting EM@1 by 0.4%. This is due the reduced redundancy allowing the model to focus more on critical views. A com- parison between the third and last rows shows that our full pipeline Feaviews outperforms the retrieval + viewNMS baseline by 0.9% FM@ 1 (30.1% % vs. 29.2%), using the same number of input views. Even after redundancy re- moval via viewNMS, the retrieval- basee approach remains constrained by its initial candidate views, which are selected based question-view semantic similarity rather than their criticality to question answering. This further highlights the strength of our learned v viewSelecte identifies views that are critical for question answering.
5.2. Ablation Study
In this section, we conduct ablation study the valida- tion set of ScanQA (Azuma et al., 2022), following (Mo & Liu, 2024). We study the impact of cdvi ews components and viewSMS thresholds. In addition, we particularly com- pare ours with the most related work: image-retrieval-ben 3D-QA (Mo& & Liu, 2024). More ablation studies are in Section C of the Appendix.
edViews Components. The experimental results marized in Table 2. The first row shows the baseline per- formance using randomly sampled 2D views as input, i.e. Funiform achieving the best result of 28.3% EM@1 with 17 views. The second and third rows present results using the image retrieval baseline. Compared to uniform sam- pling, retrieval provides better views and improves EM@ 29.1% (with 17 views). When combined with vi viewNMS, the number of input views is reduced 9, and performance slightly improves to 99.2%. The fourth row presents the alone,
which explicitly
vi wiewNMS Thresholds. We evaluate the effect of different viewNMS thresholds (0, 0.25, 0.5, 0.75, and 1.0) in Fig- ure 7. As the threshold increases, the optimal number of input views decreases from 17 to 9, demonstrating the effec- tiveness of vi viewNMS in reducing redundancy. The highest accuracy is achieved at a threshold of 0.5, with only 9 views input. When the number of views is fixed at 9, performance improves with increasing thresholds, peaking at 0.5 before declining. indicates that excessively high thresholds may
performance of Feaviews with the wiewSelecte which achieves 29.7% EM@1 I with 17 views, improving effec-
by 1.4%. This validates that the viewSelecte tively prioritizes critical views. The last row reports the full
8

---

3D Question Answering via only 2D Vision-Language Models
Method
Fretieval
Fesvisws
showing the potential of 2D LVLMS as effective alterna- tives to resource-intensive 3D LVLMss Potential societal consequences include improvements in autonomous sys- tems, assistive technologies, and interactive environments, where efficient 3D scene understanding is critical. While no immediate risks or concerns are identified.
Model
BIT (retrieval)
odViews
Parameters
644M
5.9M (-9.1%%)
FLOPs
593.6T
294.5T 50.4%)
Inference Time
2.8s
1.2s (-7.1%))
Table 3: Computational performance comparison between image retrieval and cdViews for zero-shot 3D-QA.
References
loss spatially close views, and thus miss critical information.
Azuma, D., Miyanishi, T., Kurita, S., and Kawanabe, M. Scanqa: 3d question answering for spatial scene under- standing. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp- 19129 19139, 2022.
cdViews's Efficiency. We compare the efficiency of im- age retrieval and our proposed cdViews in Table 3. As lightweight plug-in module to LVLMs, Feavieus only in- troduces 5.9M parameters, while the parameters of Fentieval is 100 times as Feaviews- Furthermore, Feaviews reduces FLOPs by half and cuts inference time by more than 50% compared Freirieall These results demonstrate the effec- tiveness of cdVi ews in improving accuracy, streamlining inference, and reducing computation.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Chen, S., Chen, X., Zhang, C., Li, M., Yu, G., Fei, H., Zhu, H., Fan, J., and Chen, LI3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning. In Proceedings of the IEEE/CVF Confer- ence Computer Vision and Puttern Recognition, pp. 26428-26438, 2024a.
6. Conclusions
In this work, we leverage 2D LVLMs in zero-shot manner (or plugging a lightweight module) to address 3D-QA and identify view selection as critical factor affecting perfor- mance. Our preliminary study reveals that effective view selection must ensure both critical and diversity. To this end, we propose cdVies aws, a view selection framework compris- ing viewSelector which prioritizes critical views, and viewNMS, which enhances spatial diversity by removing redundant views. Extensive experiments on the ScanQA and SQA datasets demonstrate that cdViews achieves state-of- the-art performance.
Chen, Y., Yang, S., Huang, H., Wang, T., Lyu, R., Xu, R., Lin, D., and Pang, J. Grounded 3d-lim with referent tokens. arXiv preprint 24N5::2005.103700 2024b.
Dai, A., Chang, A. X., Savva, M., Halber, M., Funkhouser, T., and Nießner, M. Scannet: Richly-annotated 3d recon- structions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5828-5839, 2017.
Acknowledgments
This research is supported by the RIE2025 Industry Align- ment Fund- - Industry Collaboration Projects (IAF-ICP) (Award 12301 E0026), administered by A*STAR, as well as supported by Alibaba Group and NTU Singapore, and the Major Research Program of Jiangsu Province (Grant
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXh:: 2024.
Fu, R., Liu, J., Chen, X., Nie, Y., and Xiong, W. Scene-Ilm Extending language model for 3d visual understanding and reasoning. arXiv preprint arXiv:2403.1 1401, 2024.
BG2024042).
Impact Statement
Guo, J., Li, J., Li, D., Tiong, A. M. H. Li, B., Tao, D., and Hoi, S. From images to textual prompts: Zero-shot visual question answering with frozen large language models. Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pp. 10867-10877, 2023,
This paper presents work aiming to advance machine learn- ing by introducing edViews. It integrates a viewSelector and viewNMS automatically select critical and diverse views for 3D question answering (3D-QA). By relying solely 2D views and pre-trained LVLMs,, this approach addresses the challenge of limited 3D training data and avoids the need for direct alignment between 3D and language representa- tions. The proposed method demonstrates state-of-the the performance on benchmarks such as ScanQA and SQA,
Hong, Y., Zhen, H., Chen, P., Zheng, S., Du, Y., Chen, Z., and Gan, C. 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 36:20482-222 224994 2023.

---

3D Question Answering via only 2D Vision-Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L.,, Chen, W., al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Lu, S., Liu, M., Yin, L., Yin, Z.., Liu, X., and Zheng, W. The multi-modal fusion in visual question answering: review of attention mechanisms, PeerJ Computer Science, 9:e1400, 2023.
Huang, H., Chen, Y., Wang, Z., Huang, R., Xu, R., Wang, T., Liu, L., Cheng, X., Zhao, Y., Pang, J., et al. Chat-scene: Bridging 3d scene and large language models with object identifiers. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Ma, X., Yong, S., Zheng, Z., Li, Q.. Liang, Y., Zhu, S.-C., and Huang, S. Sqa3d: Situated question answering in 3d scenes. arXiv preprint arXiv:2210.07472 2022.
Majumdar, A., Ajay, A., Zhang, X., Putta, P., Yenamandra, S., Henaff, M., Silwal, S., Mevay, P., Maksymets, O., Arnaud. S., et al. Openeqa: Embodied question answering in the of foundation models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16488-16498, 2024.
Huang, J., Yong, S., Ma, X., Linghu, X., Li, P., Wang, Y., Li, Q. 7hu, S.-C., Jia, B., and Huang, S. An embodied gener- alist agent 3d world. arXiv preprint arXiv:2311.12871, 2023.
Jin, Z., Hayat, M., Yang, Y., Guo, Y., and Lei, Y. Context- aware alignment and mutual masking for 3d-language pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 10984-10994, June 2023a.
Man, Y., Gui, L.-Y., and Wang, Y.-.. Situational awareness matters in 3d vision language reasoning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
2024a.
Jin, Z., Hayat, M., Yang. Y., Guo, Y., and Lei, Y. Context- aware alignment and mutual masking for 3d-language pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 10984- 10994, 2023b.
Man, Y., Zheng, S., Bao, Z., Hebert, M., Gui, L.-Y., and Wang, Y.-X. Lexicon3d: Probing visual foundation mod- els for complex 3d scene understanding. arXiv preprint arXiv: 2rXiv:2409.03757 2024b.
Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Li, Y., Liu, Z., and Li, C. Llava- onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326. 2024a.
Mo, W. and Liu, Bridging the gap between 2d and 3d visual question answering: A fusion approach for 3d vqa. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 4261-4266 2024.
Li, F., Zhang, R., Zhang, H., Zhang, Ma, Z., and Li, C. flaa--nxt-interleee Tackling multi- image, video, and 3d large multimodal models. arXiv preprint arXiv:2407.000 407,07895, 2024b.
Li, W.,
OpenAl. Gpt-4o: Openai's ss optimized gpt-4 model. https: / /ppenai... com/ htom//pt-aal 4o, 2024. Accessed: 2024-05- 22.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-. Bleu: a method for automatic evaluation of machine transla- tion. In Proceedings of the 40th annual meeting of the Association for Computational Linguisties, pp. 311-318, 2002.
Li, J.. Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping language- trggage--mage pre-training for unified vision vasgonelanguage understanding and generation. In International confer- ence on machine learning, pp. 12888-12900. PMLR, 2022.
Qi, C. R., Litany, O., He, K., and Guibas, L. Deep hough voting for object detection in point clouds. In proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 9277-9286, 2019.
Lin, C.-Y. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp.
71, 2004.
Liu, B., Dong, Y., Wang, Y., Rao, Y., Tang, Y., Ma, W.- C., and Krishna, R. Coarse correspondence elicit 3d spacetime understanding multimodal language model. arXiv preprint ar arXiv:2408.007: 2024a.
Shao, Z., Yu, Z., Wang, M., and Yu, J. Prompting large language models with answer heuristics for knowledge- based visual question answering. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp. 14983, 2023.
Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. J. Llava-next: Improved reason- ing, ocr, and world knowledge, January 2024b. URL https://1lavaa wtw.llawwwwww..itu.......... 2024-01-30-11ava-nee
Singh, S., Pavlakos, G., and Stamoulis, D. Evaluating zero- shot gpt-4v performance on 3d visual question answering benchmarks. arXiv preprint arXiv:2405. 10005//883 18831, 2024.
10

---

3D Question Answering via only 2D Vision-Language Models
Vedantam, R., Lawrence Zitnick, C., and Parikh, D. Cider: Consensus- based image description evaluation. In Pro- ceedings of the IEEE conference computer vision and pattern recognition, pp. 4566-446 4575, 2015.
Yang. D., Xu, Z., Mo, W., Chen, Q., Huang, S., and Liu, Y. 3d vision and language pretraining with large-scale synthetic data. arXiv preprint arXiv:2407.06084, 2024.
Zhang. T., He, S., Dai, T., Wang, Z., Chen, B., and Xia, S.-T. Vision-language pre-training with object contrastive learn- ing for 3d scene understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 7296-7304, 2024.
Zheng, D., Huang, S., Zhao, L., Zhong, Y., and Wang, L. Towards learning generalist model for embodied navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13624- 13634, 2024.
Zhou, Y. Li, X., Wang, Q., and Shen, Visual contextt learning for large vision-language models. arXiv preprint arXiv:2402. 1574, 2024.
Zhu, Z., Ma, X., Chen, Y.. Deng, Z., Huang. S., and Li, Q. 3d-vista: Pre-trained transformer for 3d vision and text alignment, 2023.
Zhu, Z., Zhang, Z., Ma, X., Niu, X., Chen, Y., Jia, B., Deng, Z., Huang, and Li, Q. Unifying 3d languaeee understanding via promptable queries. In European Con- ference Computer Vision, pp. 188-206. Springer, 2025.

---

3D Question Answering via only 2D Vision-Language Models
This supplementary includes the details of view matching in viewAnnotat (Sec. A), more comparisons with the State-of-the-Arts (Section B), more ablation studies (Section C), including ablation of view selection methods with different 2D LVLM, effectiveness of caption generation in viewAnnotator, and more case studies (Section D).
A. More Details in View Matching
This supplementary for Sec. 4.1 of the main paper. In our viewAnnotator, view matching classifies views as positive, negative, or uncertain. However, directly using a 2D LVLM with the prompt Promptas an instruction unreliable, the model lacks explicit judgment criterion. To address this, we leverage its strong in-context learning ability (Zhou et al., 2024) by providing a textual context example that guides the model through structured reasoning process. Specifically, we incorporate step-by-step system prompt in the View Matching process. As shown in Figure S1, the system prompt ensures that all key objects, attributes, and spatial relationships in the caption align with the image, reducing ambiguity and improving consistency. Uncertain views are explicitly excluded, enhancing the robustness of the annotation process. Additional examples of positive and negative views are shown Figure S2.
System Prompt 2: Consider the following example to guide your responses: Caption: "A brown cabinet with a television inside is located the right corner of the room, near the curtains. In this example, following the steps:
1. List all objects elements mentioned in the caption:
Brown cabinet Television inside the cabinet Curtains nearby
Check all objects from the caption are present image:
Yes, if all objects from the caption (brown cabinet, television, and curtains) present in the image, proceed to step 3.
Noo, answer with option B.
Verify the objects' attributes and relative positions match the caption:
Yes, the cabinet brown, the television is inside the cabinet, it is positioned in the right corner, and it is near the curtains.
If any attributes or positions do not match the caption, answer with option B.
If the image contains partial but unclear information, answer with option
< >rompt You given image and caption describing the visual content. Determine if the image matches the caption, and respond with one of the following options:
A. Yes, fully matches.
B. No, does not match. C. Uncertain, insufficient or unclear information.
Caption>:an orange storage bin is placed top of white cabinet.
Positive views
Negative views
Figure S1: Illustration of how context guides the view matching process. In the view matching process of viewAnnotator. the model follows a structured reasoning approach, using a textual example to classify views as positive, negative, or uncertain.
To further validate the reliability of the positive views, we conducted a human evaluation: We randomly selected 50 QA pairs with their associated positive views. Three human evaluators assessed whether each view could answer the question. Their accuracy rates were 96.72%, 94.28%, and 97.56%, confirming that the quality of positive views is sufficient for training.
B. More Comparisons with the State-of-the-Art Methods
This supplementary is for Section 5.1 of the main paper. Table S1 presents the quantitative results comparing LLAVA- OV (Li et al., 2024a) with different view selection methods, including uniform sampling, image retrieval, and our cdViews, against state-of-the-art methods on the validation set of ScanQA. As shown, LLAVA-OV V Feaviews outperforms these
12

---

3D Question Answering via only 2D Vision-Language Models
Caption>:the wall with pictures is the right side of the door.
Positive views
Negative views
< Caption>:the microwave is placed on top of a storage box.
Positive views
Negative
views
Caption>; a cabinet is located in the room to the left of a table
Positive
views
Negative
views
Figure S2: Examples automatically annotated positive and negative views. Each case shows a caption along with its corresponding positive and negative views. Positive views closely match the caption in terms of key objects, attributes, and spatial relations, while negative views lack full correspondence.
Method
Type
EM @
BLEU-1
ROUGE
CIDEr
ScanQA (Azuma et al., 2022)
3D
20.3
29.5
32.4
61.7
3D-LLM (Hong et al., 2023)
3D
20.5
39.3
35.7
69.4
3D-VLP (Jin al., 2023a)
3D
21.7
30.5
34.5
67.0
LL3DA (Chen et al., 2024a)
3D+2D
37.3
76.8
BridgeQA (Mo & Liu, 2024)
3D+2D
27.0
GPT-40+CC (Liu et al., 2024a)
2D
35.4
42.6
87.0
LLAVA-OV Funiform
2D
28.3
40.2
44.5
88.0
I.LAVA-OV + Fretrienal
2D
29.1
41.5
45.8
91.6
LLAVA-OV + FodViews
2D
30.1
42.6
46.8
94.0
margin over the compared best
3.1
3.3
4.2
7.0
Table S1: Result comparisons with the state-of-the-art methods on the validation set ScanQA (Azuma et al., 2022). The best and second best results are in bold and underlined.
methods by clear margins. The last row of Table highlights the performance gap between LLAVA-OV FedVieme and the best-performing baselines. Even compared to GPT-4++CC (Liu et al., 2024a), which leverages the powerful capabilities of GPT-40 (OpenAI, 2024), LLAVA-OV Faaviews surpasses it by 7.0% CIDEr. GPT-40+CC improves spatial understanding by adding object markers to track correspondences across uniformly sampled views. However, it overlooks the relevance between the selected views and the input question, limiting its effectiveness in 3D-QA.
Table S2 presents the quantitative results on SQA (Ma et al., 2022), detailing performance across different question types:
"What", "Is", "How", "Can", "Which", and "Other". Compared to state-of-the-ar methods, LLAVA-OV Feaviews achieves
13

---

3D Question Answering via only 2D Vision-Language Models
the best performance on "What", "How", "Which", and "Other" questions but shows decline of 2.8% and 10.2% "Is" and "Can" questions, respectively. This decline may be attributed to the zero-shot nature of LLAVA-OV (Li et al., 2024a), which maintains balanced performance across all question types. In contrast, other methods exhibit uneven performance, excelling in Is" and Can" questions due to dataset-specifee adaptation while potentially underperforming in other categories. ews consistently outperforms uniform sampling and image retrieval across all question types, demonstrating its effectiveness selecting critical views for 3D-QA.
Furthermore, based on the same 2D LVLM, LLAVA-OV, cdVi eww
Question Breakdown
Method
Input
Overall
What
Is
How
Can
Which
Other
GPT-3 (Brown et al., 2020)
3D
39.7
46.0
40.5
45.6
36.1
38.4
41.0
ScanQA (Azuma al., 2022)
3D
28.6
65.0
47.3
66.3
43.9
42.9
45.3
SQA3D (Ma et al., 2022)
3D
33.5
66.1
42.4
69.5
43.0
46.4
47.2
3D-LLM (Hong et al., 2023)
3D
36.5
65.6
47.2
68.8
48.0
46.3
48.1
3D-VisTA (Zhu et al., 2023)
3D
34.8
63.3
45.4
69.8
47.2
48.1
48.5
SIG3D (Man et al., 2024a)
3D
35.6
67.2
48.5
71.4
49.1
45.8
52.6
PQ3D (Zhu et al., 2025)
3D+2D
37.1
61.3
44.5
60.9
47.0
45.1
47.1
BridgeQA (Mo & Liu, 2024)
3D+2D
52.9
LLAVA-OV Funiform
2D
51.4
60.7
49.6
56.2
51.6
51.9
53.5
ILLVAA-V Fretrieval
2D
54.8
62.4
50.3
56.5
49.3
53.2
55.0
LLAVA-OV F FdVViews
2D
55.0
64.4
54.0
61.2
51.6
54.4
56.8
margin over the compared best
15.3 f
2.8 4
5.5 f
-10.2
2.5
6.3 1
3.9 f
Table S2: Result comparisons with the state-of-the-ar methods on the test set the SQA (Ma et al., 2022). The best and second-best results are bold and underlined. The decline in the "Is" and "Can" problems for LLAVA-OV with different view selections attributed the shoo- nature of LLAVA-OV, which ensures balanced performance across all question types. In contrast, the compared methods exhibit uneven performance, excelling in "Is" and "Can" questions due to dataset-specific adaptation while potentially underperforming in other categories.
C. More Ablation Studies
This supplementary is for Section 5.2 of the main paper. The ablation studies are conducted on the validation set of ScanQA (Azuma al., 2022), we evaluate the impact of different backbones, the effect of caption generation in viewAnnotaton and visualize the views within different distance thresholds, and visually compare the results of BridgeQA, LLAVA-OV Freuseval, and LLAVA-OV Fedvieus.
Ablation with Different Backbones. We evaluate the impact of different backbones by comparing LLAVA-NEXT (Li et al., 2024b) and LLAVA-OV (Li et al., 2024a), with results presented in Table $3. The results reveal two key insights: 1) View selection plays a crucial role in enhancing performance across models. Replacing uniform sampling with image retrieval improves performance by 1.1% on I.LAVA-NEXT and 0.8% on ILLVAA-V, underscoring the importance of selecting informative views for 3D-QA. Our cdViews further amplifies these gains, achieving improvements of 3.6% and 1.8%, respectively, by effectively identifying more critical views. 2) edViews demonstrates robustness and adaptability, consistently outperforming both baselines and delivering the highest performance gains across all evaluation metrics.
Effectiveness of Caption Generation in viewAnnotator. To assess the necessity of caption generation for view matching, we conduct an ablation study by removing the caption generation step in viewAnnotaten the generated caption C, the question-ans answerr pair A) is directly used as input for view matching. To better isolate the impact of caption generation, this ablation study is conducted without applying viewNMs. Specifically, Eq. 6 modified as:
Instead of using
S. LVLM(Q, A, Vi, Prompt'r).
(13)
where Prompt'y is an adapted version of Prompt M. with the term "caption" replaced by "question-answer pair." The textual context example is preserved to guide the view labeling step-by-step. The results, presented in Table S4, show that removing the caption generation step leads to 1.8% drop in CIDEr for LLAVA-OV + Feaviews. This highlights the
14

---

3D Question Answering via only 2D Vision-Language Models
Backbone
View Selection
@
BLEU-1
ROUGE
CIDEr
F Funiform
21.0
30.0
42.1
85.3
LLAVA-Next
Feetrieall
22.1 1.1f
35.2 5.21
42.3 0.21
87.0 1.71
Feavieus
3.61
9.61
44.9 2.81
93.7 8.47
F Funiform
28.3
40.2
44.5
88.0
LLAVA-OV
Ferriieal
29.1 0.81
41.5
1.31
45.8 1.3f
91.6 3.6f
FedViews
30.1 1.8t
42.6
2.1
46.8 2.37
94.0 6.07
Table S3: Ablation study results with different backbone models, LLAVA-Next (Li et al., 2024b) and LLAVA-OV (Li et al., 2024a). The best results are in bold. Subscripts indicate the relative improvement over the corresponding baseline, i.e., the 2D LVLM with uniform sampling.
Method
View Matching with Input Tuple
EM@1
BLEU-1
ROUGE
CIDEr
(Q, A, Vú, Prompt's)
29.5
41.4
45.9
91.4
LLAVA-OV
odViews
(C, Vi, Promptau)
29.7
42.2
46.4
93.2
Table S4: Ablation study on the necessity of caption generation for view matching. The key difference lies in whether the viewSelector trained with view labels generated using the caption C' the (Q,A) pair as input.
importance of generating a reformulated caption, which helps the model more effectively identify critical views compared to directly using the (Q, A) pair.
Effect of Finetuning VLVVA-OV in Hybrid Method. To assess the feasibility and effectiveness of incorporating LLAVA- OV into hybrid method, implement variant of BridgeQA-thh the strongest hybrid baseline in main comparisons. Specifically, retain the original BridgeQA architecture but replace its 2D Dision-language module (BLIP (Li et al., 2022)) with LLLVA-OV. For fair comparison, we also replace its top-1 image input with 9 views selected by our cdVi eww strategy, while keeping the full-scene point cloud features extracted by VoteNet (Qi et al., 2019).
During training, we adopt parameter-efficine tuning by updating only the last 2 of the 28 transformer layers in LLAVA-OV using LoRA (Hu et al., 2022). As shown in Table $5, the finetuned variant (BridgeQAuLAw. ov) achieves +1.4% EM improvement over the original BridgeQA baseline 128.4% vs. 27.0%), confirming the benefit of using a stronger 2D LVLM. Nonetheless, it still underperforms our edViews, which achieves 30.1% EM@1 1. This experiment demonstrates that while hybrid pipelines can benefit from stronger LVLMs, they rely complex architectures, 3D-specific modules, and computationally expensive fine-tuning. In contrast, our framework achieves superior performance by using 2D-only LVLM in zero-shot inference manner.
Method
Input
2D LVLM
EM@ 1
BLEU-1
ROUGE
CIDEr
BridgeQA
3D+2D
BLIP
27.0
BridgeQALLAvA-or
3D+2D
LLAVA-OV
28.4
37.3
42.7
84.0
cdViews
2D
LLAVA-OV
30.1
42.6
46.8
94.0
Table S5: Comparison between zero-shot cdViews and fine-tuned hybrid BridgeQA using LLaVA-OV. We compare the performance of the original BridgeQA, its fine-tuned variant with LLaVA-OV, and our zero-shot cdViews. While the hybrid variant benefits from stronger LVLM, our approach outperforms it with 2D-only LVLM in zero-shot inference manner.
D. More Case Studies
Visualize Comparison of Different Methods and Their Visual Inputs. Figure S3 presents a visual comparison of the predicted answers and visual inputs of BridgeQA (Mo & Liu, 2024), LLAVA-OV Freuieval and LLAVA-OV Feaviewe BridgeQA relies on the top-1 image retrieval view combined with point clouds as input. However, relying on point clouds
15

---

3D Question Answering via only 2D Vision-Language Models
provide the whole scene often results in answers that miss critical details. For instance, in the 4th row, while the model correctly mentions the trash more informative answer. For LLAVA-OV Frerrievll image retrieval-based view selection may miss the critical views required for accurate answers. As shown in the 3yd rows, the retrieved views tend to be redundant or incomplete. In the 3rd row, the selected views focus on the cabinet beneath the window but omit the view displaying books on top, which essential for correctly answering the question. In contrast, LLAVA-OV + Fedviews selects critical and diverse views, capturing essential context and delivering accurate, informative answers.
the floor, it overlooks surrounding objects like the toilet, which is crucial for providing
<Question >: What is the object that has lamp resting on it? BridgeQA: bed
<Answer nightstand
* LLAVA-OV retrieval desk
LLAVA-OV + +F Faviiew nightstand
<Question >: What is next to the brown rectangular shelf? BridgeQA: desk
<nnswe >: black filing cabinet
* LLAVA-OV Fretrieval door
LLAVA-OV + F Feaview black file cabinet
G COOTOO
<Question >; The small cabinet sits underneath the window with what on top of? BridgeQA: yes
<Answer >; book books
I LLAVA-OV + FFetrieval box
LLAVA-OV + FedView
<Question >; What does the trash can set? BridgeQA: on floor
<Answer >; next to toilet
LLAVA-OV Fetrieeall under counter
LLAVA-OV Feaview next to toilet
Figure S3: More Qualitative results for BridgeQA (Mo & Liu, 2024), LLAVA-OV F Feerieeall and our final model LLAVA- OV + Feaviews- Small marks A, *. and represents the selected views by each method. We see that edViews can capture critical and diverse views to answer the questions.
16


Reviews:
[{"review_id": "0C59pR2oCV", "reviewer": "Reviewer_rVhy", "summary": "This paper introduces cdViews, a zero-shot method for 3D question answering that avoids fine-tuning large vision-language models (LVLMs). Initially, viewSelector is employed to automatically select the most relevant views based on the input question. Then, viewNMS enhances diversity by eliminating redundant views, determined through their spatial overlap. Finally, the selected views, along with the question, are fed into a pre-trained 2D LVLM (LLAVA-OV) to generate the answer. Experimental results demonstrate cdViews' state-of-the-art performance on the ScanQA and SQA benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **1. “The claim of zero-shot is questionable”**\nSorry for the confusion caused. For our best method LLAVA-OV + $F_{cdViews}$, the term `zero-shot` could be more precisely scoped: we will revise Lines 153–156 to state it as `zero-shot 2D LVLM inference` (rather than fully zero-shot).\n\nWe would like to clarify that our goal in this paper is to make use of 2D LVLMs (which are much better trained than 3D ones) for free, i.e., to adapt them for 3D tasks without any large-scale 3D pre-training or fine-tuning. \nTo this end, we proposed two pure no-training methods denoted as LLAVA-OV + $F_{uniform}$ and + $F_{retrieval}$,  \nand one with small-scale training (on the viewSelector) denoted as + $F_{cdViews}$.\nDuring inference, 3D-QA is conducted by feeding the uniformly-sampled / retrieved / selected 2D views and the question into LLAVA-OV, and LLAVA-OV outputs the answer based solely on its knowledge learned from large-scale 2D pre-training.\n\nIn the paper, we will revise the motivation paragraph of the Introduction section to make these clearer.\n\n---\n### **2. “... which may limit its generalization ...  ”**\nWe address this question from two perspectives: the generalization of our 3D-QA pipeline and the trainable module viewSelector.\n\nFirst, our pipeline leverages 2D LVLMs for 3D tasks without large-scale 3D-language pretraining. It supports three view selection methods: viewSelector, random sampling, and image retrieval. The latter two require no training (assuming minimal domain gap between retrieval models' training data and 3D-QA data). This ensures flexibility and broad generalization of the whole pipeline.\n\nSecond, viewSelector is a lightweight trainable module that easily adapts to another similar task with small-scale 3D training data. Compared to large-scale 3D-language pretraining, fine-tuning viewSelector offers a practical way to enable the generalization of 2D LVLMs to 3D tasks.\n\n---\n### **3. “... the accuracy of the annotated positive views ...”**\nWe agree that using LVLMs alone for annotation may lead to unreliable views.  \n\nTo address this, viewAnnotator is designed to capture informative views beyond simply image matching. Specifically, we incorporate a step-by-step system prompt (shown in [Figure R1](https://anonymous.4open.science/r/icml_rebuttal-35FF/system_prompt_with_caption.pdf)) in the View Matching process. This prompt ensures that all key objects, attributes, and spatial relationships in the caption align with the image, reducing ambiguity and improving consistency. Uncertain views are explicitly excluded, enhancing the robustness of the annotation process. Additional examples of positive and negative views are shown in [Figure R2](https://anonymous.4open.science/r/icml_rebuttal-35FF/positive_negative_views_with_caption.pdf).  \n\nTo further validate the reliability of the positive views, we conducted a human evaluation:  \n-  We randomly selected 50 QA pairs with their associated positive views.\n-  Three human evaluators assessed whether each view could answer the question.\n\nTheir accuracy rates were 96.72%, 94.28%, and 97.56%, confirming that the quality of positive views is sufficient for training. \n\nThese details and results will be included in the supplementary document of the revised version.\n\n---\n### **4. “An ablation study is missing...”**\nAs suggested, we performed ablation comparing the retrieval + viewNMS and cdViews and show the results in Table R1. Combining image retrieval with viewNMS reduces the number of input views due to redundancy removal, but it brings very marginal performance improvement (+0.1%).\n\nThe reason is that viewNMS prioritizes diversity, but the overall performance still depends on whether the selected views contain the necessary visual evidence for the question. This highlights a key difference between image retrieval and viewSelector--while image retrieval focuses on matching text and images, viewSelector is explicitly trained to find views that are most important for reasoning.\n\n| LLAVA-OV + Method                  | Image Retrieval | viewSelector | viewNMS | Best EM@1 | Optimal k |\n|-----------------------------------|-----------------|----------------|-----------|-----------|-------------|\n| $F_{retrieval}$  | ✅              | -              | -         | 29.1      | 17          |\n| $F_{retrieval}$  | ✅              | -              | ✅        | 29.2      | 9           |\n| $F_{cdViews}$  | -               | ✅             | -         | 29.7      | 17          |\n| $F_{cdViews}$  | -               | ✅             | ✅        | **30.1**  | **9**       |\n\n**Table R1:** *Comparison of image retrieval and cdViews with and without viewNMS on the ScanQA validation set. The best EM@1 scores are reported with the corresponding optimal $k$.*\n\n---\n### **5. “Typos: ...”**\nWe thank the reviewer for pointing this out. We will revise it.", "claims_and_evidence": "1. The claim of zero-shot is questionable. The viewSelector requires training on data from existing 3D QA datasets (ScanQA and SQA3D), which may limit its generalization to other 3D QA datasets, particularly when the 3D scenes are not from ScanNet. In such cases, the domain of 2D view images would differ, potentially affecting performance.", "methods_and_evaluation_criteria": "1. The viewAnnotator relies on an LVLM, which does not guarantee the accuracy of the annotated positive views. Further analysis is required to assess its reliability.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "1. An ablation study is missing to evaluate the effectiveness of viewSelector. For instance, comparing \"Image Retrieval\" + viewNMS with cdViews (viewSelector + viewNMS) would help quantify the improvement brought by viewSelector over \"Image Retrieval.\" I believe that the combination of \"Image Retrieval\" and viewNMS could yield comparable performance, and this is a true zero-shot method.", "relation_to_broader_scientific_literature": "This work is related to 3D question answering and zero-shot learning, offering some insights, such as selecting critical 2D views and using pre-trained 2D LVLMs, for future exploration of zero-shot 3D tasks.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "Typos:\nLine 241: \"... the zero-shot 3Q-QA...\" (should it be 3D-QA?)", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kMTNL2jwzO", "reviewer": "Reviewer_pwsQ", "summary": "The paper aims to only use 2D vision-language models to address 3D question answering task. The authors propose a new framework cdViews, which select critical and diverse views and then perform 3D question answering using the 2D vision-language model. The proposed framework is evaluated on two widely used benchmarks and demonstrate the effectiveness.\n\n## update after rebuttal\nPlease see the rebuttal comment below.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **1. “... ensure the quality for the training of viewSelector.”**\n\nWe agree that using LVLMs alone for annotation may lead to unreliable views.\n\nTo address this, viewAnnotator is designed to capture informative views beyond simply image matching. Specifically, we incorporate a step-by-step system prompt (shown in [Figure R1](https://anonymous.4open.science/r/icml_rebuttal-35FF/system_prompt_with_caption.pdf)) in the View Matching process. This prompt ensures that all key objects, attributes, and spatial relationships in the caption align with the image, reducing ambiguity and improving consistency. Uncertain views are explicitly excluded, enhancing the robustness of the annotation process. Additional examples of positive and negative views are shown in [Figure R2](https://anonymous.4open.science/r/icml_rebuttal-35FF/positive_negative_views_with_caption.pdf).\n\nTo further validate the reliability of the positive views, we conducted a human evaluation:\n\n- We randomly selected 50 QA pairs with their associated positive views.\n- Three human evaluators assessed whether each view could answer the question.\n\nTheir accuracy rates were 96.72%, 94.28%, and 97.56%, confirming that the quality of positive views is sufficient for training.\n\nThese details and results will be included in the supplementary document of the revised version.\n\n---\n### **2. “Equation (3) is confusing ...”**\nWe appreciate the reviewer's detailed comments. In the original Equation (3), the number of sampled views $k$ was not clearly represented on the right-hand side. We will modify the Equation (3) as follows: \n$$\nF_{uniform}(M, k) = \\{V_{i_j}\\}_{j=1}^k, i_j \\sim {Uniform}(1, N).\n$$\n\n---\n### **3. “ ... Duplicated definition of $S_i$...”**\nBoth instances of $S_i$ on Line 289 (left column) should be updated to $\\hat{S}_i$ for consistency with Equation (7). In our notation, $S_i$ denotes the ground-truth label, and $\\hat{S}_i$ represents the predicted score from the model. The expressions on Line 289 correspond to model outputs, so the appropriate notation is $\\hat{S}_i$.\n\n---\n### **4. “... directly add these two distances ...”**\nYes, the two distances are on comparable scales. Based on statistics from the ScanQA and SQA datasets, position distance ranges from $[0, 14.72)$, and the orientation distance is within $[0, \\pi)$. Given their similar scales, they can be combined.\n\nWe tested different combination weights on the validation set (Table R1). The results show that viewNMS is fairly robust to weighting variations, with equal weighting (i.e., direct summation) yielding the best performance.\n\n| LLAVA-OV | $D_{pos} \\times$ | $D_{ori} \\times$ | Threshold $T$ | EM@1       |\n|---------------------------------------------|------------------|------------------|----------------|------------|\n| + $F_{cdViews}$                                          | 0.5              | 1                | 0.5            | 29.7       |\n| + $F_{cdViews}$                                         | 1                | 1                | 0.5            | **30.1**   |\n| + $F_{cdViews}$                                          | 1                | 0.5              | 0.5            | 29.8       |\n\n**Table R1:** *Ablation study on distance weighting in viewNMS. We vary the relative weights of position distance ($D_{pos}$) and orientation distance ($D_{ori}$) while keeping the threshold fixed at 0.5. Equal weighting (1:1) achieves the best EM@1 performance.*\n\n---\n### **5. “... viewNMS Thresholds part ...”**\nWe clarify that viewNMS does not require searching over parameter pairs. As noted in our response to the previous question (regarding Equation (12)), the position and orientation distances are directly combined with equal weights, so only a single threshold needs to be selected.\n\nThis threshold is chosen based on the validation performance. As shown in the line chart in Figure 7 of the manuscript, a threshold of 0.5 yields the best result.\n\n---\n### **6. “... the backbone matters a lot ...”**\nOur 3D-QA pipeline consists of three main steps: view selection, feeding the selected view into a 2D LVLM, and generating an answer from the LVLM. Our main contribution is the whole pipeline of using 2D LVLM for 3D-QA. Our add-on contribution is proposing cdViews for more efficient view selection in the first step.\n\nSince most network parameters (in our pipeline) reside in the 2D LVLM, the final performance heavily depends on its pre-trained knowledge. Compared to LLaVA-Next and Bridge-3D, LLaVA-OV performs better due to its improved vision-language alignment—enhancing image understanding and generating more accurate, detailed descriptions—achieved through stronger language modeling and high-quality instruction tuning on open-vocabulary tasks.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "The methods and evaluation are reasonable.", "theoretical_claims": "No theoretical claims and proofs involved.", "experimental_designs_or_analyses": "The experimental designs and analyses are reasonable.", "relation_to_broader_scientific_literature": "This paper extends the field of 3D question answering by introducing a pipeline which only uses 2D vision language models. The proposed cdViews pipeline is could also be applied to other downstream 3D understanding tasks.", "essential_references_not_discussed": "The reviewer is not aware of missing references.", "other_strengths_and_weaknesses": "Strength:\n1. The paper is well-written, with clear structure and illustrations.\n\nWeakness:\n1. The labels from the viewAnnotator come from large vision language models, which can not ensure the quality for the training of viewSelector.", "other_comments_or_suggestions": "1. Equation (3) is confusing, as the parameter k is not shown in the righthand side.\n2. $S_i$ appears in line 289 - left column is duplicated as $S_i$ in equation 6.", "questions_for_authors": "1. How do you evaluate/ensure the quality of $S_i$ generated by the LVLM (equation 6)?\n2. Does $S_i$ appears in line 289 - left column means the $\\hat{S_i}$ in equation 7? Duplicated definition of $S_i$ is confusing.\n3. For equation (12), why do you directly add these two distances together? Are they within the similar scale? Have you conducted experiments to test whether the ratio between these two would affect the performance?\n4. The conclusion for viewNMS Thresholds part (line 433-left column) is unclear. Search for parameter pairs is time-consuming. Is there any insight for how to choose appropriate thresholds if we are going to apply this pipeline on new datasets?\n5. Table S3 indicates the backbone matters a lot in such pipeline. Why llava-OV is doing much better? When using llava-next the performance is worse than bridge-3d.\n\nThe reviewer may adjust the final rating after rebuttal based on the clarifications from the authors.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vy839okjTV", "reviewer": "Reviewer_PgGq", "summary": "In this work the authors proposed to solve 3D question answering with 2D VLMs only. Specifically, 3D scenes are first rendered into 2D images, which are then used to prompt 2D VLMs (e.g., LLaVA-OneVision). Moreover the authors found that the key to good performance is how to select views that are most relevant to the question and answer. Compared to baseline view selectors, such as uniform sampling or image retrieval, training a view selector (along with a view NMS algorithm) to choose views that maximize the performance achieves the best performance.\n\n## update after rebuttal\n\nFollowing the rebuttal I raised some concerns regarding the contribution of this paper. Although I was looking forward to a meaningful discussion, reviewer pwsQ refused to defend his/her \"4: Accept\" decision or to engage in further discussion. I stand with reviewer M1zQ and recommend rejection.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **1. \"... the unfair comparison ...\"**\nWe agree that LLaVA-OV is a strong 2D LVLM. However, we do not consider the comparison unfair, as our core motivation and contribution lie in exploring how to leverage powerful 2D LVLMs in a zero-shot manner for 3D-QA.\n\n- Compared to 3D-based methods, current 3D-LLMs are limited by the scarcity of large-scale 3D-language data. This **data bottleneck** is exactly what we aim to address. By contrast, our method directly utilizes pretrained 2D LVLMs, which benefit from tons of 2D vision-language data.\n\n- Compared to 2D-3D hybrid methods, it may seem fairer to replace the 2D module with LLaVA-OV. However, these hybrid pipelines typically require additional training to align 2D and 3D features. For large 2D LVLMs like LLaVA-OV, this alignment process incurs substantial computational cost. This introduces a significant **resource bottleneck**.\n\n- Our method is designed as **a practical solution under both data and resource constraints**, showing that strong off-the-shelf 2D LVLMs can be effectively used for 3D-QA. Moreover, the framework is model-agnostic and generalizable to future 2D LVLMs with improved capabilities.\n\n---\n### **2. \"... finetuning LLaVA-OV ...\"**\nThe reviewer suggests fine-tuning LLaVA-OV within a 2D–3D hybrid pipeline, which could potentially benefit from both strong language–vision alignment and 3D spatial context. However, this setting introduces fundamental limitations: combining 2D LVLMs with 3D inputs requires additional model components, training supervision, and modality alignment, making the pipeline heavier and less scalable. As discussed in Lines 68–74 of our Introduction,\n\"2D features extracted from LVLMs are already well-aligned with language, but further alignment with 3D features requires careful model design and advanced training techniques. \"\n\nAs requested, we implement a hybrid variant based on BridgeQA, the strongest hybrid baseline in our comparisons. Specifically, we follow the BridgeQA architecture and replace its 2D LVLM with LLaVA-OV. The input views are combined with question and 3D features. For fair comparsion, we use 9 views selected by cdViews as the input. Results are reported in Table R1.\n\n| Method                              | Type   | 2D LVLM   | EM@1  |\n|-------------------------------------|--------|-----------|-------|\n| BridgeQA           | 3D+2D  | BLIP      | 27.0  |\n| BridgeQA$_{LLAVA-OV}$               | 3D+2D  | LLAVA-OV  | 28.4  |\n| LLAVA-OV + $F_{cdViews}$    | 2D     | LLAVA-OV  | **30.1** |\n\n**Table R1**: *Evaluating fine-tuned LLaVA-OV in a 2D–3D hybrid pipeline.* \n\n---\n### **3. \"... computation efficiency comparison ...\"**\n- **Number of candidate views**: Each 3D scene contains between 6 and 382 RGB views, with an average of 79 views per question.\n\n- **Computation during view selection**: All candidate views are processed by the view selection module. However, this stage contributes only a small fraction of the total cost—less than 10% of overall FLOPs—as shown in Table R2.\n\n- **Comparison with image retrieval**:\nIn the view selection stage, our method reduces FLOPs by 54.8% (26.5T vs. 58.6T) and runtime by 90% (0.04s vs. 0.40s).\nIn the QA stage, FLOPs are reduced by 49.9% (268T vs. 535T), and inference time by 51.3% (1.16s vs. 2.38s).\n\n- **GPU memory usage**:\nAs shown in Table R2, cdViews achieves a lower peak GPU memory cost, making it more suitable for deployment on devices with limited memory capacity.\n\n|     Method      |       **FLOPs**        |        |   **Inference Time**    |        |   **Peak GPU Memory Usage**    |        |  \n|:---------------:|:----------------------:|:------:|:------------------------:|:------:|:------------------------:|:------:|\n|                 | View Selection         |  QA    | View Selection           |  QA    |View Selection           |  QA    |\n| $F_{uniform}$   | 0                      | 535T   | 0                        | 2.38s  |   0   |     26.96G |\n| $F_{retrieval}$ | 0.74T/view × 79 = 58.6T| 535T   | 0.40s                    | 2.38s  |   7.89G  |   26.96G |\n| $F_{cdViews}$   | 0.34T/view × 79 = 26.5T| 268T   | 0.04s                    | 1.16s  |   9.56G   | 21.54G  |\n\n**Table R2**: *Efficiency comparison between baseline method and our cdViews.*.\n\n---\n### **4. Ablation on the number of selected views and computational cost**\nWe provide the requested ablation on computational cost with respect to the number of selected views. As shown below, FLOPs grow approximately linearly from 268 TFLOPs (9 views) to 534.98 TFLOPs (17 views). For the performance impact of view number, please kindly refer to Figure 4 of the main paper.\n| # Views | FLOPs (TFLOPs) |\n|--------:|----------------|\n|    9    | 268.00         |\n|   10    | 299.89         |\n|   11    | 332.19         |\n|   12    | 364.95         |\n|   13    | 398.12         |\n|   14    | 431.69         |\n|   15    | 465.71         |\n|   16    | 500.14         |\n|   17    | 534.98         |", "claims_and_evidence": "The major finding of this paper is that 2D VLMs can achieve state-of-the-art 3D VQA performance, when compared with other 3D-LLMs. This argument is not convincing due to the unfair comparison adopted in the experiment section (see next question). Other smaller findings and arguments are mostly supported with clear evidence.", "methods_and_evaluation_criteria": "The proposed method and evaluation criteria is sound. However, the quantitative results may not be based on fair comparison. The VLM used in the proposed method, i.e., LLaVA-OV, is significantly better than all other 3D-LLMs, in terms of both the supervised fine-tuning (SFT) data, the vision encoder, and the language model (LLaMA-3 vs Vicuna v1). The LLaVA-OV naturally has much stronger visual understanding and reasoning abilities, which presents an unfair comparison with baseline methods. Experimental results in appendix also shows that a weaker VLM (LLaVA-NeXT) significantly hurts the performance.", "theoretical_claims": "No theoretical claims presented.", "experimental_designs_or_analyses": "Again I have concerns regarding the experimental designs. See \"Methods And Evaluation Criteria\" part of my review.", "relation_to_broader_scientific_literature": "This work related to the broader discussion of spatial intelligence -- how to develop artificial intelligence with spatial understanding and reasoning, specifically the choice of the input modality and the design of the model (2D or 3D VLM). However, given the concerns regarding the experimental design, the main findings of this paper may not stand. The contribution of this paper to the broader research topic is limited.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "In general this work discusses the choices of 2D and 3D VLM to achieve strong 3D spatial understanding and reasoning, given the limited 3D-related data and abundant 2D multi-modal data. This is an interesting and important question. However I have doubts about the experimental settings of this paper -- an unfair comparisons could produce very misleading results.", "other_comments_or_suggestions": "My major concern is regarding the unfair comparison in the experimental results. This is mainly due to the fact that the authors are not retraining any of the 2D or 3D VLMs. For instance, finetuning LLaVA-OV following a strong 3D-VLM baseline could provide important insights to this problem. Most importantly the comparisons should be conducted with comparable vision encoders and large language models.", "questions_for_authors": "1. How many input views are sampled, filtered, and fed into the large language model?\n2. All frames, selected or not, go through the visual encoder. This may cause significant computational costs. What is the GFLOP of the proposed methods compared to baseline methods? What is the wall clock time of inference compared to baseline methods?\n3. Also there should be ablation studies on the number of frames rendered and selected, e.g., the final performance v.s. the computational costs.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LmINtDn4Zq", "reviewer": "Reviewer_M1zQ", "summary": "This paper proposes to address the task of 3D question-answering (3D-QA). Proposed method takes as input a set of posed RGB images and operates by only using 2D large vision language models (in this case LLAVA-OV). The method does not operate on any 3D input, and instead uses the set of available images to answer questions in 3D-QA benchmarks. As the method only relies on images, the key challenge is to select informative and diverse views that correspond to the task and more helpful for answering the question being asked about the scene. To address this, the method proposes a view-selection module named cdViews, consisting of two sub-parts: (1) viewSelect is a module trained using annotations from 3D-QA datasets to identify most critical and relevant views for answering the question, and (2) viewNMS module essentially measures pairwise view overlaps to identify and select diverse views. The proposed method is evaluated on the ScanQA and SQA datasets.\n\n### Update after rebuttal\nI appreciate the authors’ rebuttal and the clarifications provided. After carefully reading the response and considering the other reviews and discussions, I agree that the main contribution of the paper lies in demonstrating that 2D VQA models can outperform 3D-based models on current 3D-QA benchmarks. This is indeed a noteworthy and interesting finding, suggesting that strong performance on these benchmarks is possible by only relying on large vision-language models (LVLMs). But this finding does not yet persuade me that LVLMs are necessarily stronger for the 3D-QA task in the general sense as claimed in this paper. This could even be instead pointing to the limitations of the existing 3D benchmarks in terms of evaluating spatial relations. \n\nOverall, while the paper makes an interesting point, I believe the analysis of the results could be strengthened, especially in terms of exploring the role and limitations of 3D reasoning in the presented approach. As noted in my initial review, I had raised several questions related to this, such as whether LVLM-based methods are truly expected to model spatial relationships in 3D scenes. While the rebuttal addressed some of these concerns, I find that some of my reservations remain, particularly with respect to the 3D spatial reasoning ability of the method and the method limitations. In the light of this, I will be keeping my original score (3).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **1. \"... finetuning LLaVA-OV ...\"**\nDue to space limitation, please kindly refer to the response of PgGq `2`\n\n---\n### **2. \"... analysis of the types of questions\" & \"a discussion on the limitations ...\"**\nWe conduct a detailed analysis of question types on the SQA dataset, with per-type performance reported in Table S2 (Supplement). The results show that our zero-shot 2D LVLM-based method achieves strong performance across most question types, particularly excelling in “what”-, “which”-, and “other”-type questions.\n\nThese categories primarily involve identifying objects and reasoning about their spatial configuration within the 3D scene. Our method, powered by multi-view aggregation, can aggregate multi-view observations to infer spatial layouts and object relations in 3D environments, even when key objects are never observed within a single view. As shown in [Figure R3](https://anonymous.4open.science/r/icml_rebuttal-35FF/failure_case_with_caption.pdf) (1st and 2nd sample), the model correctly infers relationships between spatially separated entities across the room.\n\n**Limitation**: In contrast, we observe relatively lower performance on “can”-type questions, which often involve agent-centric reasoning and require understanding how the scene would change with respect to the agent’s actions or viewpoint shifts (e.g., “Can I see the faucet if I turn around?”). These cases demand the modeling of dynamic viewpoints and agent-scene interaction, which are not fully captured by current 2D LVLMs in a frozen, zero-shot setting.\n\nOverall, our analysis confirms that frozen 2D LVLMs, when equipped with informed view selection, can already support a broad range of 3D spatial reasoning tasks. \n\n---\n### **3. \"... broader spatial relations ...\"**\nOur method is capable of reasoning about spatial relationships between objects that are never observed in the same view. As shown in the second example of [Figure R3](https://anonymous.4open.science/r/icml_rebuttal-35FF/failure_case_with_caption.pdf), the printer, trash can, and paper cutter are distributed across different views with no co-observability. Despite this, the model accurately infers their spatial arrangement and correctly answers the question.\n\nThis demonstrates that, even without explicit 3D reconstruction, our approach can integrate multi-view information to support global spatial reasoning. \n\n---\n### **4. \"Strong claims\"**\n- [L153-156]: change from \"3D understanding tasks\" to \"3D-QA tasks\"\n- [L155-157]: correct to \"... as a critical factor in zero-shot 3D-QA, for which there is a lack of an efficient solution in prior works.\"\n- [L125-127, right column]: correct to \"... but rely on explicit 3D reconstruction, needing additional models and causing more processing steps. In contrast, our method uses 2D views and feeds them into a unified 2D LVLM, which makes a simpler pipeline\".\n- [L142-144]: correct to \"They focus more on evaluating pretrained 2D LVLMs on 3D-QA tasks, rather than developing approaches to adapt and improve their performance for spatial reasoning.\"\n\n---\n### **5. \"... cdView component contributes to the overall performance ...\"**\nWe agree that LLaVA-OV provides a strong foundation, and even the uniform view sampling yields competitive results. This actually motivated us to explore more in the direction of leveraging well-trained 2D LVLMs for tackling 3D-QA tasks, and then we proposed the light-weight learnable module cdViews.\n\nAs we can see in Table 1 (manuscript), using cdViews brings consistent and clear performance improvements over uniform sampling across multiple benchmarks. Specifically, it yields +2.0% / +2.1% EM@1 and +7.0% / +7.1% CIDEr gains on the ScanQA test set (w/o objects), and +3.4% EM@1 improvement on SQA. These gains can be regarded as \"substantial\", especially considering that prior work typically improve by only 1% EM@1 or 3–4% CIDEr (see Table 1).\n\n---\n### **6. \"claims zero-shot ...\"**\nDue to space limitation, please kindly refer to the response of rVhy `1`\n\n---\n### **7. “Problem formulation”**\n[Lines 162–164]: “..., each associated with a camera matrix containing the position and orientation.”\n\n---\n### **8. \"L323, L300, L359\"**\n We thank the reviewer for pointing these typos out. We will fix them in the revised version.\n\n---\n### **9. \"Differ from video-VQA\"**\nTraditional video-based VQA methods, while also using 2D information, are designed to process sequential frames to understand temporal dynamics.  In contrast, cdViews focuses on selecting the most informative static 2D views from a 3D scene to answer questions about the 3D space.\n\n---\n### **10. \"Computation cost and memory cost\"**\nDue to space limitation, please kindly refer to the response of PgGq `3`", "claims_and_evidence": "- The submission claims to address 3D-QA, yet the method relies solely on 2D images—with the only 3D aspect being the viewNMS module that uses camera poses to select diverse views if I understood correctly. This raises questions about whether the approach truly tackles 3D-QA, which typically requires a full 3D scene representation to capture spatial relationships. Moreover, since 2D images often suffice for answering questions about nearby object pairs, it remains unclear if the method robustly handles _spatial_ reasoning. I indeed find it impressive to see that this 2D-only method can perform much better than 3D-based methods, which is an interesting finding of this submission. But I am not yet fully convinced that these gains can be attributed to the method proposed in this work, for instance if one combines the baselines for 3D/2D hybrid methods with LLAVA-OV, would we see better results as they have access to 3D reconstruction as well (and can attend to the full 3D scene)? Also, further analysis of the types of questions answered well (and those not) by the proposed method could clarify whether the method truly understands broader spatial relations, such as identifying objects facing each other across a room and never visible together in a single image (e.g., \"what is located right across the window\" where the answer could be a TV far away).\n- [L153-156] “We are the first to leverage 2D LVLMs in a zero-shot manner to address 3D understanding tasks.” - This claim is partially incorrect. Many prior works on open-vocabulary 3D scene understanding, such as OpenScene, OpenMask3D, and Open3DIS, already rely on 2D LVLMs to extract knowledge about a 3D scene in a zero-shot manner. While the claim might hold for this specific 3D-QA setting and the LVLMs such as LLAVA (instead of CLIP-style VLMs), it should be narrowed in scope to avoid misrepresentation.\n- [L155-157]: \"We identify view selection as a critical factor in this zero-shot setting, a challenge that has not been explicitly addressed.\" – Again, I think that the scope of this statement should be narrowed as there are many open-vocabulary 3D scene understanding methods where object-related view selection has been explored.  \n- [L125-127, right column]: \"These methods improve 3D-QA performance, they come with trade-offs such as increased model complexity and data processing requirements. Furthermore, reconstructing 3D scenes from 2D views incurs significant computational costs, limiting the scalability in complex scenes.\" – While this argument is valid in terms of the costs associated with reconstruction, it does not consider the costs associated with running LVLMs multiple times on multiple views, which is the case in the proposed method.\n- [L142-144]: \"They (2D-based methods) care more about the evaluation itself rather than explicitly exploring new methods to improve 2D LVLMs performance in 3D-QA.\" – This statement is vague and should be better substantiated.", "methods_and_evaluation_criteria": "Proposed method is reasonable, and the evaluation methodology follow the established benchmarks for the task of 3D-QA.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "I think the experimental designs and analyses are generally sound, I did not identify a critical issue.", "relation_to_broader_scientific_literature": "The paper provides a comprehensive discussion of related works, and I appreciated the systematic discussion with 3D, 2D as well as 3D/2D hybrid methods. However, the paper should be more careful in making strong claims about being the first to leverage 2D LVLMs for 3D tasks ([L153-156]) in a zero shot manner, as prior works in open-vocabulary 3D segmentation (OpenScene, OpenMask3D, Open3DIS etc.) have followed similar zero-shot approaches. This could also be a language specificity issue, as these open-vocabulary methods largely use LVLMs such as CLIP, whereas the proposed method focuses on methods like LLAVA with conversational abilities. I think this should be made more clear in the discussion in order to more accurately position the paper.", "essential_references_not_discussed": "The paper provides a comprehensive discussion of related works. However, it should be more careful in making strong claims about being the first to leverage 2D LVLMs for 3D tasks ([L153-156]), as prior works in open-vocabulary 3D segmentation have done something similar. Additionally, the discussion of view selection could reference prior multi-view reasoning works.", "other_strengths_and_weaknesses": "_Strengths:_\n- It is a well-written paper with very clear explanations. \n- The proposed methodology has an interesting take on the 3D-QA task, showing that even without having an explicit 3D scene representation it is possible to reason about the scene well compared to the purely 3D or hybrid 3D-2D methods for 3D-QA task. The findings are convincing, and I think the experimental analysis sheds light on the benefit of leveraging 2D LVLMs for 3D scene understanding tasks.\n- I appreciated the comprehensive discussion on related works.\n- The evaluation methodology is reasonable, and I also appreciated that the code is provided.\n\n_Weaknesses:_\n- I was unable to see a discussion on the limitations, which is especially important given that the proposed method does not make use of a 3D scene representation of the scene while answering the questions related to the 3D scene. \n- As noted in the Claims and Evidence section, there are some open questions about the method’s 3D capabilities, especially its ability to capture relationships between objects that do not appear together in a single RGB frame. While this 2D-based approach shows promising performance, this might also reflect some limitations in the current evaluation benchmarks. In my opinion it is not substantially evaluated or demonstrated whether the method can capture broader-range spatial relationships.\n- Given the method's reliance on LLAVA-OV, it is not entirely clear to me whether the cdView component substantially contributes to the overall performance, considering that even a uniform view sampling method outperforms several 3D-based baselines.\n- Some claims such as the novelty of using 2D VLMs for 3D understanding in a zero-shot setting are a bit too strong and could benefit from  phrasing with more care.", "other_comments_or_suggestions": "_Minor comments:_\n\n- The problem formulation in [L162-164] defines the scene representation as a set of 2D views but does not explicitly include camera poses, which are necessary for the viewNMS module. This should be clarified.\n- L323: “.. indicates more criticality of V_i.” should be “higher criticality”.\n- L300 (right column): “omit the subscription for simplify” - “we omit the subscript for simplicity”\n- L359 (right column): “5.1 Comparisons with the State-of-the-Arts” should be “State-of-the-Art”", "questions_for_authors": "I am happy to reconsider my score based on the answers to the following questions and the remarks discussed earlier in the Claims section:\n\n1. How does the proposed method fundamentally differ from video-based VQA approaches, given that it does not use an explicit 3D representation? Can it reliably reason about object relationships when two objects are never observed together in a single view?\n2. How does the computational cost of running LLAVA-OV multiple times on multiple views compare to the cost of performing 3D-QA using existing 3D-based or 3D/2D-based methods? Is the reduction in computational cost by not having to reconstruct a 3D scene comparable to the difference between 3D-QA inference runtimes for the proposed method and 3D-based methods? Also, it appears to me that the proposed method needs to store all RGB images at all times if I understand correctly. How do the memory requirements compare? \n3. The paper does not explicitly discuss its limitations as far as I could see. What are the main failure cases where the method struggles, particularly in complex or large-scale scenes where a broader-range 3D understanding might be necessary?\n4. If LLAVA-OV were integrated into existing 3D/2D hybrid methods, would we expect it to outperform the proposed approach? Did the authors explore this possibility?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Almost all reviewers agree on this paper (but not the rating).

- The question: "Do 2D VLMs outperform 3D VLMs in spatial reasoning" seems intriguing. 
- The evidence provided in the paper seems incomplete.

The AC agrees with this assessment. Given the additional evidence provided in the rebuttal, the paper seems above bar. While it is not the final word in the discussion weather to 3D or not to 3D, it does provide value to the community.
The AC recommends the authors to include the reviewers feedback in the final version of the paper.
Decision: Accept (poster)