3D-LLM: Injecting the 3D World into Large Language Models
Authors: Yining Hong
Haoyu Zhen
Peihao Chen
Shuhong Zheng
Yilun Du
Zhenfang Chen
Chuang Gan
Venue: NeurIPS 2023 spotlight (2023)
Keywords: 3D Visual Reasoning
3D Large Language Model
TL;DR: We introduce 3D-LLMs, new foudation models for 3D reasoning.

Abstract: Large language models (LLMs) and Vision-Language Models (VLMs) have been proved to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models, and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D
grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi-view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs could better capture 3D spatial information.  Experiments on ScanQA  show that our model outperforms state-of-the-art baselines by a large margin (\textit{e.g.}, the BLEU-1 score surpasses state-of-the-art score by 9\%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs. Our model and data will be publicly available.

3D-LLM: Injecting the 3D World into Large Language Models
Yining Hong University of California, Los Angeles
Haoyu Zhen
Shanghai Jiao Tong University
Peihao Chen
Shuhong Zheng
South China University of Technology
University of Illinois Libana-Champarn
Yilun Du Massachusetts Institute of Technology
Zhenfang Chen MIT-IBM Watson Al Lab
Chuang Gan
UMass Amherst and MIT-IBM Watson Al Lab
Abstract
Large language models (LLMs) and Vision-Language Models (VLMs) have been proven excel multiple tasks, such commonsense reasoning. Powerful these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics. layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMS can take 3D point clouds and their features as input and perform diverse set of 3D-related tasks, including captioning, dense captioning. 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and on. Using three types of prompting mechanisms that we design, we are able to collect over IM 3D-Janguage data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi-view images. Then, use 2D VLMs 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on held-out evaluation dataset, ScanQA. SQA3D and 3DMV-VQA. outperform state-of-the-an baselines. In particular, experiments on ScanQA show that our model outperforms state-of-the-- baselines by large margin (e.g., the BLEU-1 score surpasses state-of-tt of-t-o--the-arr score by 9%). Furthermore, experiments held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMS and VLMs. Project Page: https: h //visvviiiiia www. CS umass edu/ adtu/ddlm//.
backbones to train 3D-LLMs. By introducing a
Introduction
In the past several years, have witnessed surge of large language models (LLMs)) (e-g., GPT4 (33]) that excel at multiple tasks, such as communication and commonsense reasoning. Recent works have explored aligning images and videos with LLM for new generation of multi-modal LLMs (e.g.. Flamingo [15], BLIP-2 [29]) that equip LLMs with the ability to understand and reason about 2D images. However, powerful as the models can be in communication and reasoning,
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

at the blue bad. dressed. 3D Captioning
3D Grounding
3D Question Answering
Input Prompt
This and blue leather chair. is placed on right of identical leather chairs.
Describe the scone.
Question: this famous building?
Response
A small Asian with japanese architectural style.
well-known.
Task Decomposition
3D-Assisted Dialog
3D Dense Captioning
Input Prompt
Can the light?
object.
Response
Go to the wardrobe and open clothes.
Sum Whitaasel
room.
walliwns
brewn pillows on it. It has tables
right and exit the room.
have
lamp standing on the table. On the right backpack.
4. Walk to the bathroom.
Facing the mirror and dress.
Object Navigation
Vision-Language Navigation
Embodled QA
Input Prompt
find
Instruction: the sofs and am (1). Where should go next?
am at (11. I want to know the color
to (2). Where should go noxt?
stove?
Response
You should go to (2) where may
Figure 1: Examples from our generated 3D-language data, which covers multiple 3D-related tasks.
they are not grounded in the real 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics and interaction on. Therefore, such LLMs pale in comparison with the robots depicted sci-fi movies the assistants that could understand the 3D environments, well perform reasoning and planning based the 3D understandings.
To this end, we propose to inject the 3D world into large language models, and introduce whole new family of 3D-LLMs that could take 3D representations (i.e., 3D point clouds with their features) as input, and perform series of 3D-related tasks. By taking the 3D representations of scenes as input, L.LMs are blessed with twofold advantages: (1) long-term memories about the entire scene can be stored in the holistic 3D representations, instead of episodic partial-view observations. (2) 3D properties such as affordances and spatial relationships beyond the scope of language-basee or 2D image-based LLMs.
be reasoned from 3D representations, far
One major challenge of training the proposed 3D-LLMs lies in data acquisition. Unlike the vast amount of paired 2D-images-and--es data on the Internet, the scarcity of 3D data hinders the development of 3D-based foundation models. 3D data paired with language descriptions are even harder to obtain. To address this, we propose a set unique data generation pipelines that could generate large-scale 3D data paired with language. Specifically, devise three efficient prompting procedures for communication between data and language. In this way, are able to obtain approximately one million 3D-language data covering diverse set of
make use of ChatGPT [33] and

---

tasks, including but not limited to 3D captioning, dense captioning, 3D question answering, 3D task decomposition, 3D grounding, 3D-assisted dialog, navigation and so on, as shown in Figure 1.
The next challenge resides in how to obtain meaningful 3D features that could align with language features for 3DD-LLMS. One way is to train 3D) encoders from scratch using a similar contrastive- learning paradigm for the alignment between 2D images and language (e.g., CLIP [36]). However, this paradigm consumes tremendous data, time, and GPU resources. From another perspective, there are numerous recent works that build 3D features from 2D multi-view images (e.g. concept fusion [24], 3D-CLR [20]). Inspired by this, we also utilize a 3D feature extractor that constructs 3D features from the 2D pretrained features of rendered multi-view images. Recently, there are also quite few visual-language models (e.g.. BLIP-2 [29]. Flamingo [15]) utilizing the 2D pretrained CLIP features for training their VLMs. Since our extracted 3D features are mapped to the same feature space as 2D pretrained features, we can seamlessly use 2D VLMS as our backbones and input the 3D features for the efficient training of 3D-LLMs.
One crucial aspect of 3D-LLMs, different from vanilla LLMs and 2D VLMs, is that 3D-LLMs are expected to have an underlying 3D spatial sense of information. Thus, we develop a 3D localization mechanism that bridges the gap between language and spatial locations. Specifically, we append 3D position embeddings to the extracted 3D features to better encode spatial information. In addition, we append series of location tokens to the 3D-LLMs, and localization can be trained via outputting location tokens given the language descriptions of specific objects in the scenes. In this way, 3D-LLMs could better capture 3D spatial information.
To sum up, our paper has the following contributions:
We introduce a new family of 3D-based Large Language models (3D-LLMs) that can take 3D points with features and language prompts as input, and perform a variety of 3D>rellted tasks. We focus on tasks beyond the scope of vanilla L.LMs or 2D-LLMMs, such as tasks about holistic scene understanding, 3D spatial relationships, affordances and 3D planning.
We devise novel data collection pipelines that could generate large-scale 3D-language data. Based on the pipelines, we collect a dataset that has over IM 3D-lanuagg data that cover a diverse set of 3D-related tasks, including but not limited to 3D captioning. dense captioning, 3D question answering, task decomposition, 3D grounding, 31D-assisted dialog, navigation, and so on.
We use a 3D feature extractor that extracts meaningful 3D features from rendered multi-view images. We utilize 2D pretrained VLMs as our backbones for efficient training. We introduce a 3D localization mechanism for training the 3D-LLMs to better capture 3D spatial information.
Experiments on held-out evaluation dataset, ScanQA, SQA3D and 3DMV- VQQ, outperform state- of-the-art baselines, In particular, 3D LLMs outperform baselines by a large margin on ScanQA (e.g.. 9% for BLEU-1 and 10% for CIDER). Experiments on held-in datasets for 3D captioning, task composition, and DDsssiited dialogue show that our model outperforms 2D VLMs. Qualitative studies further demonstrate that our model is able to handle diverse set of tasks.
We release our 3D-LLMs, the 3D-language dataset, and Kanguage-aligner 3D features of the dataset for future research development !
2 Related Works
Large Language Models. Our work is closely related to large language models [4, 14, 37, 10, 34] (LLMs) like GPT-3 [4] and PaLM [10], which are able to handle different language tasks with single model and show strong generalization abilities. These models are typically trained on massive textual data with superviued training targets like predicting the next tokens [4, 37] or reconstrueting the masked tokens [14, 38]. To better align these LLMs' predictions to human instructions, improve the models' generalization abilities on unseen tasks, a series of instruction tuning methods [35, 42] and datasets [11, 13] have been proposed. In this work, we aim to inject the 3D world into large language models, understanding rich 3D concepts such as spatial relations, affordances, and physics.
Vision-Language Pre-trained Models. Our work is also related to vision-language pre-trained models that connect images and natural language [30, 31, 18, 36, 25J. Some research [36, 25] learn to train models from scratch with massive image-language pairs and apply them to downstream tasks like visual question answering [19, 47], captioning [7], and referring expression comprehension [46] with finetuning. Other researchers have connected pre-trained vision models and pre-trained LLMs
bhttps//gituubb.om/MMMaa Moot:iiiiiccccooo/massoooosotooo Sowoddattioo-Mdd///---LL
3

---

Box- Demoonstrationnnssrrcttoo based Prompting
Bounding Box Context:
Room 1: table: [0.4, 0.1, 0.1], chair: (0.4, 0.3, 0.4]... Room2: desk: (0.5, 0.2, 0.3)
Instruction (Shortened Version):
You are an Al visual assistant / conversation generator question generator in the room.
You need to generate highilevel task its low aevlevee actions/ 4-10 round conversations between a human and a robot assistant / question answer pains about the scone.
Demonstration: For Room 1:
<Takk Decomposition High-Level Task: make Low-Level Actions: go the cabinet 2. take out cosmetics from the cabinet 3. bring the cosmetics to the table with mirror open the lamp make up
toathbrush. Robot: Sure! Where is it?. Human: Maybe in the sink. Robot: have arrived mat the sink and find no toothbrush. I found an closed bathroom cabinet. it possible to contain what you need?...
<3D -ssisisseed Dialog? Human:
Question:
table
the Answer: Generate similar responses for Room
Response: For Room 2:
Wash hands sink cabine. 3. Gog to the mirror 4, Brush teeth and wash face 5. Turn on the lamp 6. Leave the room through the door
Decomposition> High-Level Task: ready for work. Low-Level Actions: go
<3D-Assisted Dialog> Human:
on light.
Sure! Which light?, Human: The lamp. Robot: have
on, is bright enough? Human: No. Are there other lights? Robot: see desk lamp.
<0AA> Question: Viewing the tv from the desk, there gultar on the right? Answer
ChatCaptioner based Prompting
Images from different views
ChatGPT
Captioner
Answerer
can you see from the right?
Chair, desk
Can you tell me what's on the desk?
laptop
Aything else to the desk?
curtain
Here's 's description the scene: This room has work stations with desk and a chair. The desk has a laptop on and curtain besides. also has blue bed, blue sofa and kitchen counter with chairs close
Response:
What can you see from the near left?
bed
What's the color of the bed?
Blue
Revision based Prompting
Context: The white chair is near the table.
Instruction: Generate question answering pairs based on the caption.
Response: Question: What color the chair near the table?
Answer: The chair the table is white.
Figure 2: 3D-language data generation pipelines.
with additional learnable neural modules like perceiver [2] and QFormers [30], leveraging perception abilities in pre-trained vision models, and reasoning and generalization capacities in LLMs. Inspired by these previous works, we plan to build an Al assistant that could understand the 3D world and perform corresponding 3D reasoning and planning. This is not trivial and we need to overcome obstacles like how to handle the problem of data sparsity, how to align the 3D world with 2D images, and how to capture 3D spatial information.
3D & Language. Another line of research that is similar to ours 3D and language [5, 45, 20, 1, 16, 22, 45, 31. ScanQA [45] requires a model to answer questions related to the 3D world; ScanRefer [5] asks a model localize region that the text expression refer to; 3D captioning [8] tests models' abilities to generate captions describing the 3D scenes. However, these 3D tasks and their corresponding models are usually task-specific and could only handle cases within the same distribution of the training sets without generalization. Different from them, we aim to build a 3D model that could handle different tasks at the same time and enable new abilitics like DDaassistant dialog and task decomposition.
3 3D-Language Data Generation
The community has witnessed the proliferation of multi-modal data thanks to easy access to tremendous amount of 2D image and text pairs on the internet. However, when it comes to 3D-related data, obtaining multimodal resource is not easy, due to not only the scarcity of 3D assets, but also the difficulty of providing language data for 3D assets. There are some existing datasets that contain 3D- 3D-lanuuage data (e.g.. ScanQA [45], ScanRefer [5]). However, they are limited with regard to both quantity and diversity, restricted to only one task per dataset. How to generate a 3D-language dataset that can be utilized for all kinds of 3D-related tasks is well worth delving into.
Inspired by the recent success of large language models like GPT [33], we propose to leverage such models for 3D-language data collection. Specifically, as shown in Figure 2, we have three ways to prompt text-only GPT for generating data. 1) doxees-demonstaaionnnnnnttt iinstraciooonnnnttttttiinn based prompting. We input the axis- axis-aligned bounding boxes (AABB) of both the rooms and the objects in the 3D

---

3D Scene
Direct Reconstruct
3D Feature
Question
where the microwave?
Image Point Cloud
Multi View
gradSLAM
3D LLM
Perceiver
Unified Vocab.
Feature
LLM
what
Answer: on
Figure 3: Overview of our 3D-LLM framework. The first two columns show 3D feature extractor. We first render few multi-view images from the 3D scene, extract 2D dense features, and then construct 3D) features from these multi-view images using three kinds methods. And then, the 3D features and input language prompts are input to the 3D-LLMMs to generate responses.
scenes, providing information about the semantics and spatial locations of the scene. We then provide specific instructions to the GPT model to generate diverse data. We give 0-3 few-shot demonstration examples of the GPT model showing what kind of data is instructed to generate. 2) ChatCaptioner based prompting. We utilize techniques similar to [48]. in which ChatGPT is prompted to ask a series of informative questions about image and BLIP-2 [29] answers the questions. In order collect 3D-related data, we first sample several images from different views of a 3D scene. These images fed into ChatGPT and BLIP-2 2 to get the caption of each image. We then leverage ChatGPT to summarize all these captions, which contain information about different regions, to form a global 3D description of the entire scene. 3) Revision based prompting. It can be used to transfer one type of 3D data to another.
Given the prompting pipelines, GPT is able to generate various types of 3D- 3D-language data as summarized in Figure I. More data generation details and prompt designs are shown in the Appendix.
We mainly establish our Il>llaggaage dataset upon several 3D assets:
Objaverse a universe of 800K objects. However, since the language descriptions were extracted from online sources and not examined by humans, most objects have very noisy descriptions (e.g., with urls) or no descriptions. We utilize ChatCaptioner based prompting generate high-quality 3D-related descriptions for the scenes and divvison-based prompting to generate questions.
Scannet [12] richly-annotated dataset of approximately 1k 3D indoor scenes. It provides semantics and bounding boxes of the objects in the scenes.
SaabitttMMtteeppa (HM3D) [39] is a dataset of 3D environments of embodied AI. HM3DSem [44] further adds semantic annotations and bounding boxes for more than 200 scenes of HM3D. We use the pre-segmented rooms of HM3D in 3D-CLR [20].
4
3D-LLM
4.1
Overview
In this section, we introduce how we train our 3D-LLMs. We argue that it's hard to train 3D-LLMM from scratch, since our collected 3D-language dataset is still the size of billion-scale image- language dataset used to train 2D VLMs. Furthermore, for 3D scenes, there are no available pretrained encoders like those for 2D images (c.g.. CLIP ViT encoders). Thus, retraining 3D-language models from scratch is data-inefficient and resource-heave Recently, researchers have proposed to extract 3D features from 2D multi-view images [24, 20]. Using these alignment methods, we could use pretrained image encoders to extract image features, and then map the features to the 3D data. Since the pretrained image features serve as inputs to 2D VLMs, the mapped 3d features of the same feature space can also be seamlessly fed into the pretrained 2D VLMs, which we use as our backbones to train 3D-LLMMs. We also propose a 3D localization mechanism to boost the model's ability to capture 3D spatial information. Figure shows our framework.
5

---

4.2 3D Feature Extractor
The first step of training 3D-LLMs is to build meaningful 3D features that could be aligned with language features. For 2D images, there exist feature extractors like CLIP, which learn visual models from language supervision. The models are pretrained using billion-scale internet data of image- language pairs. It's hard to pre-train such feature learners from scratch, since there are no 3D-language assets comparable to internet-scale image-language pairs in terms of quantity and diversity.
On the contrary, numerous methods have been proposed to extract 3D features from 2D multi-view images [24, 20, 17, 21J. Inspired by these works, we extract features for 3D points by rendering the 3D scenes in several different views, and construct 3D features from rendered image features.
We first extract pixel-aligned dense features for rendered images following [24]. Then, we utilize three methods to construct 3D features from rendered image features. These methods are designed for different types of 3D data.
Direct Reconstruction. We directly reconstruct point cloud from rghd images rendered from the 3D data using ground-truth camera matrixes. The features are directly mapped to the reconstructed 3D points. This method is suitable for rendered rgbd data with perfect camera poses and intrinsics.
Feature Fusion. Similar to [24], we fuse 2D features into 3D maps using gradslam [27]. Different from dense mapping methods, the features are fused in addition to depths and colors. This method is suitable for 3D data with noisy depth map renderings, or noisy camera poses and intrinsics.
Neural Field. We utilize [20], which constructs 3D compact representation using neural voxel field [40]. Specifically, each voxel in the field has a feature in addition to density and color. Then we align 3D features in the rays and 2D features in the pixels using MSE loss. This method is for 3D data with RGB renderings but no depth data, and noisy camera poses and intrinsics.
In this way, we are able to obtain the < N, Da >-dim 3D features of each 3D scene, where N is the number of points in the point cloud, and D. is the feature dimension.
4.3 Training 3D-LLMs
4.3.1 2D VLMs as backbones
In addition to the feature extractor, training 3D-LLMs from scratch is also non-trivial. In fact, according to [29, 15], the training of 2D VLMs only begins to show "signs of life" after consuming half billion images. They usually use frozen and pre-trained image encoders such as CLIP to extract features for 2D images. Considering that with 3D feature extractor, the 3D features can be mapped into the same feature space 2D images, it's reasonable to use these 2D VLMS our backbones.
The perceiver architecture proposed by [23] leverages an asymmetric attention mechanism to itera- tively distill inputs into a tight latent bottleneck, allowing it to handle very large inputs of arbitrary input sizes, thus can tackle different modalities. This architecture is utilized in VLMs like Flamingo [15]. BLIP-2 [29] also utilizes a similar structure called QFormer. The 2D image features, output from frozen image encoders, are flattened and sent to the perceiver to generate a fixed-sized input. Given that our 3D features are in the same feature space the 2D features by the 3D feature extractor, and that perceiver is able to handle inputs of arbitrary input sizes of the same feature dimension, the
point cloud features with arbitrary sizes could also be fed into the perceiver. Therefore, 3D feature extractor to extract the 3D) features in the same feature space as the features of the frozen image encoders. Then, use pretrained 2D VLMs as our backbones, input the aligned 3D features to train 3D-LLMS with our collected 3D-laugaege dataset.
4.3.2 3D Localization Mechanism
Notice that since 3D features are reconstructed via 2D pretrained feature extractor that has been aligned with language (e.g., CLIP [36] and EVA-CLIP [41]). localization can be performed by directly calculating the similarity between 3D features and language features. However, Apart from building 3D features, which can be aligned with language semantics, it's also essential that the model itself could capture 3D spatial information. To this end, we propose 3D localization mechanism that boosts 3D LLMs' abilities to absorb spatial information. It consists of two parts:
Augmenting 3D features with position embeddings Besides the 3D features aggregated from 2D multi-view features, we also add position embeddings to the features. Supposing the feature dim D.,, we generate sin/cos position embeddings of the three dimensions, each has embedding size

---

De/3. We concatenate the embeddings of all three dimensions, and add them to the 3D features with a weight.
Augmenting LLM vocabularies with location tokens In order to align 3D spatial locations with LLMs, we propose embed 3D locations in the vocabularies, following [6] and [43]. To be specific, the region to be grounded can be denoted as a sequence of discrete tokens representing the bounding box in the form of AABB. The continuous corner coordinates of the bounding boxes are uniformly discretized to voxel integers as location tokens (Tmin Imin, Zminn &maz Uman, Zmaz). After adding these additional location tokens, we unfreeze the weights for these tokens in the input and output embeddings of language models.
5 Experiments
We first introduce the architecture, and training and evaluation protocols. In Sec 5.1, we analyze the held-out experiments on ScanQA [3]. SQA3D [32], and 3DMV-VQA [20] Dataset. Sec 5.2 covers more analysis on held-in evaluation and qualitative examples.
Architecture We experiment on three backbone 2D VLMs for 3D-LLMs: Flamingo 9B, BLIP-2 Vit-g Opt2.7B, BLIP-2 Vit-g X-L. For BLIP-2, during pre-training the 3D-LLMs, we initialize the model from BLIP-2 checkpoints released in LAVIS library [28], and finetune the parameters for the Qoomerr. 3D features are 1408-dim features, same as EVA_ CLIP [41] hidden feature dim used by BLIP-2. We keep most parts of the LLMs (i.e. Opt and F5anT5) frozen, except the weights for the newly-added location tokens in the input and the output embeddings. For Flamingo, we initialize the model from the Flamingo9B checkpoint released in OpenFlamingo repository [2]. We finetune the parameters for perceiver, gated cross attention layers, and the weights for additional location tokens in the input and output embeddings. 3D features are 1024-dim features, same as CLIP hidden feature dim used by Flamingo. For generating class-agnostic (generic) object masks for the 2D pixel-aligned dense feature extraction, we follow [24] and use the Mask2Former (M2F) [9] or the segment anything (SAM) [26].
Training& & Evaluation Datasets & Protocols We split our datasets into two genres, held-in datasets and held-out datasets. Specifically, our 3D-language data generation pipeline generates the held-in datasets of multiple tasks. We utilize training sets of held-in datasets for pre-training foundation 3D-LLMs, and their validation sets can be applied for held-in evaluation. During pre-training, we mix the held-in datasets of all tasks. The models are trained with the standard language modeling loss to output responses. Held-out datasets, on the other hand, are not used in training the foundation 3D-LLMs.. We use three held-out 3D question answering datasets for held-out evaluation: ScanQA, SQA3D and 3DMV-VQA.
5.1
Held-Out Evaluation
5.1.1
Experiments on ScanQA
We finetune our pretrained 3D-LLMs on the ScanQA dataset and compare with baseline models.
We include representative baseline models on the benchmark. ScanQA is the state-of-the-art method on the benchmark that uses VoteNet to obtain object proposals, and then fuse them with language embeddings. ScanRefer+MCAN is a baseline that identifies the referred object and the MCAN model applied to the image surrounding the localized object. VoANNet+MCAN detects objects in a 3D space, extracts their features, and uses them in a standard VQA model. Notably, these baseline models all extract explicit object representations from pretrained localization module. In addition to these baselines, we also design several L.L.-based baselines. LLaVA is visual instruction tuning that connects a vision encoder and LLM for general- purpose visual and language understanding. We use its pretrained model and do zero-shot evaluation on our dataset. We use a single random image as input. We use LLaVA 13B model. ULIP encoders + LLMS use existing pre-trained 3D encoders with LLMs, for comparison between 3D pre-trained encoders, and 2D encoders for feature encoding. Single Image + Pretrained VLMs use our 2D VLM backbones (i.e flamingo and BLIP-2), replace the 3D inputs of 3D-LLMs with single image features to train the models, and then finetune on ScanQA dataset.
Baselines & Evaluation Metrics
Image + Pretrained VLMs use our 2D VLM backbones, replace the 3D inputs of 3D-LLMS with concatenated features of nulti-view images to train the models, and then finetune on ScanQA dataset. We report BLEU, ROUGE-L, METEOR, CIDEr for robust answer matching. We also exact match (EM) metric.
7

---

B-1
B-2
B-3
B-4
METEOR
ROUHE-L
CIDER
EM
VoteNer+MCAN
28.0
16.7
10.8
6.2
11.4
54.7
17.3
ScanRefer-MCANN
26.9
16.6
11.6
7.9
11.5
55.4
18.6
ScanQA*
30.2
20.4
15.1
10.1
13.1
33.3
64.9
21.0
(.IaA(zzeo-shhot
7.1
2.6
0.9
0.3
10.5
12.3
5.7
0.0
C.IPPointMI.P-Aamm
18.4
7.2
2.7
1.4
7.4
18.1
26.9
7.5
ULIPPoinUMLPPopE
19.1
7.3
2.7
1.9
7.4
18.2
28.0
8.4
ULIPPoin(BERT--laatt
29.2
17.9
10.3
6.1
11.6
28.1
50.9
14.5
ULIPPoindBERTTopt
28.8
16.9
9.7
5.9
11.3
27.9
50.5
13.8
Samingo-Singlelmage
23.8
14.5
9.2
8.5
10.7
29.6
52
16.9
flamingo-Mun Maatingo-Mutiiiiii View
25.6
15.2
9.2
8.4
11.3
31.1
55
18.8
Sor-ffaant-Sinnlllmaat
28.6
15.1
9.0
5.1
10.6
25.8
42.6
13.3
BLIP2-BantS-Mullte
29.7
16.2
9.8
5.9
11.3
26.6
45.7
13.6
3D-LLM (M2F, flamingo)
30,3
17.8
12.0
7.2
12.2
32.3
59.2
20.4
3D-LLM (M2F, BLIP2-opt)
35.9
22.5
16.0
9.4
13.8
34.0
63.8
19.3
3D-LLM (SAM, BLIP2-opt)
35.0
21.7
15.5
9.5
14.0
34.5
67.1
19.8
3D-LLM (M2F, BLIP2-flant5)
39.3
25.2
18.4
12.0
14.5
35.7
69.4
20.5
3D-LLM (SAM, BLIP2-fAamtS)
37.5
24.1
17.6
12.9
15.1
37.5
74.5
21.2
Table I: Experimental results on ScanQA validation set.
Mcans the models use explicit object representations. B-1, B-2, B-3, B-4 denote BLEU-I, BLEU-2, BLEU-3, BLEU-4 respectively. M2F denotes mask2former, SAAMM denotes Segment Anything.
test set
Format
Avg.
What
Is
How
Can
Which
Others
Blind test
SQ A
26.75
63.34
43.44
69.53
37.89
43.41
43.65
ScanQA(w/o s sExt)
VQ - A
28.58
65.03
47.31
66.27
43.87
42.88
45.27
ScanQA
VSQ -
31.64
63.80
46.02
69.53
43.87
45.34
46.58
ScanQA+aux task
VSQ - AL
33.48
66.10
42.37
69.53
43.02
46.40
47.20
MCAN
VSQ -
28.86
59.66
44.09
68.34
40.74
40.46
43.42
ClipBERT
VSQ - A
30.24
60.12
38.71
63.31
42.45
42.71
43.31
Unified QA
VSQ 7 A
33.01
50.43
31.91
56.51
45.17
41.11
41.00
Unified QA
VSQ +
27.58
47.99
34.05
59.47
40.91
39.77
38.71
GPT-3
VSQ + A
39.67
45.99
40.47
45.56
36.08
38.42
41.00
GPT-3
VSQ -
28.90
46.42
28.05
40.24
30.11
36.07
34.57
3D-LLM
VSQ 7 A 37.05
65.18
45.81
67.46
51.00
49.82
49.79
Table 2: Experimental Results on SQA3D test set. In the Format column, "V" means the 3D visual inputs, "S"
means the situation inputs, "Q" and "A" denote questions and answers respectively. Here we use 3D-LLM (SAM, Ml..a-..lant..
Result Analysis We report results on ScanQA validation set in Table 1. We observe significant increase in the evaluation metrics. For example, for BLEU-1, our model outperforms the state-of- the-art ScanQA model by ~9% for validation set. For CIDER, report ~00% gain compared to ScanQA, and much higher than other 3D-based baselines. These results show that by injecting 3D into L.MMs, the models generate answers that much more similar the ground-truth answers. Furthermore, 3D-based baselines use object detectors like VoteNet to segment the objects, and then send per-object features into their models, while inputs holistic 3D features without explicit object representations. This shows that our model could perform visual reasoning about objects and their relationships even without explicit object representations. We then examine whether 2D VLMs have the same ability. We find that by taking single-view images or multi-view images as inputs, the performances drop much compared 3D-LLMs. Specifically, multi-view images also contain information about the whole scene. However, they have much lower performances compared to 3D-LLMs, probably because features multi-view images are disorganized, thus losing 3D-related information.
5.1.2 Experiments on SQA3D
SQA3D [32] requires the tested agent to first understand its situation (position, orientation, etc.) in the 3D scene as described by text, then reason about its surrounding environment answer a question under that situation. We finetune our pretrained 3D-LLMS on the SQA3D dataset and compare with baseline models. We include all baseline models introduced by the original paper. Specifically, ScanQA+aux task achieves the SOTA performance by adding two auxilliary tasks: prediction the

---

position and rotation of the agent situation. Table shows the results. We can see that our 3D-LLM outperforms all baseline models lot, even without training with auxiliary tasks and losses.
Methods
Concept
Counting
Relation
Comparison
Overall
NS-V NS-VQA*
21.5
33.4
61.6
38.0
SD-FFaatrr+LSTM
61.2
22.4
49.9
61.3
48.2
3D-CLR*
66.1
41.3
57.6
72.3
57.7
Hamingo-Sime Single
58.7
18.5
38.4
60.1
40.3
flamingo-Munn Mero-MuliVViiw
60.0
18.3
40.2
61.4
41.6
Im-linglee
58.0
20.4
42.3
62.3
43.1
BLIP-Multi View
61.9
21.1
48.0
62.3
47.1
3D-LLM (M2F, flamingo)
68.9
32.4
61.6
58.6
3D-LLM (M2F, BLIP2-opt)
63.4
30.7
57.6
65.2
54,9
3D-LLM (SAM, BLIP2-opt)
73.4
24.5
63.2
77.6
61.5
3D-LLLM (M2F, BLIP2-flanT55)
68.1
31.4
55.1
69.7
54.6
3D-LLM (SAM, BLIP2-flanTS)
76.3
30.2
64.3
80.2
64.0
Table 3: Experimental results 3DMV-VQA dataset. neuro-symbolie reasoning.
denotes using explicit object representations and
5.1.3 Experiments on 3DMV-VQA
finetune our pretrained 3D-LLMS on the 3DMV-VQA dataset and compare with baseline models. We include all baseline models introduced by the original paper. Specifically, 3D-CLR [20] is the SOTA achieves the SOTA performance via neuro-symbolic reasoning based on 3D features.
Result Analysis Table 3 shows the performances on 3DMV-VQA. We can see that 3D-LLMs outperform state-of-the-art baseline model in the question types concept and relation, and also in the overall performance. Our model also outperforms 3D-FFeauur++LSTTM demonstrating the power of L.LMs over vanilla language models with similar 3D features as inputs. Overall, 3D-based methods outshine 2D-based versions of the methods. Our 3D-LLMs outperform their corresponding 2D) VI.Ms with image input, further demonstrating the importance of 3D representations for 3D-LLMs.
Tasks
Models
BLEU-1
BLEU-2
BLEU-3
BLEU-4
METFOR ROUGH-T.
29.0
17.9
12.5
12.1
12.4
28.2
flamingo- MultiView
Bomiino-MMultiieee
29.5
18.6
13.7
12.4
14.0
29.0
BL.IP2-flnt5-Siin Single
30.3
18.3
14.5
12.0
13.1
30.9
BL.IP2-flat5--Mull View
34.4
23.9
18.0
14.1
17.5
35.7
Captioning
3D-LLM (flamingo)
36.1
24.5
18.7
15.6
17.6
35.8
3D-LLM (BLIP2-opt)
35.7
26.7
20.3
15.9
18.7
40.1
3D-LLM (BLIP2-LS)
17.7
3D-LLM (SAM, BLIP2-1S)
44.5
38.6
29.5
24.2
22.1
45.4
flant5
27.4
16.5
11.1
8.7
9.5
27.5
29.4
18.7
11.3
9.4
10.0
26.8
Hamingo-Mulln View
11.9
9.1
10.4
27.9
Sev22--aat5-Singeelmae
28.4
17.3
10.6
9.1
10.2
27.4
3D-assisted Dialog
BLIP2- frrat--flnt5-Aa Multi View
32.4
20.9
12.1
9.5
11.0
29.5
3D-LLM (flamingo)
35.0
22.8
15.4
10.6
16.0
34.2
3D-LLM (BLIP2-opt)
16.2
18.4
38.6
3D-LLMM (BI.IP2-flant5)
39.0
21.2
16.6
18.9
39.3
3D-LLM (SAM, BLIP2-t5)
40.5
29.4
23,9
21.4
19.6
40.8
flant5
25.5
21.1
16.7
6.0
13,9
28.4
Hamingo-Singlelin
31.4
23.0
18.8
7.1
15.6
30.6
flamingo-Min Multi View
33.1
24.7
21.4
7.3
16.1
33.2
1022-flatt-SSinleemmme
32.2
25.3
18.2
6.9
15.0
31.0
Task Decomposition
GLIP2-flan5--Mulf View
33.1
27.0
20.6
6.9
15.5
34.0
3D-LLM (flamingo)
32.9
25.6
20.2
6.4
16.0
33.5
3D-LLM (BLIP2-opt)
34.1
7.6
16.5
35.4
3D-LLM (BLIP2-flame
33.9
28.1
20.7
7.4
15.9
37.8
3D-LLM (SAM, BLIP2-15)
31.6
22.3
17.2
8.8
14.0
38.3
Table 4: Experimental Results on Held-In Datasets. 3D-LLMs outperform VLMs.

---

5.2 More Extensive Evaluation
Held-In Evaluation We earry experiments on held-in datasets of three tasks: 3D captioning. 3D-assited dialog and task decomposition. The baselines include 2D VLMs as for the held-out evaluation. We add one language-oniy baseline: FlanT5, which examines LLMs' ability complete these tasks without any visual input. To evaluate the quality of responses, we include BLEU, ROUGE- I., METEOR, CTDEr as our metrics. We report the held-in evaluation performances in Table From the table, we could see that 3D-LLLM could generate high-quality responses, outperforming both 2D VLMs and language-only LLMs.
Qualitative Examples In Figure 4, we show qualitative examples of 3D-LLM's predictions. We can that our 3D-LLM is able to perform a variety of tasks.
hels
Describe
lime?
white abare
Desoribe the
near
lound your stoes, they are you anything
Yos, neod
see towel on clithess honger near the
50
No, a towel you
week dienn up the scon What chould
Sum lene wo sowels hanging on door. do you
dlean dlean showel curtan with dleaningg solution eprigge rlean the mimor with dinoning noluriion snd sponge. il welll the to paser despeneer with Jolle paper d. relil soas diaponser wich soup
cleating edlution
want lo prepare documentta Whatshal! do reod?
bolat dearing solution and
1. cocumert cadénei spen
prepored
Where the wooden stee dask loceled?
comer room
refigeaatos
Can you help me find phone?
Thore &s woodon lilo,, ploood ned stars
Of coursel Do you member where you
iight
coject
Whnden Thilk
Figure 4: Qualitative examples 3D-LLM's prediction.
Conclusion
In this paper, we propose new family of 3D-LLMs that can take 3D representations as inputs and generate responses. We introduce series of 3D-language data generation pipelines to generate dataset of 1M 3D-lanuuage pairs to train our 3D-LLMs. Our 3D-LLMs leverage 2D pretrained VLMs as backbones and novel 3D localization mechanism. Experiments show that our 3D-LLMs outperform state-of-the-an baseline models on ScanQA datasets, and could perform a diverse set of 3D-related tasks. A limitation is that the 3D feature extractor relies 2D multi-view images, and thus all 3D scenes need be rendered so that they can be trained in 3D-L.LMMs, which introduces an additional rendering process.
10

---

7 Acknowledgements
This work was supported by the MIT-IBM Watson AT Lab, DARPA MCS. DSO grant DSOCO21072 and gift funding from MERL, Cisco, Sony, and Amazon. We would also like thank the computation support from AiMOS, server cluster for the IBM Research Al Hardware Center.
References
[1] P. Achlioptas, A. Abdelreheem, F. Xia, M. Elhoseiny, and L. J. Guibas. Referlt3D: Neural listeners for fine-grained 3D object identification in real-world scenes. In ECCV, 2020.
[2] A. Awadalla, Gao, Gardner, J. Hessel, Hanafy. W. Zhu, K. Marathe, Bitton, Gadre, Jitsev,
S. Kornblith, W. Koh, G. Ilharco, M. Wortsman, and L., Schmidt. Openftamingo, Mar. 2023.
[3] D. Azuma, T. Miyanishi, S. Kurita, and M. Kawanabe. ScanQA: 3D question answering for spatial scene understanding. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19107-19117, 2022.
[4] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems. pages 1877-1901, 2020.
[5] D. Z. Chen, A. X. Chang, and M. Nicßner. ScanRefer: 3D object localization in RGB-D scans using natural language. 16th European Conference on Computer Vision (ECCV), 2020.
[6] T. Chen, S. Saxena, L. Li, J. Fleet, and G. E. Hinton. Pix2seq: A language modeling framework for object detection, ArXiv, abs/2109. 10852, 2021.
[7] X. Chen, H. Fang, T.-Y. Lin, Vedantam, Gupta, P. Dollár, and C. Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv: 150.:0050 2015.
[8] Z. Chen, A. Gholami, M. Nießner, and A. X. Chang, Scan2cap: Context-awarc dense captioning in rgb-d scans. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 3193-3203, 2021.
[9] B. Cheng, A. Choudhuri. 1. Misra, Kirillov, R. Girdhar, and A. Schwing. Mask2former for video instance segmentation, ArXiv, abs/21 10.1644,, 2021.
[10] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, Sutton,
S. Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arX1v:2204.02311, 2022.
[11] H. W. Chung, I.. Hou, Longpre, B. Zoph, Y. Tay, W. Fedus, F. Li, X. Wang. M. Dehghani, S. Brahma, et al. Scaling cewtuucttiooiinntueeee language models. arXiv preprint arXiv:22 11416 2022.
[12] A. Dai, A. X. Chang, M. Savva, M. Halber, T. A. Funkhouser, and M. Nießner. ScanNet: Richly-annotated 3D reconstructions of indoor scenes. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages
2017.
[13] Databricks. Free dolly: Introducing the world's first truly open instruction-tuner 11m, 2023.
[14] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1810.0488, 2018.
[15] J. A. et al. Flamingo: a visual language model for few-shot lcarning. 2022.
[16] M. Feng, 7. Li, Q. Ii, L. Zhang, X. Zhang, G. Zhu, H. Zhang, Y. Wang, and A. S. Mian. Free-form descrip- tion guided 3d visual graph network for object grounding in point cloud. 2021 IRFE/CVF International Conference on Computer Vision (ICCV), pages 3702-3711, 2021.
[17] S. Gadre, M. Wortsman, G. Ilharco, L. Schmidt, and S. Song. CLIP on wheels: Zero-shot object navigation as object localization and exploration. ArXiv, abs/2203.10427 2022.
[18] T. Gong, C. Lyu, S. Zhang, Y. Wang, M. Zheng, Q. Zhao, K. Liu, W. Zhang, P. Luo, and K. Chen. MultiModal-GPi A vision and language model for dialogue with humans. arXiv preprint mmXiv:2305.04790 2023,

---

[19] Y. Goyal, T. Khot, D. Summers-Shee Stayy D. Batra, and D. Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[20] Y. Hong, C. Lin, Y. Du, 7. Chen, J. B. Tenenhaum, and C. Gan. 3D concept learning and reasoning from multi-view images, 2023.
[21] C. Huang, O. Mees, A. Zeng, and W. Burgard. Visual language maps for robot navigation, 2023.
[22] P.-H. Huang, H.-H. Lee, H.-T. Chen, and T.-L.. Liu. Text-guided graph neural networks for referring 3D instance segmentation. In AAAI, 2021.
[23] A. Jaegle, F. Gimeno, A. Brock, A. Zisserman, O. Vinyals, and J. Carreira. Perceiver: General perception with iterative attention. In International Conference on Machine Learning, 2021.
[24] K. M. Jatavallabhula, Kuwajerwala, Q. Gu, M. Omama, T. Chen, Li, G. Iyer, S. Saryazdi, N. Keetha, A. Tewari, J. B. Tenenbaum, C. M. de Melo, M. Krishna, L. Paull, F. Shkurti, and A. Torralba. Conceptiu- sion: Open-set set multimodal 3D mapping. 2023.
[25] C. Jia, Y. Yang, Y. Xia, Y.TT. Chen, Z. Parckh, H. Pham, Le, Y.H. Sung. Z. Li, and T. Duerig, Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904-4916. PMLR, 2021.
[26] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. B. Girshick. Segment anything. ArXiv, (as/2304.02643. 2023.
[27] Krishna Murthy, S. Saryazdi, G. lyer, and I. Paull, gradslam: Dense slam meets automatic differentiation. arXiv, 2020.
[28] D. Li, J. Li, H. Le, G. Wang. S. Savarese, and S. H. Hoi. LAVIS: A library for language-vision intelligence, 2022.
[29] Li, D. Li, S. Savarese, and S. Hoi. BLIP-2: Bootstrapping language-image pro-training with frozen image encoders and large language models, 2023.
[30] J. Li, D. Li, S. Savarese, and S. Hoi. BLIP-2: image encoders and large language models. arXiv preprint arXiv:2301. /2597, 2023.
Bootstrapping language-image pre-training with frozen
[31] H. Liu, C. Li. Q. Wu, and J. Lec. Visual instruction tuning. arXiv preprint arXi::2304.080:: 08485, 2023.
[32] X. Ma, S. Yong, Z. Zheng, Q. Li, Y. Liang, S.-C. Zhu, and S. Huang. Sqa3d: Situated question answering in 3d scenes, 2023.
[33] OpenAT. GPT-4 technical report, 2023.
[34] OpenAL. GPT-4 technical report. ArXiv, abs/2303.08774, 2023.
[35] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, al. Training language models follow instructions with human feedback. Advances Neural Information Processing Systems, 35:27730-2774 2022.
[36] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[37] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, (8)99 2019.
[38] C. Raffel, N. Shazeer, A. Roberts, K. Lee. S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-5511 2020.
[39] S. K. Ramakrishnan, A. Gokaslan, F. Wijmans, O. Maksymets, A. Clegg, J. M. Turner, F. Undersander, W. Galuba, A. Westbury, A. X. Chang, M. Savva, Y. Zhao, and D. Batra. Habitatt 3D dataset (HM3D): 1000 large-scale 3D environments for embodied AI. In Thirty-fifth Conference on Neural Information Procensing Systems Datasets and Benchmarks Track, 2021.
[40] C. Sun, M. Sun, and H.-T. Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition June 2022.
(CVPR), pages
12

---

[41] Q. Sun, Y. Fang, L. Wu, X. Wang, and Y. Cao. Eva-clip: Improved training techniques for clip at scale, 2023.
[42] Z. Sun, Y. Shen, Q. Zhou, H. Zhang, Z. Chen, D. Cox, Y. Yang, and C. Gan. Principle-driven self-alignment of language models from scratch with minimal human supervision. arXiv e-prints, pages arXiv- 2023.
[43] P. Wang, A. Yang. R. Men, Lin, S. Bai, Z. Li, J. Ma, C. Zhou, Zhou, and H. Yang. Unifying architectures, tasks, and modalities through simple International Conference on Machine Learning, 2022.
learning framework. In
[44] Yadav, R. Ramrakhya,
Ramakrishnan, Gervet, Turner, Gokaslan, N. Maestre, A. Chang. D. Batra, M. Savva, et al. Habitat-matterprt 3D semantics dataset. arXiv preprint urXiv:2210006633. 2022.
[45] S. Ye, D. Chen, S. Han, and J. Liao. 3D question answering. IEEE transactions on visualization and computer graphics, PP, 2021.
[46] L. Yu, P. Poirson, S. Yang, A. C. Berg, and L. Berg. Modeling context in referring expressions. In Computer Vision ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 1 11-14, 2016, Proceedings, Part 14, pages 69 85. Springer, 2016.
[47] P. Zhang. Y. Goyal, D. Summers-Stay, D. Batra, and D. Parikh. Yin and Yang: Balancing and answering binary visual questions. In Conference Computer Vision and Pattem Recognition (CVPR), 2016.
[48] D. Zhu, J. Chen, K. Haydarov, X. Shen, W. Zhang, and M. Elhoseiny. Chatgpt asks, blip-2 answers: Automatic questioning towards enriched visual descriptions, 2023.
13


Reviews:
[{"review_id": "VZxG89I4uj", "reviewer": "Reviewer_KXNR", "summary": "In this paper, the authors tried to leverage LLM to understand the 3D scene. Specifically, the authors use both grounding and captioning/QA datasets to tune the model. Specifically, the authors adopt the three 2D to 3D feature transformation techniques to let the model have a sense of the 3D features.", "questions": "See comments above. ", "limitations": "See comments above. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation is clear.\n2. The paper is easy to read. ", "weaknesses": "- When you aggregate the 2D features to 3D. It could be time-consuming and ill-posed. \n- The performance under the pre-trained weights upon all sources of the pretraining data is missing. Authors should report such results. In this way, you will see the effect of finetuning. \n- How to measure hallucination?\n- In 3D captioning, authors should report the baseline methods from recent papers. \n- I think this paper is more like a transition paper. What if we are given a (1) pure 3D point cloud, such as Modelnet? (2) Pointed cloud with limited images, such as KITTI? Here the ill-posed problem is that some 3D points can be never mapped into RGB pixel(s). In this sense, RGBD cameras are the only lucky sensor that can do the 2D to 3D projection. \n- The results on ScanRefer are far from satisfactory. I know that this task is hard, however, the baselines are not moderate/strong enough. I do not it make sense to put the random guessing or one-stage method numbers here...\n- What is the resolution of the position embeddings and location tokens? Given a large 3D scene such as LiDAR, accurately localizing the objects needs quite a lot of tokens. For example, if a scene is 50m*50m*6m, then you need a lot of tokens.\n- For grounding, what we expect is <loc_x><loc_y><loc_z>, what if the outputs are not what we expected, such as <loc_x><loc_z> or <loc_x>text1<loc_y>text2<loc_z>?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We'd like to express our sincere gratitude for your thorough review of our paper. We greatly appreciate your suggestions which are crucial in improving the quality of our paper. \n> Q1: Aggregating 2D features to 3D is ill-posed \n\nThanks for raising this concern. We want to emphasize that building LLMs on 3D world is extremely challenging, due to severely limited amount of existing 3D data and extreme difficulty to gather more such data. There are two potential solutions to approach the unexplored area:\n\nThe first one is to encode 3D features using pre-trained 3D encoders and input to LLMs, which is non-trivial due to the limited scale and diversity of 3D assets. Existing 3D encoders focus on simple objects instead of real-world scenes[1,2]. Taking the suggestion of reviewer 6ag4, we replace 3D-LLM's features with features from pretrained 3D encoders (ULIP). Results below show their performances are very poor.\n||EM|B1|B2|B3|B4|METEOR|ROUHEL|CIDER\n-|-|-|-|-|-|-|-|-\nULIPPointMLP+t5|7.5|18.4|7.2|2.7|1.4|7.4|18.1|26.9\nULIPPointMLP+opt|8.4|19.1|7.3|2.7|1.9|7.4|18.2|28.0\nULIPPointBERT+t5|14.5|29.2|17.9|10.3|6.1|11.6|28.1|50.9\nULIPPointBERT+opt|13.8|28.8|16.9|9.7|5.9|11.3|27.9|50.5\n3DLLM(BLIP2opt)|19.3|35.9|22.5|16.0|9.4|13.8|34.0|63.8\n3DLLM(BLIP2t5)|20.5|39.4|25.2|18.3|12.3|14.9|35.9|69.3\n\nThe second one is learning models of the 3D world by leveraging the abundance of 2D multiview images and aggregating 2D features to 3D. There has been a recent surge of works using 2D features to construct 3D representations[3,4], and there is significantly more plentiful 2D multiview data of scenes than there are 3D scans of these scenes. These methods further show superior ability in zero-shot and open-vocabulary reasoning, indicating this is a promising and approariate strategy for processing 3D scenes at the moment.\n\n**While we do not come to the conclusion that the second solution, utilized by 3D-LLM, is better than the first one, and indeed has several limitations, it is the most practical solution given the limited data for the time being.** 3D-LLM serves as a promising first step into exploring LLMs grounded in the 3D physical world and brings inspiration into the community. We believe that in the future, more powerful models will be built upon the combination of the two solutions. More discussions will be added in the revision. \n\n> Q2: Performance under pre-trained weights \n\nAttached below.\n\n||EM|B1|B2|B3|B4|METEOR|ROUHEL|CIDER\n|-|-|-|-|-|-|-|-|-\n3DLLM(flamingo)|14.0|22.5|12.1|5.5|2.7|10.0|24.3|49.8\n3DLLM(BLIP2opt)|13.2|26.4|14.3|7.2|3.4|11.9|27.1|52.7\n3DLLM(BLIP2t5)|13.8|28.6|17.0|9.9|6.6|12.3|28.0|52.7\n> Q3: Hallucination\n\nWe explore two metrics[5].\n\nCHAIR: Portion of hallucinated objects in all mentioned ones\n\nHRF@k: Portion of frequent objects in hallucinated ones\n\nWe report scores of two tasks.\n\n|||CHAIR↓|HRF@10↓\n-|-|-|-\nTask Decom.|t5|55.6|69.7\n||BLIP2t5-Image|35.4|56.8\n||3DLLM(BLIP2t5)|7.5|39.2\nDialog|t5|51.6|59.9\n||BLIP2t5-Image|29.2|54.6\n||3DLLM(BLIP2t5)|5.5|33.2\n> Q4: 3D captioning baseline\n\nWe attach the result of OpenShape[6], a model trained on 3 kinds of captions on 800k Objaverse data. We evaluate on OpenShape testset using pre-trained 3D-LLMs without finetuning on their training set. No data leakage among splits. \n\n||B4|B3|B2|B1|METEOR|ROUGHL\n-|-|-|-|-|-|-|\nOpenShape|1.8|3.6|8.4|19.7|5.9|18.4\n3DLLM(BLIP2opt)|8.5|11.0|15.2|21.7|10.3|29.4\n3DLLM(BLIP2t5)|9.0|11.4|16.7|23.6|11.0|31.3\n\nResults on our Objaverse testset.\n\n||B4|B3|B2|B1|METEOR|ROUGHL\n-|-|-|-|-|-|-\nOpenShape|0.9|2.0|4.8|11.7|4.3|14.9\n3DLLM(BLIP2t5)|20.1|24.7|31.0|39.8|17.7|42.6\n\n3D-LLMs outperform OpenShape a lot, even on OpenShape's testset, with  less than 10% training data of OpenShape's. We give both models' qualitative results in PDF. We'll run more models if the reviewer has suggestions.\n\n> Q5: Some 3D points can be never mapped into pixels. \n*  We agree that our framework is not suitable for all kinds of point clouds, but note that many existing pointclouds can be mapped and rendered with 2D images to get features.\n*  In the PDF, we show the results on ModelNet with rendered images, features and 3D-LLM responses. For KITTI, we could take a partial point cloud at each step, which we already did that with our navigation task. KITTI results are also in the PDF.\n* For pointclouds that may not be easily rendered, we could learn a separate stream to directly encode 3D features from the pointclouds and align them with the 3D features from RGB images. This allows us to take advantage of both 3D data and plentiful 2D multiview data.\n\n> Q6: ScanRefer results\n* We gladly share with you our new result, outperforming previous one by 5%. We also add the required stronger baselines. New result is achieved by a few modifications: 1) Before we followed the original version of [3] using MaskFormer to get dense 2D features, now we use Segment Anything 2) Pos embeddings are added to the features rather than concatenated. \n\n||Acc@0.25\n-|-\nOracleRand|29.9\nOracleRefer|40.6\nVoteNetRand|10.0\nSCRC|18.7\nOneStage|20.4\nVoteNetGRU|39.5\nScanRefer|41.2\n3DLLM(BLIP2t5)|30.3\n3DLLM(BLIP2t5)-new|35.2\n\n> Q7: Resolution\n\nPos embedding: $256^3$. Location token: $64^3$ (64 tokens,applied in 3 dims). We could expand the token number (e.g., 64 to 256) for larger scenes. \n> Q8: Wrong grounding output \n\nGrammarly-incorrect output is considered wrong with 0 IOU.\n\n[1]PointCLIP: Point Cloud Understanding by CLIP \n\n[2]ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding \n\n[3]ConceptFusion: Open-set Multimodal 3D Mapping \n\n[4]3D Concept Learning and Reasoning from Multi-View Images\n\n[5]Evaluating Object Hallucination in Large Vision-Language Models \n\n[6]OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understanding \n\nWe wish that our response has addressed your concerns and turns your assessment to the positive side. If you have more questions, feel free to let us know during the rebuttal window. Thank you! "}, {"review_id": "SvEjtMcT4P", "reviewer": "Reviewer_G3Mn", "summary": "This paper proposes a new framework named 3D-LLM which leverages LLM to understand the 3D world. Specifically, 3D-LLM can take 3D point clouds as inputs to conduct various 3D tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. To achieve this goal, this paper designs three types of prompting mechanisms to generate over 300k 3D-language data. It proposes a 3D feature extractor that obtains 3D features from rendered multiview images and takes pretrained 2D VLMs as backbones to train 3D-LLM. Both benchmark results on held-out and held-in data show the effectiveness of the proposed framework, which achieves SOTA performance on ScanQA benchmark. ", "questions": "See weakness. ", "limitations": "The limitations are not discussed. ", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. This paper is well written.\n2. The idea of injecting the 3D world into large language models is novel.\n3. The technical contributions, including data generation, overall framework, and experimental analysis are solid and convincing. \n4. The proposed 3D-LLM achieves impressive results on both quantitative and qualitative results.", "weaknesses": "This paper is satisfactory. I only have some minor comments.\n1. In Table 4, the ablation study could be more comprehensive, where the baseline should remove all of the position embedding, location tokens, and localization. The ablation study should begin with this baseline and show all possible combinations of these three designs.\n2. In Figure 1, three approaches generate 3D features, what is the effectiveness of each of them? The experimental results should be included. \n3. Limitations of 3D-LLM should be discussed. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "&nbsp;\n\n*We appreciate the positive and constructive comments from you, which are essential for improving the paper! We have conducted your suggested experiments. We will update all results in the paper.*\n\n&nbsp;\n\n> **Q1: In Table 4, the ablation study could be more comprehensive, where the baseline should remove all of the position embedding, location tokens, and localization. The ablation study should begin with this baseline and show all possible combinations of these three designs.** \n\nSorry for the confusion! The localization in the table actually means position embeddings plus location tokens. As can be seen from the paper Line 202: *\"It consists of two parts: 1) Augmenting 3D features with position embeddings 2) Augmenting LLM vocabularies with location tokens\"*. Therefore, we think we have covered all the cases. We will modify the writing to make it more understandable here.\n\n\n&nbsp;\n\n> **Q2: In Figure 1, three approaches generate 3D features, what is the effectiveness of each of them? The experimental results should be included.** \n\n* The three approaches are meant for different kinds of data. For example, for real-world scans like ScanNet, camera perspective distortion is inevitable and thus we cannot use direct reconstruction to reconstruct the 3D scenes (in fact, we find that no two adjacent partial point clouds could align using direct reconstruction). Thus, we use *feature fusion with SLAM* to build the features for ScanNet. For 3DMV-VQA, the depth data is not released, and thus we can only use *neural field* to reconstruct the 3D scenes. On the other hand, for Objaverse data, since we are rendering with blender which gives us correct camera poses and depths, we use *direct reconstruction*.\n* To shed light on the performances of different approaches, we conduct an experiment where we use all three different features to construct the ScanNet features, with 3D-LLM (BLIP2-flant5) as our model, and show them in Table A. We can see that the *direct reconstruction* result is inferior to the two others, mainly because the scannet features we obtain via direct reconstruction have noises due to camera perspective distortion. The results of *feature fusion* and *neural field* are on par since they can both correctly reconstruct the 3D scenes. We will update more experimental results to the camera ready version concerning this question you raised.\n\n\n&nbsp;\n\n**Table A. Comparison among 3D feature generation approaches.**\n|                           | B-1           | B-2           | B-3         | B-4          | METEOR        | ROUHE-L       | CIDER         | EM            |\n|----------------------------|---------------|---------------|-------------|--------------|---------------|---------------|---------------|---------------|\n| 3D-LLM   (Direct Reconstruction)                  | 34.6 | 22.1 | 15.7        | 9.1          | 13.5 | 33.0 |55.7 | 18.9 |\n| 3D-LLM   (feature fusion)                 |  39.4 | 25.2 | 18.3 | 12.3 | 14.9   | 35.9    | 69.3   | 20.5 |\n| 3D-LLM   (Neural Field)                  | 39.1 | 25.0 | 18.5      | 12.1        | 15.2 | 36.0 | 67.3 | 20.3 |\n\n\n&nbsp;\n\n> **Q3: Limitations of 3D-LLM should be discussed.** \n\nIn Line 295 of the paper, we gave one limitation of 3D-LLM: *\"A limitation is that the 3D feature extractor relies on multi-view images\"*.\n\nWe would like to share with you some more limitations which are crucial for further improvement of this paper:\n\n* For the grounding mechanism, we input detailed text descriptions to refer to an object and train 3D-LLMs to output location tokens for these objects. However, the referring sentence might contain multiple hops of relations, and directly training on such corpus is non-trivial since the models need to simultaneously learn both semantics and relationships. A better way to improve the grounding mechanism is to assign location tokens to each noun in all data, like in [1].\n* We find that 3D-LLMs, like a lot of 2D VLMs, are bad at grounding relationships [2]. This problem is more salient for 3D tasks which have more complex spatial relationships. Relational modules need to be added to the models.\n* We do not have ego-centric or robot-centric data in our 3D-language data. Therefore, current 3D-LLMs are unable to solve embodied robotics tasks. Such data and tasks are crucial for equipping 3D-LLMs with the ability to understand the complex 3D physical world. \n\n\n&nbsp;\n\n[1] Kosmos-2: Grounding Multimodal Large Language Models to the World. Zhiliang Peng et al. 2023\n\n[2] Going Beyond Nouns With Vision & Language Models Using Synthetic Data. Paola Cascante-Bonilla et al. 2023\n\n&nbsp;\n\n*Please let us know if you have any further questions for our paper. We sincerely appreciate your time for reviewing this paper and raising the valuable suggestions! Thank you again!*\n\n&nbsp;\n\nBest,\nAuthors "}, {"review_id": "5kFJj2bs1i", "reviewer": "Reviewer_6ag4", "summary": "This paper proposes a new family of 3D-LLMs that can take 3D representations as inputs and generate responses, it introduces a series of 3D-language data generation pipelines to generate a dataset of 300K 3D-Language pairs from different tasks for the training.", "questions": "Typo: there is an extra \".\" in line 132.\nThe reviewer is curious whether the authors have tried to train it directly using the language-3D dataset for the 3D encoder directly without leveraging the images as the bridge, and how will it perform differently.\nIn response to that suggestion, the reviewer would like to mention some related works that have explored the alignment of 3D-image-language triplets and consequently trained the 3D encoder to have language context [1][2]. Given the abundance of 3D object data, employing a pre-trained 3D encoder, and subsequently fine-tuning it with the LLM in this case could be a potentially promising strategy, and it might yield more fruitful results compared to training a model from scratch using the same data the authors have collected.\n[1]: \"ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding\" -- CVPR2023\n[2]: \"CLIP2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Data\" -- CVPR2023\n", "limitations": "refer to the weaknesses section", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed approach seems to be valid and functioning, and the authors say they plan to release the 3D-Language dataset as well.\nOverall it demonstrates how to use 2d features to gather 3D features and then inject them into LLM.", "weaknesses": "The weakness of this paper is on how to gather the 3D representation, current pipeline seems to be relying on 2D multi-view images, which will introduce extra complexity/latency and limitations. Also, when you project 3D data to a series of multi-view images, it's likely that there will be some information loss, even though the reviewer does appreciate that the authors do realize this issue and come up with some remedy approaches like the \"3D Localization Mechanism\", it seems functioning to some extent, but it might not be the optimal approach eventually.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "&nbsp;\n\n\n*We appreciate the positive and insightful comments from you! We address your concerns in detail below.*\n\n&nbsp;\n\n> **Q1: The reviewer is curious whether the authors have tried to train it directly using the language-3D dataset for the 3D encoder directly without leveraging the images as the bridge, and how will it perform differently. In response to that suggestion, the reviewer would like to mention some related works that have explored the alignment of 3D-image-language triplets and consequently trained the 3D encoder to have language context. Given the abundance of 3D object data, employing a pre-trained 3D encoder, and subsequently fine-tuning it with the LLM in this case could be a potentially promising strategy, and it might yield more fruitful results compared to training a model from scratch using the same data the authors have collected.** \n\nThank you for suggesting! The suggestion will be immensely helpful in enhancing the paper’s quality and assisting the readers in understanding the paper's contributions.\n\nWe replace 3D-LLMs' features with features from pretrained 3D encoders (ULIP) [1]. The results shown in Table A and B suggest that LLMs with 3D encoders have very poor performances, inferior to 3D-LLMs by a large margin.\n\n&nbsp;\n\n**Table A. Experimental Results of Pretrained 3D Encoder with LLMs, on ScanQA.**\n|| EM|B-1|B-2|B-3|B-4|METEOR|ROUHE-L|CIDER|\n|-|-|-|-|-|-|-|-|-|\n|VoteNet+MCAN*|17.3|28.0|16.7|10.8|6.2|11.4|29.8|54.7|\n|ScanRefer+MCAN*|18.6|26.9|16.6|11.6|7.9|11.5|30|55.4|\n|ScanQA*|21.0|30.2|20.4|15.1|10.1|13.1|33.3|64.9|\n|flamingo-SingleImage|16.9|23.8|14.5|9.2|8.5|10.7|29.6|52.0|\n|flamingo-MultiView|18.8|25.6|15.2|9.2|8.4|11.3|31.1|55.0|\n|BLIP2-flant5-SingleImage|13.3|28.6|15.1|9.0|5.1|10.6|25.8|42.6|\n|BLIP2-flant5-MultiView|13.6|29.7|16.2|9.8|5.9|11.3|26.6|45.7|\n|**ULIP_PointMLP+flant5**|7.5|11.0|18.4|7.2|2.7|1.4|7.4|18.1|26.9|\n|**ULIP_PointMLP+opt**|8.4|19.1|7.3|2.7|1.9|7.4|18.2|28.0|\n|**ULIP_PointBERT+flant5**|14.5|29.2|17.9|10.3|6.1|11.6|28.1|50.9|\n|**ULIP_PointBERT+opt**|13.8|28.8|16.9|9.7|5.9|11.3|27.9|50.5|\n|3D-LLM (flamingo)|20.4|30.3|17.8|12.0|7.2|12.2|32.3|59.2|\n|3D-LLM (BLIP2-opt)|19.3|35.9|22.5|16.0|9.4|13.8|34.0|63.8|\n|3D-LLM (BLIP2-flant5)|20.5|39.4|25.2|18.3|12.3|14.9|35.9|69.3|\n\n&nbsp;\n\n**Table B. Experimental Results of Pretrained 3D Encoder with LLMs, on Captioning.**\n|Models|BLEU-1|BLEU-2|BLEU-3|BLEU-4|METEOR|ROUGH-L|\n|-|-|-|-|-|-|-|\n|**ULIP-PointMLP+flant5**|24.6|20.8|14.9|10.4|12.5|35.3|\n|**ULIP-PointMLP+opt**|24.4|20.4|14.3|10.8|12.1|34.1|\n|**ULIP-PointBERT+flant5**|26.0|21.9|15.7|10.0|14.8|33.8|\n|**ULIP-PointBERT+opt**|24.3|20.7|15.8|11.3|12.1|39.5|\n|3D-LLM (flamingo)|36.2|24.8|19.0|16.0|17.6|40.8|\n|3D-LLM (BLIP2-opt)|35.7|26.7|20.3|15.9|18.7|40.1|\n|3D-LLM (BLIP2-t5)|39.8|31.0|24.7|20.1|17.7|42.6|\n\n&nbsp;\n\n\n> **Q2: The weakness of this paper is on how to gather the 3D representation, current pipeline seems to be relying on 2D multi-view images, which will introduce extra complexity/latency and limitations. Also, when you project 3D data to a series of multi-view images, it's likely that there will be some information loss, even though the reviewer does appreciate that the authors do realize this issue and come up with some remedy approaches like the \"3D Localization Mechanism\", it seems functioning to some extent, but it might not be the optimal approach eventually.** \n\n* We admit that one limitation of the paper is that the 3D representation relies on 2D multi-view images, which may result in extra complexity and information loss. This limitation is also covered in Line 295 of our submission.\n* However, We want to emphasize that building LLMs on 3D world is extremely challenging, due to the severely limited amount of existing 3D data and extreme difficulty to gather more such data. There are two potential solutions to approach the unexplored area:\n\n    * The first one is to encode 3D features using pre-trained 3D encoders and input to LLMs, which is non-trivial due to the limited scale and diversity of 3D assets. Existing 3D encoders focus on simple objects instead of real-world scenes[1,2]. In Q1, we show that such pre-trained 3D encoders have very poor performances.\n    \n    * The second one is learning models of the 3D world by leveraging the abundance of 2D multiview images and aggregating 2D features to 3D. There has been a recent surge of works using 2D features to construct 3D representations[3,4], and there is significantly more plentiful 2D multiview data of scenes than there are 3D scans of these scenes. These methods further show superior ability in zero-shot and open-vocabulary reasoning, indicating this is a promising and appropriate strategy for processing 3D scenes at the moment.\n\n   * **While we do not come to the conclusion that the second solution, utilized by 3D-LLM, is better than the first one, and indeed has several limitations, it is the most practical solution given the limited data for the time being.** 3D-LLM serves as a promising first step into exploring LLMs grounded in the 3D physical world and brings inspiration into the community. We believe that in the future, more powerful models will be built upon the combination of the two solutions. More discussions will be added in the revision.  \n\n&nbsp;\n\n[1] ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding \n\n[2] CLIP2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Data\n\n&nbsp;\n\n\n*We sincerely appreciate your comments. Please feel free to let us know if you have further questions. Thank you again for your time!*\n\n&nbsp;\n\nBest,\nAuthors"}, {"review_id": "uTZ5aRTysj", "reviewer": "Reviewer_QPPK", "summary": "Due to the limited perception of 3D space by LLMs and VLMs, this paper proposes 3D-LLMs to understand spatial relationships, affordances, physics, and layout in 3D scenes. The authors generate 300K 3D-language pairs to train the 3D-LLMs, which enable better performance on various 3D understanding tasks.", "questions": "1. Which 3D feature extractors were used for Objaverse, ScanNet, and HM3D, respectively? How many pairs were extracted for each?\n2. Since the LLMs and VLMs have strong few-shot capabilities, how do the 3D-LLMs perform in zero/few-shot scenarios on held-out datasets?\n3. As mentioned in the weaknesses, what results would be obtained if other baselines were also trained using the generated data?", "limitations": "It would be better to present the performance of 3D-LLMs in zero/few-shot scenarios on held-out datasets.", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The motivation is reasonable. The idea is clear.\n2. The experimental results cover a wide range of tasks, including 3D captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, and navigation. The results are substantial and solid.\n3. The organization and writing of the paper are fluent.", "weaknesses": "1. Is there a potential issue of data leakage? For instance, when generating the 3D-language pairs using ScanNet, the authors utilized object semantic information and bounding boxes, which might benefit the model's performance on downstream tasks such as ScanRefer and ScanQA.\n2. The baselines, such as VoteNet+MCAN and ScanRefer+MCAN, should also be trained using the generated data to ensure fairness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "&nbsp;\n\n*Thank you for your insightful and constructive comments! We have added additional experiments and modified our paper according to your comments.*\n\n&nbsp;\n\n> **Q1: Which 3D feature extractors were used for Objaverse, ScanNet, and HM3D, respectively? How many pairs were extracted for each?**\n\n* For Objaverse, since we render images using Blender, which gives us correct depths and camera poses, we use *direct reconstruction* to reconstruct Objaverse scenes.\n* For ScanNet, we use the original images they took to construct the scans. Since camera perspective distortion is inevitable for sensors used to take real-world pictures, we cannot use direct reconstruction to reconstruct the 3D scenes (in fact, we find that no two adjacent partial point clouds could align using direct reconstruction). Thus, we use *feature fusion with SLAM* to build the features for ScanNet. \n* For HM3D, we use mixed ways. For 3DMV-VQA [1], they do not give the depths for multi-view images. We thus use their released codes that build the features using *neural field*. However, we also use habitat-lab to collect the data of more scenes to train diverse tasks, and we are able to use *direct reconstruction* to reconstruct the data generated by habitat-lab. For example, for navigation data we need to reconstruct the partial point cloud at each observation step, and thus *direct reconstruction* is faster than *neural field*.\n\n&nbsp;\n\n> **Q2: Since the LLMs and VLMs have strong few-shot capabilities, how do the 3D-LLMs perform in zero/few-shot scenarios on held-out datasets?**\n\nThank you for suggesting! We attach the results of the pre-trained models below.\n\n&nbsp;\n\n**Table A. Zero-shot Performances of Pretrained Models, on ScanQA.**\n|| B-1  | B-2  | B-3  | B-4  | METEOR | ROUHE-L | CIDER | EM   |\n|-|-|-|-|-|-|-|-|-|\n| VoteNet+MCAN*  |28.0 | 16.7 | 10.8 | 6.2  | 11.4   | 29.8    | 54.7  | 17.3 |\n| ScanRefer+MCAN* |26.9 | 16.6 | 11.6 | 7.9  | 11.5   | 30      | 55.4  | 18.6 |\n| ScanQA* | 30.2 | 20.4 |15.1 | 10.1 | 13.1   | 33.3    | 64.9  | 21.0 |\n| LLaVA(zero-shot)| 7.1 | 2.6  | 0.9  | 0.3  | 10.5   | 12.3    | 5.7   | 0.0  |\n| **3D-LLM (flamingo) - PT** |  22.5  |  12.1   |   5.5   |    2.7    |   10.0    |   24.3   |  49.8    | 14.0 |\n| **3D-LLM (BLIP2-opt) - PT**  | 26.4 | 14.3 | 7.2  | 3.4  | 11.9   | 27.1    | 52.7   | 13.2|\n| **3D-LLM (BLIP2-flant5) - PT** |  28.6 | 17.0 | 9.9  | 6.6  | 12.3   | 28.0    | 52.7   |13.8 |\n| 3D-LLM (flamingo) |  30.3 | 17.8 | 12.0 | 7.2  | 12.2   | 32.3    | 59.2  | 20.4  |\n| 3D-LLM (BLIP2-opt) |  35.9 | 22.5 | 16.0 | 9.4  | 13.8   | 34.0    | 63.8   | 19.3   |\n| 3D-LLM (BLIP2-flant5) |  39.4 | 25.2 | 18.3 | 12.3 | 14.9   | 35.9    | 69.3   | 20.5 |\n\n&nbsp;\n\n**Table B. Zero-shot Performances of Pretrained Models, on 3DMV-VQA.**\n Methods | Concept | Counting | Relation | Comparison | Overall \n---|---|---|---|---|---\n CNN+LSTM | 57.8 | 22.1 | 35.2 | 59.7 | 37.8 \n MAC | 62.4 | 19.7 | 47.8 | 62.3 | 46.7 \n MAC(V) | 60.0 | 24.6 | 51.6 | 65.9 | 50.0 \n NS-VQA | 59.8 | 21.5 | 33.4 | 61.6 | 38.0 \n ALPRO | 65.8 | 12.7 | 42.2 | 68.2 | 43.3 \n LGCN | 56.2 | 19.5 | 35.5 | 66.7 | 39.1 \n 3D-Feature+LSTM | 61.2 | 22.4 | 49.9| 61.3 | 48.2 \n 3D-CLR (Ours) | 66.1 | 41.3 | 57.6 | 72.6 | 57.7 \n **3D-LLM (flamingo) - PT** | 60.5 | 18.9 | 47.6 | 63.7 | 45.7 \n **3D-LLM (BLIP5-opt) - PT** |60.1|17.8 | 44.2 | 58.9 | 42.9 \n **3D-LLM (BLIP2-flanT5) - PT** | 61.0 | 18.4 | 45.0 | 62.0 | 43.9 \n 3D-LLM (flamingo) | 68.9 | 32.4 | 61.6 | 68.3 | 58.6 \n 3D-LLM (BLIP5-opt) | 63.4 | 30.7 | 57.6 | 65.2 | 54.9 \n 3D-LLM (BLIP2-flanT5) | 68.1 | 31.4 | 55.1 | 69.7 | 54.6 \n\n&nbsp;\n\n**Table C. Zero-shot Performances of Pretrained Models, on ScanRefer.**\n||Acc@0.25\n-|-\nOracleRand|29.9\nOracleRefer|40.6\nVoteNetRand|10.0\nSCRC|18.7\nOneStage|20.4\nVoteNetGRU|39.5\nScanRefer|41.2\n**3DLLM(BLIP2t5)-PT**|20.1\n**3DLLM(BLIP2t5)-new-PT**|26.7\n3DLLM(BLIP2t5)|30.3\n3DLLM(BLIP2t5)-new|35.2\n\n*\"new\" means the new model we train after submission*\n\n&nbsp;\n\n> **Q3: what results would be obtained if other baselines were also trained using the generated data?**\n\n* Thank you for suggesting! However, these baselines rely on privileged information that is not available in our current pre-training dataset. The training of VoteNet and ScanRefer requires the ground-truth segmentations of the objects in the scenes. However, we do not have these annotations in the Objaverse dataset. We do think such information is crucial and one future step for us is to label the segmentations of the Objaverse dataset to help improve the performances of 3D-LLMs.\n* The ScanQA baseline uses an answer classification module in addition to object localization, detection and object classification modules to propose answers. Thus, it does not have the language generation ability which is essential for our tasks (*e.g.*, captioning and dialogue).\n* Despite these limitations, we still provide the results of VoteNet+MCAN and ScanRefer+MCAN below, where we use pre-trained VoteNet and ScanRefer and train the MCAN part (the results might be not so meaningful since there is non-trivial domain gap between the data VoteNet and ScanRefer trained on and our pre-training data). We pre-train the models on our pre-training dataset and finetune them on ScanQA. \n\n&nbsp;\n\n**Table D. Performances on ScanQA baselines when pretrained on 3D-language data.**\n| | B-1  | B-2  | B-3  | B-4  | METEOR | ROUHE-L | CIDER | EM   |\n|---|------|------|------|------|--|-|-|------|\n| VoteNet+MCAN*  | 28.0 | 16.7 | 10.8 | 6.2  | 11.4   | 29.8    | 54.7  | 17.3 |\n| VoteNet+MCAN*  (pretrained)| 26.4 | 15.4 | 9.1 |5.8  |10.4  | 25.3    | 50.6  | 15.9 |\n| ScanRefer+MCAN* |26.9|16.6| 11.6| 7.9 | 11.5| 30| 55.4  | 18.6 |\n| ScanRefer+MCAN* (pretrained)  | 28.3 |17.0|12.1|7.6|11.2| 29.7| 52.6  | 16.9 |\n\n&nbsp;\n\n[1] 3D Concept Learning and Reasoning from Multi-View Images. Hong et al. 2023\n\n&nbsp;\n\n*We sincerely appreciate your comments. Please feel free to let us know if you have further questions.*\n\n&nbsp;\n\nBest,\nAuthors"}]

Author rebuttal:
We sincerely appreciate all reviewers’ time and efforts in reviewing our paper. In addition to the response to specific reviewers, here we would like to highlight our contributions and the new experiments that we add in the rebuttal.

&nbsp;

**[Our Contributions]**

We are glad to find out that the reviewers generally acknowledge our contributions:
 * The motivation is reasonable and clear. [KXNR, QPPK]
 * The idea of injecting the 3D world into large language models is novel,  valid and functioning. [G3Mn, 6ag4]
 * The technical contributions, including data generation, overall framework, and experimental analysis are solid and convincing. The 3D-Language dataset will be released. [G3Mn]
 * The experimental results cover a wide range of tasks, including 3D captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, and navigation. The results are substantial and solid. [QPPK]
 * The organization and writing of the paper are fluent.[KXNR, G3Mn, QPPK]

**[New Experiments]**

In this rebuttal, we have added more supporting experiments to address reviewers’ concerns.
* Results using existing pre-trained 3D encoders for 3D-LLMs [6ag4]
* Results on pre-trained 3D-LLMs without finetuning [KXNR, QPPK]
* Comparision with recent 3D captioning model [KXNR]
* Measurement of Hallucination [KXNR]
* New results on ScanRefer [KXNR]
* Comparison among 3D feature generation approaches [G3Mn]
* ScanQA baselines pretrained on 3D-language data [QPPK]

**[Qualitative examples]**

We attach two qualitative examples in the PDF:
* Results on ModelNet and KITTI [KXNR]
* Qualitative examples of comparison between OpenShape and 3D-LLM on captioning [KXNR]

&nbsp;

We hope our responses below convincingly address all reviewers’ concerns. We thank all reviewers’ time and efforts again!

Meta-review: Reviewers were unanimously positive about this paper. There are a good number of definite strengths: clear, novel and very timely motivation; wide range of tasks covered; solid experimental results.

Nonetheless, there were also a number of issues/questions raised in the initial reviews, including baseline methods, number of tokens needed, potential data leakage, use of 2D multi-view images instead of "pure" 3D point cloud data, etc. However, the rebuttals included a good number of experimental results (e.g. zero-shot performance, baselines, etc.), addressing the issues raised, and reviewers were satisfied that their concerns were addressed. Perhap one issue that was not quite satisfactorily addressed in this work itself, is the issue of a true 3D point cloud encoder -- but I concur that this can be stated as a limitation and left for future work.

Overall, this is a solid and timely piece of work that deserves attention from the NeurIPS community.
Decision: Accept (spotlight)