Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers
Authors: Haifeng Huang
Yilun Chen
Zehan Wang
Rongjie Huang
Runsen Xu
Tai Wang
Luping Liu
Xize Cheng
Yang Zhao
Jiangmiao Pang
Zhou Zhao
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D Scene Understanding
Multi-modal Large Language Model
Primary area: machine_vision
TL;DR: A 3D large language model unifying referencing and grounding capabilities.

Abstract: Recent advancements in 3D Large Language Models (LLMs) have demonstrated promising capabilities for 3D scene understanding. However, previous methods exhibit deficiencies in general referencing and grounding capabilities for intricate scene comprehension. In this paper, we introduce the use of object identifiers and object-centric representations to interact with scenes at the object level. Specifically, we decompose the input 3D scene into a set of object proposals, each assigned a unique identifier token, which enables efficient object referencing and grounding during user-assistant interactions. Given the scarcity of scene-language data, we model the scene embeddings as a sequence of explicit object-level embeddings, derived from semantic-rich 2D or 3D representations. By employing object identifiers, we transform diverse 3D scene-language tasks into a unified question-answering format, facilitating joint training without the need for additional task-specific heads. With minimal fine-tuning on all downstream tasks, our model significantly outperforms existing methods on benchmarks including ScanRefer, Multi3DRefer, Scan2Cap, ScanQA, and SQA3D.

Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers
Haifeng Huang 1.21
Yilun Chen
Zehan Wang" Yang Zhao
Rongjic Huang", Jiangmiao Pang"
Runsen Xu²
Tai Wang"
Luping Liu'
Xize Cheng'
Zhou Zhao 1.2
Zhejiang University
SShaggaai AI Laboratory (huanghaifeng) edu.cn
'Bytedance Inc.
Abstract
Recent advancements in 3D Large Language Models (LLMs) have demonstrated promising capabilities for 3D scene understanding. However, previous methods exhibit deficiencies in general referencing and grounding capabilities for intricate scene comprehension. In this paper, we introduce the use of object identifiers and object-centric representations to interact with scenes at the object level. Specif- ically, decompose the input 3D scene into set of object proposals, each assigned a unique identifier token, which enables efficient object referencing and grounding during user-assistant interactions. Given the scarcity of scene-language data, we model the scene embeddings as a sequence of explicit object-level em- beddings, derived from semantic-rich 2D 3D representations. By employing object identifiers, we transform diverse 3D scene-language tasks into a unified question-answering format, facilitating joint training without the need for addi- tional task-specific heads. With minimal fine-tuning on all downstream tasks, our model significantly outperforms existing methods benchmarks including Scan- Refer, Multi3DRffer, Scan2Cap, ScanQA, and SQA3D. Code has been released at https://gituu..com/ZZ/ http:///iitbb..omm/////ZCCCCC//C
Introduction
Recent advancements in Large Language Models (LLMs) [14, 39, 47, 15, 65, 32] have established language universal interface for creating general-purpose assistants. This breakthrough has been instrumental in the development of Multi-modal LLMs (MLLMs), which effectively tackle broad spectrum of multi-modal tasks. While significant strides have been made in 2D MLLMs [33, 35, 77, 79, 34, 31, 67], current 3D MLLMs still face significant challenges that must be overcome achieve a general- purpose assistant for 3D scene understanding.
general-purpose
Object referencing and grounding are essential for advanced scene understanding. Object referencing involves model's 's precise comprehension of the semantics associated with user-specified object, while object grounding requires the model's ability to localize target object within the scene. These capabilities
vital for various 3D scene-language tasks such dense captioning [12] and visual grounding [4, 75, 1]. However, current 3D MILMs lack general referencing and grounding capabilities, often failing in tasks that necessitate precise object referencing or grounding-contral to the objectives of addressing general-purpose tasks.
Regarding the object referencing capability, several 3D MLLMs [6, 23, employ additional prompt encoders to comprehend user-specified objects, but they still lack grounding capabilities. The 3D-
Equal contribution. Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

LLM [21] incorporates location tokens to enable object grounding, a technique validated in the 2D domain [67]. However, this approach underperforms on the 3D grounding benchmark, ScanRefer [4], compared to traditional expert models. The ineffectiveness of location tokens in the 3D domain primarily arises from the significant data scareity in the scene-language area. Current 3D scene- language datasets [4, 75, 1] contain only tens of thousands of grounding instances, a scale much smaller than the million-level datasets used for training 2D MLLMS [67. 72]. Given the exponentially greater complexity of 3D spaces compared to 2D spaces, robust training of location tokens for 3D MLLMs may require substantially more data than is currently used for 2D MLLMs. Therefore, our objective is to explore more efficient methods for object referencing and grounding and to mitigate the impact of data scarcity.
We observe that most existing 3D MLLMs convert the 3D scene into hidden 3D scene embeddings, employing either a Q- Q-Former-based module [21, 6] or direct projection methods [23]. Such archi- tectures inherently lack the capability to efficiently interpret individual object instances. To address this limitation, we propose a novel approach for representing and interacting with 3D scenes at the object level within the language model. This method incorporates two principal designs: (i) referencing 3D scene using object identifiers, and (ii) representing 3D scene using well-trained object-centric representations. The first component offers a unified format for object referencing and grounding, while the second alleviates the requirement for extensive scene- Lene-lang datasets.
Reference 3D scene using object identifiers. Objects play a crucial role in defining and inter- preting a scene, as their organization shapes the entire 3D landscape. This intuition evident in most 3D scene understanding benchmarks, including 3D grounding, VQA, and dense cap- tioning, all of which annotate at the object level. To effectively model scene embeddings at the object level, the entire 3D scene can be decomposed into a set of object proposals via reliable 3D detectors [45, 27, 71, 50, 30]. Importantly, we assign objects with object identifiers-al set of learnable identifier tokens (<OBJ>>) k= to distinguish them during language modeling. This design allows the LLM reference respective objects using discrete identifier tokens. As the example shown in Figure I, the chair and the two trash cans are labeled as "<OBJ013>", "<OBJO23>>,, and "<OBJ032>", respectively. This avoids the text ambiguity that arises from subjective viewing words like "rightmost". Besides, the lengthy description like "the chair located at the southwest corner of the rightmost table" often complicates user- assiisattt interaction. These identifiers enable efficient object referencing and grounding during user-assistant interactions. By using these identifiers, we convert diverse 3D scene-language tasks into a unified format of duestion-answere pairs, facilitating joint training without any additional task-specific heads.
thed clous raac his comen the rigtrmn siili.
SOBIRLS
-OHURIES
both locaud
Our of Licr is gesdl
purtolly
other his place on the foor apannd Dar
Fied the mach his 10-0B110..
1o and COEU both lecrted naor the
Figure I:
An example of using object identifiers
during the conversation.
Represent 3D scene using well-trained object-centric representations. The scene-level repre- sentation requires a large amount of paired scene- language data for training, which is generally unaffordable and labor-intensive due its complex real-world scenarios. To address this challenge, our model represents the scenes using a set of object-level embeddings, which obtain object-centric representations from well-trained 2D and 3D models. Specifically, after obtaining object propos- als from prior detectors (either 2D or 3D), we extract the object features using well-trained 3D) object-centric representations [78] or 2D representations [40]. Due the million-level pre- pre-training. these representations contain abundant semantic and visual cues. Through simple linear layers, we project them into the embedding space of the language model. Combined with the object identifier, the sequences of object-level embeddings are thus constructed into scene embeddings and fed into the L.M. With merely two epochs of fine-tuning on all downstream tasks, extensive experiments on either 3D, 3D+2D, or 2D-only settings demonstrate the effectiveness of our model in various downstream 3D scene understanding tasks.
We perform comprehensive experiments across five representative 3D scene-language datasets, including ScanRefer [4], Multi3DRefer [75], Scan2Cap [12], ScanQA (2), and SQA3D [38]. Our model consistantly enhances state-of-ther thetharrt performance across all these datasets without fine-tuning

---

on specific tasks. Notably, it surpasses previous methods by 3.7% (Acc@0.5) on ScanRefer, 14.0% (FI @0.5) on Mulei3DRefer, 8.7% (CIDEr@0.5) on Scan2Cap. and 7.7% (CIDEr) on ScanQA.
Our contributions are summarized as follows:
We propose an enhanced 3D MILM which models and interacts with 3D scenes at the object level. We introduce object identifiers to enable efficient referencing and grounding within 3D scenes. By leveraging these identifiers, we convert diverse 3D scene-language tasks into unified question- answering format, facilitating joint training without necessitating additional task-specific heads.
We effectively represent the 3D scene through a sequence of multi-modal object-centric representa- tions derived from well-trained foundation models, which alleviate the impact of scene-language data scarcity.
Our model significantly enhances state-of-the-art performance across various 3D scene-language datasets without fine-tuning for specific tasks. Extensive experiments on either 3D, 3D+2D, or 2D-only settings demonstrate the effectiveness our model for 3D indoor scene understanding.
2
Related Work
3D Scene-language Understanding In the rapidly evolving field of 3D scene understanding, there is an increasing focus on using language to provide both contextual knowledge and query conditions, thus enabling precise interpretation of user intentions. This process, known as "3D scene-language understanding", " leverages language to more effectively grasp the intricacies of 3D environments in a manner consistent with human cogmition. The primary tasks in this domain include: 1) 3D Visual Grounding [4. 1, 25, 75, 7, 52, 76, 53, 48], which involves locating specified objects within 3D scene based on textual queries; 2) 3D Dense Captioning [12. 69, 28, 8, 9], which demands proficiency in both localizing and captioning objects densely in the scene; 3) 3D Visual Question Answering [2, 43, 38), which focuses on general scene question answering. Initial efforts concentrated on specialized tasks, resulting in limited generalizability across different 3D scene understanding tasks. Recent initiatives such as 3DJCG [3] and D3Net [?] have aimed to unify tasks like 3D visual grounding and dense captioning. leveraging their synergistic benefits to enhance overall model performance. Advances like 3D-VisTA [80] and 3D-VLP [29] are working to develop a more general 3D visual-language framework through pre-training techniques for better scene- language alignment. However, despite these models' adeptness at handling various 3D scene tasks, their reliance on task-specific heads limits their adaptability for broader uscr-assistant interactions.
Multi-modal Large Language Models. Recent advancements in large language models (LLMs) have exhibited impressive capabilities in intricate reasoning and interactive dialogues with humans [14, 39, 47, 15]. There is a growing interest in enhancing the scope of LLMs to encompass additional modalities [31, 33, 35, 77, 79, 20, 19, 21, 54, 65, 32, 61, 23, 10]. In the 3D realm, PointLLM [61] directly maps point clouds into the embedding space of the LLM. Both Imagebind-LLLn Point-LLM [19] integrate the 3D modality into LLMs by establishing joint embedding space among 3D point clouds, images, audio, and text. These models perform well at the object level but encounter difficulties when interpreting complex spatial relationships in 3D scenes. To improve scene understanding, 3D-LLM [21] incorporates positional embedding and Icarns location tokens. Nevertheless, it projects 3D features into the input space of pre-trained 2D Vision-Language Models (VLMs). Involving 2D encoders make it difficult to grasp the 3D spatial structure and intricate relationships among objects. Chat-3D [54] tackles this limitation by directly utilizing 3D scene-text data to align the 3D scene with the LLM, overcoming the challenge of limited data availability through pre- alignmegmment phase. However, the architectural design of this model limits its focus on specific target objects during interactions. Current 3D MLLMS face challenges in precise object referencing and grounding, limiting their functionality to straightforward tasks. By incorporating object identifiers into the I.M,, we significantly enhance the object referencing and grounding capabilities of 3D MLLMs. thereby showing potential for complex real-world applications.
[20] and
3D Representation Learning Recently, numerous efforts have been made to learn discriminative and robust representations for 3D point clouds, which serve as a fundamental visual modality. Approaches such as POintBERT [68). Point-MAR [41], Transformer-OcCo [51], and Point-m2ae [73] employ self-supervised learning techniques to extract meaningful representations of 3D objects from unlabeled point cloud data. Another set of works [62, 36, 74, 17, 58, 56, 55, 57] seeks to extend
3

---

Object Proposals
Projected Masks
Multiciow Imagge
Ditesoor
3D Encoder
*
2D Encoder
Language Tokenizer
3D-language Projector
2DD 2D-langunge Projector
User: Generate a grounded caption scene.
O,
Language Model
Assistant: ... Thee living area is furnished with an L-shaped couch (<0BJ132>) lacing brown entertainment center housing a black TV (=OBJ015>)) A gray ottoran coffee table (=ORM134>) placod in front of the couch. wonden shelf (-OR8056>) enclosing two sections of adds cozy
space...
Figure 2: Overall model architecture. The model processes a 3D scene's point cloud input by decomposing it into object proposals via pre-trained detector. Subsequently, the 3D and 2D encoders are employed to extract object-centric representations. After projection layers, they are combined with object identifiers form the scene emheddings sequence of object-level embeddings, which are then fed into the LLM. The assigned unique identifiers enable efficient object referencing in subsequent interactions.
representation from other modalities the 3D domain. For example, ULIP [62] and OpenShape [36] construct 3D- image-text triplets to align point clouds within the CLIP [44] representation space. C- MCR [58] and Ex-MCR [56] learn contrastive representations between various modalities, including 3D point clouds. They leverage knowledge from existing MCR spaces to tackle to challenge of lacking paired data. These robust 3D representations effectively capture detailed information about 3D objects. Our approach involves segmenting the 3D scene at the instance level and extracting set of object features represent the entire scene.
3 Method
3.1 Overview
Our motivation to facilitate object referencing and grounding for 3D MLLMs while simultaneously addressing the scarcity of scene-language data. We propose representing 3D scenes the object level by using object identifiers for referencing and employing well-trained, object-centric representations for scene depiction. Section 3.2 delineates the model architecture, which primarily consists of generating sequence of object-level embeddings to represent the entire scene, Section 3.3 provides illustrations of the prompt template through examples. Lastly, Section 3.4 details the training methodology our model.
3.2 Model Architecture
As illustrated in Figure 2, our method processes 3D scene's point cloud by decomposing into object proposals using pre-trained detector. We then employ pre-trained 3D and 2D encoders to derive object-centric representations from point clouds and multi-view images, respectively. These representations are subsequently mapped into the token embedding space of language model. By incorporating n object identifiers into the language model's vocabulary, we link these identifiers' to corresoonding object proposals, thereby facilitating efficient object referencing and grounding during user-assistant interactions. Finally, the scene embeddings, composed of sequence of object-level embeddings, input into the LLM.

---

Object Detector. Given a point cloud from 3D scene, we decompose it into n objects using the pre-trained detector Mask3D [45]. Compared to object detection models [11, 46]. the instance segmentation model is preferred in this work due to due to its capability to generate accurate masks, which are essential for subsequent projection into 2D masks on multi-view images. The point cloud of the i-th object is denoted as P, E IR" x6 where my represents the number of points in the i-th object, and the 6D information for each point comprises its 3D coordinates and RGB colors.
Object Identifiers. To achieve a localized understanding of 3D scenes, we introduce a set of learnable identifier tokens {<OBJi>),=
"__.. .x' designated as object identifiers, into the token vocabulary of the language model. These identifiers are processed by the tokenizer to produce their respective token embeddings {O,},=1,,. . The identifier tokens are then integrated with the object tokens to establish one-to-one correspondences, thereby enabling object referencing and grounding using identifiers in subsequent interactions.
Object-level Embeddings. After extracting object proposals from the 3D scene, we derive object features using well-trained 3D and 2D object representations. Owing the million-level scale of pre-training, these representations are rich in semantic and visual cues. By employing simple linear layers, we project them into the embedding space of the language model. Together with identifier token embeddings, this process yields object-level embeddings for each object.
3D Encoder. The 3D encoder excels in extracting spatial and shape attributes from point clouds. We employ a pre-trained 3D encoder Uni3D [78] to derive object-centric 3D representations. This embedding processes each object's point cloud P, 1. outputting the feature Zí, for each object.
2D Encoder. The 2D encoder adeptly extracts semantically rich features from 2D images. We project the point clouds for each object onto multi-view images, creating sequence of 2D masks. Utilizing pre-trained DINOv2 [40], we extract and aggregate local features from all masked regions across the multi-view images of each object, taking into account both mask areas and multi-view information. We opted for DINOv2 over the more common CLIP [43] due to its superior handling of local features within images. The 2D encoder processes the multi-view images and their corresponding projected masks from each object's 's point cloud P,, generating the visual feature Z for each object. Details are provided in Appendix A.
Visual- Visual-Language Projectors. To align the extracted object representations with the language model, we employ 3D-language projector fp(-) and 2D-Langaage projector f-(.) to map the 3D point cloud features and 2D visual features into the token embedding space of the language model. For the i-th object, these features are represented token embeddings F. and F.P
F.- (Z,); F. (Z,))
(1)
Scene Embeddings. Following the process described above, we obtain an object identifier token embedding O, a 3D object token embedding F. and 2D object token embedding F. for each object. We combine the identifier token embeddings and object token embeddings in an one-to-one correspondence manner to formulate a sequence of object-level embeddings, which represents the constructed scene embeddings and then be fed into the LLM represent the whole scene.
3.3 Prompt Template
Despite variations in task formulations, both referencing and grounding are unified using object identifiers. As illustrated in Table 1, the system message encodes object information in the scene as sequence of "<OBJr> <object>", where <OBJi> denotes the identifier token for the i-th proposal, and <object> serves the placeholder for object tokens. The language tokenizer converts <OBJý> into its token embedding O and <object> into the combined object token features F. and F. As illustrated by the following interaction, the user can directly employ identifier tokens to reference specific objects, while the assistant uses these tokens in responses to precisely ground target objects.
3.4 Training Strategy
Most existing MLLMs [18, 35, 23] adopt a two-stage training approach, comprising an initial alignment phase to train the projector exclusively, followed by fine-tuning phase for both the projector and the language model. This method not only demands extra data and extended training duration for alignment but also complicates determining the optimal duration for the initial phase. Consequently, we opt for a single-stage process, concurrently training both the projectors and the
5

---

Table 1: Prompt template for the language model.
System: A chat between curious and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. The conversation centers around an indoor scene: [<0BJ001> <object> <OBJ002> <object> <0BJn> <object>|. User: Find the closest trash bin to <OBJ013>.
Assistant: There are two trash bins, <0BJ023> and <BBJ032>, both located near the chair.
Single odjeot Grounding
ebjeot Grounding
Whet the ID or the objact the reachus the Aloutppiin "weedic
Are (has ole devecieed ou "the bunkgrad
abjoct beated
sink
in the right comerr of the reser"
c0183844 and Hb881854.
Are these de dosoriec cn "ulorr the lncheress
Visual Question
object.
Whaceia rucianzcla kischn ouzaer lsceant Ancair grestion uning siggle diggle planse
No.
Danae Captioning
Provide detalel Ancripizn
Answering
000.111 bettre analystng qutal connection with ofuer clammaes the aceas.
raking olanser dinssr
dospeaa thes ias th laasly zurpor iua toklaci Anoker granioe wenng single grarss.
laptop When
dod they a hed fras of and wall calona lalk
Figure 3: Examples of various 3D scene-language understanding tasks. All the tasks are unified to single-turn question-answering pairs without extra task heads. Object identifiers used reference and ground the object during the conversation.
language model. In our experiments, we observe that this jointly trained model already exhibits superior performance without the necessity of fine-tuning for specific downstream tasks.
Training Data We aggregate essential training data for downstream tasks and standardize into uniform instruction formats. The downstream tasks encompass 3D visual grounding (ScanRefer & Multi3DRef), 3D dense captioning (Scan2Cap), and 3D visual question answering (ScanQA & SQA3D). We incorporate the training sets from these datasets into our training corpus. Each task is adaptable to single-turn user-assistant interaction, as illustrated in Figure 3.
Training Objective We have unified all tasks into consistent user-assistant interaction format, and result, the sole training loss in the joint-training phase is the Cross-Entropy loss of the language model. The training objective is to optimize the trainable parameters, denoted by 0, aiming minimize the negative log-likelihood of the target response sequence ges generated by the assistant. Specifically, given the input prefix sequence preffxx which encompasses both system messages and user instructions, the loss function is expressed as follows:
k
(()) = log ((
Σ
rea
2es
1...tt1) Prefix)
where & the number of tokens in the response sequence, and res f denotes the sequence the previous tokens the response. The set trainable parameters 0 includes two vision-languaae projectors, newly added n token embeddings for object identifiers, and the language model itself.

---

Table 2: Performance comparison. "Expert models" are tailored for specific tasks using task-
oriented heads, while "LLM--bssed models" are designed for general instructions and responses.
ScanRefer
Refee
Scan2Cap
ScanQA B4
SQA3D
Method
Acc@0.25
A.c00..5
F1
F1@05
C@005
B-400.5
EM
EM R
ScanRefer [4]
37.3
24.3
Expert Midels
ScanQA [2]
64.9
10.1
3DJCG [3]
49.6
37.3
49.5
31.0
3D-VLP [29]
$1.4
39.5
54,9
32.3
67.0
11.1
M3DRef-CLIP [75]
44.7
42.8
38.4
3D VisTA (80)
50.6
45.5
66.9
34.0
72.9
13.1
48.5
ConcreteNet [48]
Vote2Cap DETR++ [9]
67.6
37.1
L.M Models
LAMM
3.4
42.4
5.8
Chat-3D [54]
53.2
6.4
3D-LLM [21]
L.L3DA [6]
65.2
36.8
76.8
13.5
LEO [23]
36,9
80.0
11.5
53.7
Scene-I.IA J.I.M
80.0
12.0
54.2
Char-Soene (Ours)
55.5
50.2
57.1
52.4
77.1
36,3
87.7
14.3
54.6
57.5
4 Experiments
4.1 Datasets and Metrics
Datasets. We conducted experiments five benchmarks: ScanRefer [4] for single-object visual grounding, Multi3DRefer [75] for multi-object visual grounding, Scan2Cap [12] for dense captioning. and both ScanQA [2] and SQA3D [38] for visual question answering. These benchmarks
based on the ScanNet dataset [16], which comprises richly annotated RGB-D scans of real-world indoor scenes, including both 2D and 3D data across 1,513 scans. All benchmarks adhere to the same train/validatiin cuin/validation/testt splits, facilitating joint training and evaluation.
Metrics. We adhere to the commonly used metrics in these benchmarks. For ScanRefer [4], we assess thresholded accuracy with Acc@0.25 and 0.5,, where predictions are deemed positive if they exhibit higher Intersection over Union (IoU) with the ground truths than the thresholds of 0.25 and 0.5, respectively. In evaluating grounding for a flexible number target objects in Multi3DRefer [75], employ the Fl score IoU thresholds of 0.25 and 0.5. For Scan2Cap [12], we utilize CIDEr@0.5 and BLEU-4@0 0.5 (abbreviated as C@0.5 and B-4@0.55 integrating image captioning metrics with the IoU scores between predicted and target bounding boxes. For ScanQA [2), the metrics CIDEr [49] and BLEU-4 [42], abbreviated as C and B-4, are used. SQA3D [38] is evaluated using exact match accuracy (EM) and its refined version, EM-R, proposed by LEO [23].
4.2 Comparison with State-of-the-are Methods
The baseline models can be classified into two principal categories: traditional expert models and general multi-modal LLMs. Traditional expert models generate outputs in fixed format tailored to specific tasks. Conversely, LLM-based models yield open-ended outputs suitable for broader range of applications.
Compared Baselines.
Models such ScanRefer [4] and ScanQA [2] establish initial benchmarks for the ScanRefer and ScanQA datasets, respectively. 3DICG [3] integrates multiple tasks within single architecture, unifying visual grounding and dense captioning tasks due their synergistic nature. Both 3D-VLP [29] and 3D-Vis VisTA [80] aim to develop versatile 3D visual-language frameworks by focusing pre-training strategies for 3D scene-language alignment. M3DRef- CLIP [75] introduces multi-object grounding, enhancing single-object grounding performance. ConcreteNet [48], the state-of-the-aan (SOTA) model the ScanRefer benchmark, innovates three methods to augment verbo-visual fusion for dense 3D visual grounding. Jote2Cap-EETR. [9] decouples the processes of caption generation and object localization through parallel decoding, making it the SOTA model on the Scan2Cap benchmark.
Expert Models:
LLM--baed Models: LAMM [66] extends research on 2D MLLM to point clouds but lacks design tailored for 3D scene tasks. Chat-3D [54] introduces an object-centric method yet fails to address general 3D scene tasks comprehensively. 3D-LLLM [21] utilizes location tokens for object
grounding but is constrained by data scarcity. LL3DA [6] develops assistant that processes

---

Table 3: Ablation studies on object identifiers. "Plain Text" employs plain text for object numbers, "Gaussian" uses fixed Gaussian embeddings, and "Learnable" learns new identifier tokens. "Token Cost" denotes the total tokens for N objects, including object identifiers.
ScanRefer A.c@0.5
Multi3DRef @.55
Scan2Cap C@0.5
ScanQA CIDEr
SQA3D
Identifier Token Type
Token Cost
EM
Plain Text
6.V
47.2
49.6
73.1
84.9
53.7
Gaussian Learnable
3NV
46.1
49.4
71.7
82.5
53.4
3N
50.2
52.4
77.1
87.7
54.6
Table 4: Ablation studies on multi-modal object-centric representations. "Early Fusion" merges object features before language model input, whereas "Separate Token" keeps them distinet. "Single" denotes using a single image to extract 2D feature of an object, while "Multi" uses multi-view images.
ScanRefer Acc@0.5
Multi3DRef @1
Scan2Cap C@0.5
ScanQA CIDEr
SQA3D
Fusion Method
Token Cost
3D
2D
EM
41.2
43.8
64.9
80.3
53.4
2N
Single Multi
32.9
37.2
65.9
83.7
53.2
45.8
49.1
75.7
88.2
54.4
Single Multi
45.7
49.1
73.8
86.5
53.2
Separate Token
3/V
50.2
52.4
77.1
87.7
54.6
Single Multi 46.9
42.4
46.8
70.9
85,9
52.9
Early Fusion
2NV
50.2
53.9
point cloud data directly, responding to textual instructions and visual prompts. LEO [23] pioneers embodied generalist approach by incorporating action tokens. Scene-LLM [18] merges scene- level and egocentric 3D information, enhancing understanding and reasoning in 3D environments. However, LL3DA, LEO, and Scene-LLM lack grounding capabilities.
Analysis. As shown in Table 2, our model surpasses previous methods across almost all metrics without task-specific fine-tuning, suggesting a promising unified framework for 3D scene under- standing. For visual grounding tasks, our model boosts the state-of-the- the--rrtttttttto performance by 3.7% (Ace@0.5 0.5) on ScanRefer and 14.0% (F1 @0.5) on Multi3DRefer, demonstrating excellent grounding capabilities. For the dense captioning task, improve the SOTA performance by 8.7% (CIDEr@0.3) on Scan2Cap, highlighting strong object referring and captioning ability. For VQA tasks on ScanQA and SQA3D, which do not require object referencing and grounding, still achieve consistent performance enhancement, demonstrating improved overall 3D scene understanding and reasoning.
4.3 Ablation Study
Object Identifiers. Table 3 shows that the format of object identifiers affects both performance and token cost. For comparing token costs, we consider scenes with hundreds objects. A straightforward approach is to plain text for object identifiers such "Obj001", which is tokenized into four tokens ("Obj", "0", "0", "1"). Including two object tokens (3D & 2D), representing a single object requires six tokens in total. This high token cost makes the approach impractical for real-workh scenarios. Thus, we explored using a single token per identifier by adding new tokens to the language's vocabulary. We assess strategies: employing fixed random Gaussian embeddings ("Gaussian") and using learnable tokens ("Learnable"". The results show that learnable tokens enhance performance and reduce token costs simultaneously. Lowering token costs from 6.V 3N significantly reduces memory usage and accelerates training/inference when handling 3D scenes with numerous objects.
Multi-modal Object-centric Representations. We evaluate various methods for retrieving object features and combining features from different sources (3D and 2D). as shown Table 4. As deseribed in Section 3.2, the 3D and 2D features derived from the 3D encoder and 2D encoder, respectively. We assess two methods of extracting 2D features: one from a single-view image ("Single"") and another from multi-view images ("Multti").
First, we evaluate the performance of using either a single 3D feature or a single feature for the object token. The results show that using a 2D feature derived from multi-view images yields

---

punel of room, couch
Ours
There is recianguler bine bin, It is next to a white trush bug.
Ours
Figure 4: Visualization results of video grounding for video input. "GT" denotes the projected 2D) masks derived from the ground-truth 3D point cloud mask.
better performance than using 3D feature. This suggests that semantic information from 2D visual contexts is more crucial than spatial information from 3D point clouds. It may also indicate that the pre-trained 2D encoder is more reliable than the pre-trained 3D encoder due the abundance 2D image-text data compared to 3D-text data for pre-training.
Next, assess the combination of both 3D and 2D features using two fusion methods. The first method, named "Separate Token" in the table, involves using two separate object tokens (3D and 2D object tokens) to represent object information. The second method, termed "Early Fusion", combines the 3D and 2D features into a single token for each object. The results indicate that combining 3D and 2D features consistently improves performance compared using a single 3D/2D feature, highlighting the importance of utilizing both modalities. Fusing 3D and 2D tokens reduces the token cost, while results slight performance drop. This provides an option for situations where the token limit is tight, suggesting that combining multi-modal features into one token is acceptable.
4.4 Experiments with 2D Video Input
In practical applications of 3D scene understanding. acquiring indoor RGB (video) sean is simpler than obtaining processed 3D point cloud from RGB-D images. We examine model's ability to adapt to video input (without depth) for 3D indoor scenes based on the ScanNet [16] dataset. For video input, we use a tracking-based video detector DEVA [13] extract object proposals. This process involves detecting objects in each frame and merging these proposals across frames via the tracking module. After extracting objects from the video, perform the same operations as for the 3D tasks. The grounding results can then be evaluated on video frames with 2D masks.
Tasks and Metrics. We assess video ground- ing and VQA tasks for video input. For video descriptions annotated in ScanRefer and project the ground-truth object's point cloud to 2D masks in video frames, allow- ing us to compute the IoU between the predicted masks and GT masks in 2D images. Given a video with a frame length of L, the predicted masks are a series of 2D masks denoted as (M, e IR # Te=... and the GT masks denoted as (Mã E R where H and W are the height and width of an image, respectively. We concatenate these masks along the temporal axis to obtain predicted spatial-temporat mask MP E xx and a GT spatial-temporal mask M Hx X We propose calculating the Spatial-Tempone IoU (ST-loU) between the predicted mask MP and the GT mask M Thus, similar to the metrics for the grounding task on ScanRefer, we use Acc@0.25 and 0.5@0.5 measure accuracy based on the ST-IoU threshold. For VQA tasks, use the annotations of ScanQA and SQA3D along with their respective metrics for evaluation.
Table Evaluation results for video input.
grounding,
Video Grounding
ScunQA CIDEr
SQA3D
Method
Acc@0.25
A.c@0.
EM
Random
1.4
0.5
Ours
22.8
10.8
85.6
52,9
Upperbound 54.9
22.2

---

Performance Analysis. Table presents the evaluation results of video grounding and VQA tasks. For video grounding. we compute upper bound results assess the quality of object masks extracted by the video detector. Compared to the upper bound and random results, our method demonstrates strong grounding ability. Visualization results are provided Figure 4. The second example shows that the tracking-based video detector might lose track of an object after it has been out of sight for a prolonged period. This is primary reason for the low quality of the extracted object masks. Missing parts of the frames that contain the target object leads to the low Acc @0.5 result of the upper bound. For VQA tasks, we achieve comparable results to those using objects extracted from 3D inputs. This indicates that despite the lower quality of extracted objects from video input, approach constructing scene embeddings from sequences of object-level embeddings efficiently enhances overall scene comprehension, thereby improving QA performance ScanQA and SQA3D.
5
Conclusion
To enable efficient object referencing and grounding abilities in 3D MLLMs, this paper proposes modeling and interacting with 3D scenes the object level. It decomposes the input 3D scene into a set of object proposals assigned with object identifiers. Given the scarcity of scene-language data, we model the scene embeddings as a sequence of explicit object-level embeddings, derived from semantic-rich 2D and 3D representations. By using object identifiers, we transform diverse 3D scene-language tasks into unified guestion-answerrngg format. With minimal fine-tuning on all downstream tasks, our model significantly outperforms existing methods across various benchmarks.
Acknowledgments and Disclosure of Funding
This work was supported by National Key R&D Program China (2022DD0162000)
10

---

References
[1] P. Achlioptas, A. Abdelreheem, F. Xia, M. Elhoseiny, and L. Guibas. Referit3d: Neural listeners for fine-grained 3d object identification in real- cal-world scenes. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part I 16. pages 422 440. Springer, 2020.
[2] D. Azuma, T. Miyanishi, S. Kurita, and M. Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In proceedings the IEEE/CVF conference on computer vision and pattern recognition. pages 9919-19139, 2022.
[3] D. Cai, L. Zhao, J. Zhang, L. Sheng, and D. Xu. 3djeg: A unified framework for joint dense captioning and visual grounding on 3d point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 16464 16473,, 2022.
[4] D. Z. Chen, A. X. Chang, and M. Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In European conference on computer vision, pages 202-221. Springer, 2020.
[5] J. Chen, W. Luo, X. Wei, L. Ma, and W. Zhang. Ham: Hicrarchical attention model with high performance for 3d visual grounding. arXiv preprint ar 22: 12511,.//11,1 2, 2022.
[6] S. Chen, X. Chen, C. Zhang, M. Li, G. Yu, H. Fei, H. Zhu, J. Fan, and T. Chen. LI3da: Visual interactive instruction tuning for ommi-3d understanding, reasoning, and planning. arXiv preprint arXiv:2311. 2023.
[7] S. Chen, P.-... Guhur, M. Tapaswi, C. Schmid, and I. Laptev. Language conditioned spatial relation reasoning for 3d object grounding, Advances in Neural Information Processing Systems, 35:20522-200 20535, 2022.
[8] S. Chen, H. Zhu, X. Chen, Y. Loi, G. Yu, and T. Chen, End-to-end 3d dense captioning with vote2cap- detr. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 11124-11133, 2023.
[9] S. Chen, H. Zhu, M. Li. X. Chen, P. Guo, Y. Lei, Y. Gang, T. Li, and T. Chen. -etecaap-eet++: Decoupling localization and describing for end-to-end 3d dense captioning. IEEF Transactions on Pattern Analysis and Machine Intelligence, 2024.
[10] Y. Chen, S. Yang. H. Huang. T. Wang. R. Lyu, R. Xu, D. Lin, and J. Pang. Grounded 3d-llm with referent tokens. arXiv preprint arXiv:2405. 200244000777 2024.
[11] Y. Chen, Z. Yu, Y. Chen, S. Lan, A. Anandkumar, J. Jia, and J. M. Alvarez. Focalformer3d: focusing on hard instance for 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8394 8405, 2023.
[12] Z. Chen, A. Gholami, M. Nießner, and A. X. Chang. Scan2cap: Context-aware dense captioning in rgh-d scans. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3193-3203, 2021.
[13] H. K. Cheng, S. W. Oh, B. Price, A. Schwing, and Loe. Tracking anything with decoupled vidco segmentation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1316-1326, 2023.
[14] W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See http://vicuma. Imsys. org (accessed 14 April 2023), 2023.
[15] A. Chowdhery, S. Narang. J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
[16] A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 8828-5839) 2017.
[17] R. Dong, Z. Qi. I.. Zhang. J. Zhang, J. Sun, Z. Ge, L. Yi, and K. Ma. Autoencoders as cross-modal teachers: Can pretrained 2d image transformers help 3d representation learning? arXiv preprint arXiv:2212.08322 2022.
[18] R. Fu, J. Liu, X. Chen, Y. Nie, and W. Xiong, Scene- lmm Extending language model for 3d visual understanding and reasoning. arXiv preprint arXiv:2403. 14401, 2024.
[19] Z. Guo, R. Zhang, X. Zhu, Y. Tang, X. Ma, J. Han, K. Chen, P. Gao, X. Li, H. Li, et al. boint-bind & llint-lim: Aligning point cloud with multi mudii-yo for 3d understanding, generation, and instruction following. arXiv preprint 2rXXi:2309.00111 2023.
[20] J. Han, R. Zhang. W. Shao, P. Gao, P. Xu, H. Xiao, K. Zhang, C. Liu, S. Wen, Z. Guo, et al. Imagebind- Multi-modalit instruction tuning, arXiv preprint arXiv:2309.0m 039000 2023.
11

---

[21] Y. Hong, H. Zhen, P. Chen. S. Zheng. Y. Du, Z. Chen, and C. Gan. 3d-llm: Injecting the 3d world into large language models. arXiv preprint arXiv:2307.12988 12981, 2023.
[22] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu Y. Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank 2021.
adaptation of large language models. arXiv preprint
[23] J. Huang, S. Yong. X. Ma, X. Linghu, P. Li, Y. Wang, Q. Qi S.C. Zhu, B. Jia, and S. Huang. An embodied generalist agent in 3d world, arXiv preprint arXiv:2311.1287,, 2023.
[24] P.-H. Huang, H.-H. Lee, H.-T. Chen, and T.-L. Liu. Text-guided graph neural networks for referring 3d instance segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1610-1618, 2021.
[25] S. Huang, Y. Chen, Jia, and L. Wang. Multi-view transformer for 3d visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15524-15533, 2022.
[26] A. Jain, N. Gkanatsios, Mediratta, and K. Fragkiadaki. Bottom up top down detection transformers for language grounding in images and point clouds. In European Conference on Computer Vision, pages 417-433. Springer, 2022.
[27] L. Jiang. H. Zhao, Shi, Liu, C.-W. Fu, and Jia. Pointgroup: Dual-set point grouping for 3d instance segmentation. In Proceedings of the IEEE/CVE conference on computer vision and Puttern recognition. pages 4867-4876, 2020.
[28] Y. Jiao, S. Chen, Z. Jie, Chen, L. Ma, and Y.-G. Jiang. More: Multi-order relation mining for dense captioning in 3d scenes. In European Conference on Computer Vision, pages 528-545. Springer, 2022.
[29] Z. Jin, M. Hayat, Y. Yang. Y. Guo. and Y. Lei. Context-aware alignment and mutual masking for 3d- language pre-training. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 10984-10994, 2023.
[30] M. Kolodiazhnyi, A. Vorontsova, A. Konushin, and D. Rukhovich. Oneformer3d: One transformer for unified point cloud segmentation. arXiv preprint arXiv:2311. 14405, 2023.
[31] X. Lai, Z. Tian, Y. Chen, Y. Li, Y. Yuan, S. Liu, and J. Jia. Lisa: Reasoning segmentation via large language model. arXiv preprint arXiv:2308.0m 00692, 2023.
[32] B. Li, Y. Zhang, L. Chen, J. Wang. J. Yang, and Z. Liu. Otter: A multi-modal model with in-context 2023.
instruction tuming. arXiv preprint arXiv:2305.0377
[33] K. Li, Y. He, Y. Wang, Y. Li, W. Wang. P. Luo, Y. Wang. L. Wang, and Y. Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305. 230i::330000635,, 2023.
[34] H. Liu, C. Li, Y. Li, and Y. J. Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744 2023.
[35] H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. arXiv preprint arXiv:2304. 2023.
[36] M. Liu, R. Shi, K. Kuang, Y. Zhu, X. Li, S. Han, H. Cai, F. Porikli, and H. Su. Openshape: Scaling up 3d shape representation towards open- world understanding. arXiv preprint arXiv:2305. 2023.
[37] J. Luo, J. Fu, X. Kong, C. Gao, H. Ren, H. Shen, H. Xia, and S. Liu, 3d-sps: Single-stage 3d visual grounding via referred point progressive selection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 16454-16463, 2022.
[38] X. Ma, S. Yong, Z. Zheng, Q. Li, Y. Liang. S.-C. Zhu, and S. Huang. Sqa3d: Situated question answering in 3d scenes. arXiv preprint arXiv:2210.07474, 2022.
[39] R. OpenAT. Gpt-4 technical report. arxiv
View in Article, 2, 2023.
[40] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint
arXiv:2304.07193, 2023,
[41] Y. Pang, W. Wang, F. E. Tay, W. Liu, Y. Tian, and L. Yuan. Masked autoencoders for point cloud sulf-supervised learning. In European conference on computer vision, pages 604 621. Springer, 2022.
[42] K. Papineni, Roukns, T. Ward, and W.-J. 7hu. Blew: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 31 1-318, 2002.
[43] M. Parelli, A. Delitzas, N. Hars, G. Vlassis, S. Anagnostidis, G. Bachmann, and T. Hofmann. Clip-guided vision-language pre-training for question answering in 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5606-5611, 2023.
[44] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin,
J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748 8763. PMLR, 2021.
12

---

[45] J. Schult, F. Engelmann, A. Hermans, O. Litany. S. Tang. and B. Leibe. Mask3d: Mask transformer for 3d semantic instance segmentation. In 2023 IEEE International Conference on Robotics and Automation (ICRA). pages 8216-8223. IEEE, 2023.
[46] Y. Shen, Z. Geng, Y. Yuan, Y. Lin, Z. Liu, C. Wang, H. Hu, N. Zheng, and B. Guo. V-detr: Detr with vertex relative position encoding for 3d object detection, arXiv preprint arXiv:2308.04400) 2023.
[47] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro,
F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 2xXiv:2302.13977 2023.
[48] O. Unal, C. Sakaridis, S. Saha, F. Yu, and L. Van Gool. Three ways to improve verbo-visual fusion for dense 3d visual grounding. arXiv preprint arXiv:2309. 0000000461.1.. 2023.
[49] R. Vedantam, C. Lawrence Zitnick, and D. Parikh. Cider: Consensus- based image description evaluation. In Proceedings of the IEEF conference on computer vision and pattern recognition, pages 4566-4575, 2015.
[50] T. Vu, K. Kim, T. M. Luu, T. Nguyen, and C. D. Yoo. Softgroup for 3d instance segmentation on point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2708 2717, 2022.
[51] H. Wang. Q. Liu, X. Yue, J. LLsenby, and M. J. Kusner. Unsupervised point cloud pre-training via occlusion completion. In Proceedings of the IEEF/CVF international conference computer vision, pages 9782-9792, 2021.
[52] Z. Wang. H. Huang, Y. Zhao, L. Li, X. Cheng, Y. Zhu, A. Yin, and Z. Zhao. 3drp-net: 3d relative position-aware network for 3d visual grounding. arXiv preprint arXiv:2307. 13363, 2023.
[53] Z. Wang. H. Huang. Y. Zhao, L. Li, X. Cheng, Y. Zhu, A. Yin, and Zhao. Distilling coarse-tor to--tto-ffii fine semantic matching knowledge for weakly supervised 3d visual grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2662-2671, 2023.
[54] Z. Wang, H. Huang, Y. Zhao, Z. Zhang, and Z. Zhao. Chat-3d: Data-efficiently tuning large language model for universal dialogue of scenes. arXiv preprint arXiv:2308.08760 2023.
[55] Z. Wang, Z. Zhang, X. Cheng, R. Huang, L. Liu, Z. Ye, H. Huang. Y. Zhao, T. Jin, P. Gao, et al. Freebind: Free lunch in unitied multimodal space via knowledge fusion. In Forty-first International Conference on Machine Learning.
[56] Z. Wang, Z. Zhang, L. Liu, Y. Zhao, H. Huang, T. Jin, and Z. Zhao. Extending multi- modal contrastive 2023.
representations. arXiv preprint arXiv:2310.08880
[57] Z. Wang. Z. Zhang. H. Zhang, I.. Liu, R. Huang, X. Cheng, H. Zhao, and Z. Zhao. Omnibind: Large-scale omni multimodal representation via binding spaces. arXiv preprint 2407 1185, 2024.
[58] Z. Wang. Y. Zhao, X. Cheng, H. Huang. Liu, L. Tang, L. Li, Y. Wang, A. Yin, Z. Zhang, et al. Connecting multi- modal contrastive representations. arXiv preprint arXiv:2305. 14381, 2023.
[59] T.-.. Wu, S.-Y. Huang, and Y.-C. F. Wang. Dora: 3d visual grounding with order-aware referring. arXiv preprint 2rXiv:2403.16533 2024.
[60] Y. Wu, X. Cheng, R. Zhang, 7. Cheng, and Zhang. Eda: Explicit text-decoupling and dense alignment for 3d visual grounding. In Proceedings of the IERE/CVF Conference on Computer Vision and Pattern Recognition, pages 19231-19242 2023.
[61] R. Xu, X. Wang, T. Wang, Y. Chen, Pang, and Lin. Pointllm: Empowering large language models to understand point clouds. arXiv preprint arXiv:2308. 16911, 2023.
[62] L. Xue, M. Gao, C. Xing. R. Martin Martín, J. Wu, C. Xiong, R. Xu, J. C. Niebles, and S. Savarese. Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1179-1189, 2023.
[63] J. Yang, X. Chen, S. Qian, N. Madaan, M. Iyengar, D. F. Fouhey, and J. Chai. Lim-grounder: Open- vocabulary 3d visual grounding with large language model as an agent. arXiv preprint (xXXxx3309.23111 2023.
[64] Z. Yang. S. Zhang, L. Wang, and J. Luo. Sat 2d semantics assisted training for 3d visual grounding. In Proceedings of the IEFE/CVF International Conference on Computer Vision, pages 1856-1866. 2021.
[65] Q. Yc, H. Xu, G. Xu, J. Ye, M. Yan, Y. Zhou, J. Wang, A. Hu, P. Shi, Y. Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint ar 2304::2304.14178. 2023.
[66] Z. Yin, J. Wang. J. Cao, Z. Shi, D. Liu, M. Li, X. Huang. Z. Wang, L. Sheng. L. Bai, et al. Lamm: aanguage-assisted multi modal instruction tuning dataset, framework, and benchmark. Advances in Neural Information Processing Systems, 36, 2024.
13

---

[67] H. You, H. Zhang. Z. Gan, X. Du, B. Zhang, Z. Wang, L. Cao. S.-E. Chang, and Y. Yang. Ferret: Refer and ground anything anywhere any granularity. arXiv preprint 1rX0v:2310.07704, 2023.
[68] X. Yu, L. Tang, Y. Rao. T. Huang, J. Zhou, and J. Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 19313 19322,. 2022.
[69] Z. Yuan, X. Yan, Y. Liao, Y. Guo, G. Li. S. Cui, and Z. Li. X-trans2cap: Cross-modal knowledge transfer using transformer for 3d dense captioning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8563-8573, 2022.
[70] Z. Yuan, X. Yan, Y. Liao, R. Zhang. S. Wang. Z. Li, and S. Cui. Instancerefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual referring. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1791-1800, 2021.
[71] D. Zhang, I. Liang, H. Yang. Z. Zou, X. Ye, Z., Liu, and Bai. Sam3d: Zero-shot 3d object detection via segment anything model. arXiv preprint arXiv:2306.02245. 2023.
[72] H. Zhang, H. You, P. Dufter, B. Zhang, C. Chen, H.-Y. Chen, T.-J. Fu, W. Y. Wang, S.-F. Chang. Z. Gan, et al. Ferret- v2: An improved baseline for referring and grounding with large language models. arXiv preprint arXiv:2404.0799
2024.
[73] R. Zhang. Z. Guo, P. Gao, R. Fang. B. Zhao, D. Wang, Y. Qiao, and H. Li. Point-m2ae: multi-scale masked autoencoders for hierarchical point cloud pre-training. Advances neural information processing systems. 35:27061-270744 2022.
[74] R. Zhang. L. Wang. Y. Qiao, P. Gao, and H. Li. Learning 3d representations from 2d pre-trained models via image-to-poing masked autoencoders. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 27780, 2023.
[75] Y. Zhang, Z. Gong, and A. X. Chang. Multi3drefer: Grounding text description to multiple 3d objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15225-15536,, 2023.
[76] L. Zhao, D. Cai, L. Sheng, and D. Xu. 3dvg-transformer: Relation modeling for visual grounding on point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2928-2937, 2021.
[77] Y. Zhao, Z. Lin, D. Zhou, Z. Huang, J. Feng, and B. Kang. Bubogpt: Enabling visual grounding in multi-modal lms.. arXiv preprint arXiv:2307.0m 08581,, 2023.
[78] Zhou, Wang. B. Ma. Y.-S. Liu, T. Huang, and X. Wang. Uni3d: Exploring unified representation scale. arXiv preprint 23X0022110.0677. 2023.
[79] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 2023.
[80] Z. Zhu, X. Ma, Y. Chen, Z. Deng, S. Huang, and Q. Li. 3d-vista: Pre-trained transformer for 3d vision and text alignment. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages
2911-2921, 2023.
14

---

Table 6: Performance comparison on the validation set of ScanRefer [4].
Unique
Multiple
Overall
Method
Venue
Acc@0.25
A.c@0.5
Acc@0.25
Acc@0.5
Acc@0.25
Acc@0.5
ScanRefer [4]
BCCV20
53.51
32.73
21.11
41.19
27.40
TGNN [24]
AAAT2I
68.61
56.80
29.84
23.18
37.37
29.70
SAT [64]
ICCV21
73.21
50.83
37.64
25.16
44.54
30.14
InstanceRefer [70]
ICCV21
75.72
64.66
29.41
22.99
38.40
31,08
DDVG-Transformer [76]
ICCV21
81.93
60.64
39.30
28.42
47.57
34.67
MVT [25]
CVPR22
77.67
66.45
31.92
25.26
40.80
33.26
3D-SPS [37]
CVPR22
84.12
66.72
40.32
29.82
48.82
36.98
ViLSDRel [7]
NeurIPS22
81.58
68.62
40,30
30.71
47.94
37.73
3DICG [3]
CVPR22
83.47
64.34
41.39
30.82
49.56
37.33
D3Net [?]
ECCV22
72.04
30.05
37.87
BUTD-DETR (26)
BCCV22
35.1
52.2
39.8
HAM [5]
ArXiv22
79.24
67.86
41.46
34.03
48.79
40.60
3DRP-Net
EMNLP23
83.13
67.74
42.14
31.95
50.10
38.90
3D-VLP [29]
CVPR23
84.23
64.61
43.51
33.41
51.41
39,46
EDA [60]
CVPR23
85.76
68.57
49.13
37.64
54.59
42.26
M3DRef-CLIP [75]
ICCV23
85.3
77.2
43.8
36,8
51.9
44.7
3D- VisTA [80]
ICCV23
81.6
75.1
43.7
39.1
50.6
45.8
ConcreteNet [48]
ECCV24
86.40
82.05
42.41
38.39
50.61
46.53
DOrA [59]
ArXiv24
52.80
44.80
Ours
89.59
82.49
47.78
42.90
55.52
50.23
A Implementation Details
For the 3D point cloud input, utilize the 3D instance segmentation model Mask3D [45] to extract 100 object proposals. We then employ the pre-trained encoder Uni3D [78] to obtain 3D features and DINOv2 [40] for 2D features. Both the 3D-language projector (-(() and the 2D-Janguage projector f-(-) are three-layer MLPs. For the video input, we use the open-vocabulary video segmentation model DEVA [13] extract object proposals, with average object number of 48. We choose the Vicuna-7B-a 7B-v1.5 model [14] as the language model, which is based on LLaMA 2 [47]. We fine-tune the language model using LoRA [22], with the rank set 16. The base learning rate is
5c-6 with cosine annealing schedule, and the batch size is 32. The training takes 2 epochs and the total training step 3200. The entire training process takes approximately hours on 4 NVIDIA A100 GPUs.
2D Encoder. We describe the method for deriving 2D representations from the projected input masks and multi-view images. The 2D model DINOv2 [40] computes 257 x 1024 feature vector for each image, comprising 1 1024 CLS feature and 256 1024 patch features. These patch features represent local areas in the images divided into 16x 16 patches. For each object mask within an image, we extract patch features only where the patch intersects with the mask. We then average these extracted patch features along the patch axis to obtain a 1024 feature vector per image. For patch features sourced from multi-view images, we average them weighted by the mask size on each image. We apply this procedure to extract DINOv2 features from multi-view images for each object, thus generating the final object representations. introduce a "Single" approach where we select the image with the largest mask for the current object and extract features solely from this image.
our ablation study, described in Section 4.3, we
Quantitative Results
For the five datasets, employ evaluation metrics proposed in their respective original publica- tions. We compare our model against a comprehensive array of state-of-the-art methods, as detailed in Table for SeanRefer [4], Table 7 for Multi3DRefer [75], Table for Scan2Cap [12], Table 9 for ScanQA [2], and Table 10 for SQA3D [38].
C Qualitative Analysis
In this section, we perform a comprehensive qualitative analysis of ScanQA [2] and ScanRefer [4]. Additional analysis of other datasets will be included in the final version.
15

---

Table 7: Performance comparison on the validation set of Melei [75].
Method
Venue
2T wio
F1
21
Fi
FL.0.22
ST
F180.5
D
wT F1
F1G05
F1
MT
F1
F16:0.25
ALL
F1
3DVG [76]
ICCV21
$7.1
45.8
27.5
16.7
26.5
25.5
D3Net (Grounding)
ECCV22
3DJCG (Gounding)
CVPR22
94.1
66.9
26.0
16.7
26.2
26.6
M3DRef CLDDRf-CLIP [75]
JCCV25
$1.8
39.4
53.5
47.8
34.5
30.6
43.6
37.9
42.8
38.4
Ours
90.3
62.6
82,9
75,9
49.1
44.5
45.7
41.1
57.1
52.4
Table 8: Performance comparison on the validation set of Scan2Cap [12].
@0.25
@0.5
Method
Venue
C
B-4
M
R
C
B-4
M
R
Scan2Cap [12]
CVPR21
56.82
34.18
26.29
55.27
39.08
23.32
21.97
44.48
3DJCG [3]
CVPR22
64.70
40.17
27.66
59.23
49.48
31.03
24.22
50.80
X A-Trans2Cap [69]
CVPR22
61.83
35.65
26.61
54.70
43.87
25.05
22.46
45.28
D3Net [?]
ECCV22
62.64
35.68
25.72
53.90
3D-VLP [29]
CVPR23
70.73
41.03
28.14
59.72
54.94
32.31
24.83
51.51
Vote2Cap-DEr CaCCCCap-TRRRII [8]
CVPR23
71.45
39.34
28.25
59.33
61.81
34.46
26.22
54.40
3D-VisTA
ICCV23
71.0
36.5
28.4
57.6
66.9
34.0
27.1
54.3
LL3DA
CVPR24
74.17
41.41
27.76
59.53
65.19
36.79
25.97
55.06
LEO
ICML24
68.4
36.9
27.7
57.8
Vole2Cap-EETR++ [9]
T-PAMI24
76.36
41.37
28.70
60.00
67.58
37.05
26.89
55.64
Ours
81.94
38.23
29.01
60.57
77.19
36.34
28.01
58.12
C.1 3D Question Answering
We present four evaluation results for 3D question answering on ScanQA [2] dataset, as shown in Figure In this dataset, both the input and the output do contain object referencing. This dataset does not include object referencing in either the input or output. Example (a) necessitates the model's ability perceive an object's 's appearance, specifically its color. Example (b) demands that the model identify target object based on a descriptive prompt. Example (c) involves the model describing position of a target object, while example (d) tests the model's capability to count objects. Our model demonstrates relatively high performance with the fourth, particularly when the count of target objects large. Accurately perceiving and localizing each object is essential for the counting task; failure to do results in incorrect answers. This challenge persists both our method and previous methods. Moreover, the inferior annotation quality within the ScanQA dataset exacerbates this issue. For instance, the question in example (d), "How many black chairs are on the right?" lacks precise definition of "right", leading to potential confusion for the model.
the first three types of tasks but often struggles
C.2 3D Visual Grounding
We present six evaluation results for 3D visual grounding on ScanRefer [4] dataset, as illustrated in Figure This task challenges the model localize a target object based on descriptive prompt. For simpler scenarios, such as those in examples (a) and (b), our model performs adequately. However, it struggles with the remaining four examples for various reasons.
In example (c), the model is tasked with identifying a chair "against the wall" but erroneously selects chair that is positioned described. This highlights deficiency in model's understanding of interior structural elements like walls, ceilings, and floors. Despite the presence of segmented annotations for these surfaces, they are typically not utilized in training because they are not considered objects per se. This limitation is likely shared by many current methods. Future work is necessary enhance the model's recognition of these elements, given their significance in comprehending the entirety of 3D scene. In example (d), the challenge involves identifying two pillows "placed on the armchair'. one black and the other white. The model correctly locates a pillow on the armchair but fails to distinguish it by color. Example (e) presents a scenario where the model confuses a window for door, likely due to their similar appearances and the often incomplete nature of the input point cloud. In example (f), the model's selection meets the description, illustrating flaw in the SeanRefer dataset annotations: some descriptions may correspond to multiple objects, rendering them ineffective for evaluating the visual grounding of single object.
16

---

Table 9: Performance comparison on the validation set of ScanQA [2].
Method
Venue
EM@1
B-2
B-3
B-4
ROUGE-L
METEOR
CIDEr
SPICE
ScanQA [2]
CVPR22
21.05
30.24
20.40
15.11
10.08
33.33
13.14
64.86
13.43
3D-VLP [29]
CVPR23
21.65
30.53
21.33
16.67
11.15
34.51
13.53
14.18
3D-LLMM [21]
NeurIPS23
20.5
39.3
25.2
18.4
12.0
35.7
14.5
69.4
LL3DA [6]
CVPR24
13.53
37.31
15.88
76.79
LEO [23]
ICML24
11.5
39.3
16.2
80.0
Scene-LLM [18]
ArXiv24
27.2
43,6
26.8
19.1
12.0
40.0
16.6
Ours
21.62
43,20
29.06
20.57
14.31
41.56
18.00
87.70
20.44
Table 10: Performance comparison on the test set of SQA3D [38].
Test set
Method
Venue
Avg.
What
How
Can
Which
Others
SQA3D [38]
ICLR23
31.64
63.80
46.02
69.53
43.87
45.34
46.58
3D- VisTA [80]
ICCV23
34.8
63.3
45.4
69.8
47.2
48.1
48.5
Scene-LLM [18]
ArXiv24
40.9
69.1
45.0
70.8
47.2
52.3
54.2
Ours
45.38
67.02
52.04
69.52
49.85
54.95
54.57
What attached wall just above the sink counter?
Q: What color the filing cabinet on the floor?
White.
Paper dispenser.
(4)
what part the brown piano located?
Q: many chuirs the right?
A: Left corner.
A: Eleven.
X (GT: Twelve.)
Figure 5: Visualization results of 3D question answering on ScanQA [2].
17

---

deseristion, givan betwern the bathtuh and the countes; please pruvide the ID of object closdly mutches this descriptin.
description, the cheir on the left side." please provide the ID of the object closdly mutiches this description.
A: +00/062
Ar 10DJM0>
Q: According description "There black cotton pillow, Placed oo the armehwis, please provile the ID object closely description.
According the piven deserigtion, office chair againet the middle." please provide ID the object that cloacly mutches this descraption.
At +CIB0015
A: +0
(c)
(d)
Accoeding deseription, open door lends into the buthroom across foom some bothroom cabunets" closely description.
According
description,
It at the für end the table" please provide the ID of closely muiches this descriptio.
the the
A: -OBI066
A: =0RR177
Figure 6: Visualization results of 3D visual grounding on ScanRefer [4]. The predicted blue box transformed from the segmented point cloud of the predicted object (ID). The green ground truth box is provided when the prediction wrong.
This black chair. It the chair forthest from the door the side the table closest the hanging
Ours
picture affixed to left of the
Figure 7: Additional visualization results of video grounding for video input.
18

---

Table 11: Comparison of input/output formats. The of object identifiers make it possible to solve tasks with single or even multiple object reference in the input/output.
Query
Answer
Tisk
Expert Models
50 MELMs
Ours (Exemple)
Narmal test
Nenrol vor
VQA
TD VisTA 3D-VPP
30-LLM, Cau--3D.
LLSDA, LED
Q: Uoo many meb cae thene somm? Three
Noumal text
Reloriing oljos
Visual Grouzding
VDTA, MSDited
Grousder
1D-1MM, 110,
Q.
nest to whitchoord
Cast-ll, LLIDA
Refering to engli
Nonnal text
Object Copcioning
ViTA JD-VLP
30LLLM, Clay-
Q: Deseibe -ost0s-
A: The is yellow mecingle with foor II
MSDitel MVT
LLM Gronder
is the clonest toble to the daoe
MSDRe
Q: Find
objocs that sutisiy the desor prion "This
Nocmal texi
Reforing miliipe objects
Viaud Geounding
3D-V 3D-VLP.
X
A: They are -0B1000 -0BM007 and -DB.000
is komp It's boonse in color and sis os bille"
Greeded Seane
Captaning
X
A: There is sibke (-080054-) with tne lanaa
Q Desorbe Bee scere.
(-08 100000000000 and -0B30065) on
Q
Retering to muiliple oujocs
Nonnal lox
Multi ole
Summary
*
x
-0B0004- sove??
talles which likey interdel fir coliburaiiv warkopacz.
A:
Befaring to mulippe oijers
single otgeen
Mulippic Chrice
QN
X
whick one the descit the doee?
A: K0
Q: comuntts -onusses and -D000125
Refaring mutiile obyen
Referiig muttplo obyerts
Complex Pisseing
x
<Din 1220 from momm What doaald dk?
A: Find,
two (-0B83007 and
<DB (00000-- from thil (-0820004)) and the backprak (-083030-) from tible (-080022))...
D Input/Output Format Comparison
We present comparative examples of input/output formats in Figure 11. We juxtapose our model against leading expert models (3D-VisTA [80]. 3D-VLP [29]. 3DICG [3], M3DRef [75], and MVT [25)) as well as 3D MLLMs (3D-LLM [21]. Chat-3D [54], LLM-Grounder [63]. LL3DA [6], and LEO [23]). We enumerate several common tasks for varied combinations of input and output formats and provide examples of interactions using object identifiers with our model. It is necessary to acknowledge that some responses shown here are not directly produced by our model due to lack of adequate data for fine-tuning on these tasks, such as multi-object summaries, multiple-choiee QA, and complex planning. The comparison underscores the potential for employing object identifiers to address complex tasks involving single or multiple object references in the input/output, representing a significant enhancement over previous methods restricted to simple tasks with basic formats.
E Limitation and Societal Impact
Limitation. The primary limitation of our method stems from its reliance pre-trained foundation models, including 2D/3D detectors and 2D/3D encoders. our experiments, we froze these models under the presumption of their robust performance. Yet, they occasionally produce incorrect results, as evidenced by the failure cases detailed in the supplementary material. To establish an end-to-end pipeline and enhance model performance further, future work will involve integrating these foundation models into the entire training process.
Another significant challenge is the scarcity of data. While the development of 2D vision-language models has benefited from the availability of millions of image-text pairs for pre-training, the 3D- language domain grapples with dearth of corresponding data, adversely affecting the alignment between 3D and language spaces. This issue is particularly acute in 3D scene understanding, where the limited availability of scene-language pairs restricts training by failing to provide adequate spatial relationship data. Despite our model's 's impressive performance in various evaluations, it occasionally misclassifies objects, notably in underrepresented classes such as "hair dryer" and "soap dish." Future initiatives should aim to enhance data volume to bolster the 3D MLLM's scene understanding capabilities.
Societal Impact. Our model has achieved consistent performance improvements in multiple tasks related to 3D indoor scene understanding, which is beneficial and potentially applicable to downstream
19

---

Table 12: Prompt templates for different tasks. <Description> is replaced by object's description in SaaaReet/MuuliiDRRefrrSsmm2.C... <Question>> and <Answer> denotes the question and answer in ScanQA/SQA3D. <Situation> is the situation in SQA3D.
Single-object Visual Grounding (ScanRefer):
User:
What is the ID the object that matches the description "<Description>"
Assistant: <OBJXXX>
Multi-object Visual Grounding (ScanRefer):
User: Are there objects described as "<Description>>? there are, please provide the IDs for those objects.
Assistant: Yes. <OBJXXX>, <OBJXXX>, and <OBJXXX>, (for matched cases)
Assistant: No. (for unmatched cases)
Dense Captioning (Scan2Cap):
User: Provide a detailed description of the appearance of <OBJXXX> before analyzing its spatial connections with other elements in the Assistant: <Description>
Visual Question Answering (ScanQA):
User: <Question> The answer should be a phrase or a single word. Assistant: <Answer>
Situated Question Answering (SQA3D):
User:
<Situation> <Question> The answer should be phrase or single word. Assistant: <Answer>
applications. However, due the training data covering all possible scenarios, the model may experience hallucinations during prediction, thereby posing some risks in system applications.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our contribution of building 3D MLLM with object identifiers is clearly claimed in both abstract and introduction (Section 1).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed in Appendix E.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
21

---

The paper focuses on practical aspects of training neural networks, and no new theoretical results are included.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The implementation details in given in Appendix The model design, training data, input/output format is illustrated in Section 3.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
22

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
We provide the code in the supplementary material with README which includes instructions for preparing data, training, and evaluating. Also, we will release the code to public after careful organization.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/ mablic/guider /t/g//gds//ooeuuupee CoodeSubmissionPoo onnooocy)yy) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /nips. wwcp/pbllccgguiiiicec gwwdes/ies///// httpc////giiiee//edeuuuuiiiiiii wwweeesummisiioooooo..
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We provide the implementation details in Appendix A. The training and test details are included in the code and instructions in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We keep the same basic training settings (learning rate, batch size, training steps, random seed) across all the experiments reported in Section 4 to produce meaningful ablation study. However, we do not report error bars due to the limited time and resources available for training the LLM multiple times.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
23

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report the required hardware resources (GPU type, GPU number, and execution time) in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:/ //nerrips/// cc
Answer: [Yes]
Justification: We've reviewed the NeurIPS Code of Ethics and we make sure the research conforms with the Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the societal impact in Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The model in our paper is based on an open-source large language model, and our fine-tuning procedure does not introduce additional risks. Users are encouraged to adhere to the safeguards provided by the open-source LLM.
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The creators of the used datasets and models in our paper are properly eited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
25

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. cowc dataetts has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
The code/model provided in the supplementary material is documented with detailed instructions.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
26

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "KcyiAq3kxx", "reviewer": "Reviewer_2kgC", "summary": "This paper aims to enhance the efficiency in interpreting individual object instances and improve referencing and grounding capabilities for intricate scene comprehension. The method decomposes the input 3D scene into object identifier tokens. Experimental results on 3D scene-language datasets demonstrate the effectiveness of the proposed approach.", "questions": "1. More ablations to show the effectiveness of object identifier, such as the order of  identifiers, only combining a 3D object token embedding and a 2D object token embedding for each object without the specifical token for identifier.\n\n2. More ablations about the LLM used for 3D-VL learning, such  as different types of LLMs , different sizes of LLM, multimodal large models such as LLAVA[b] or other 3DLLM[22]\n\n3.The details about the training set, such as the scale of 3D, 2D and language, the statistics of identifiers . \n\n4.The details about the inputs at inference,  like will all objects of a scene be combined as input, and how to combine?\n\n[b] Visual Instruction Tuning, NeurIPs2023", "limitations": "The authors provide the limitations about the reliance on pre-trained models, data scarcity, and the experience hallucinations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tThe paper is well-written and easy to follow.\n\n2.\tThe experimental results on 3D scene-language datasets shows the performance improvement and effectiveness of the proposed method.\n\n3.\tThe training schema of the model is single-stage yet effective on downstream task.", "weaknesses": "1．\tThe authors do not provide the details about detectors, encoders,  multi-modal inputs and LLMs used in other methods in the Table 1. These designs has already built a high-performance baseline, for example, the baseline results in Table 2 have exceeded other methods in the Table 1. It is difficult to judge the fairness of the comparisons in  Table 1.\n\n2．\tThis paper lacks the performance on ScanQA test set.\n\n3．\tThe ablations are insufficient. The paper lacks the ablations about different size of LLM, training and fine-tuning schema.\n\n4．\tThe paper lacks the computation and time cost about the proposed method.\n\n5．\tThis paper does not provide sufficient discussion with object-centric representation learning methods on 3D vision-language. For example,[a] explored the object-centric representation learning with contrastive learning.  And the object-level tokens are used in LEO[22], 3D-LLM[20], etc.\n\n[a] Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understanding, AAAI2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## W1: Details about multi-modal inputs, detectors, encoders, and LLMs used in other methods.\n\nPlease refer to Table 1 in the attached PDF in “Author Rebuttal”.\n\n## W2: Results on ScanQA test set.\n\n|  | Test w/ object |  |  |  | Test w/o object |  |  |  |\n|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| **Method** | **CIDEr** | **B-4** | **METEOR** | **ROUGE** | **CIDEr** | **B-4** | **METEOR** | **ROUGE** |\n| ScanQA | 67.29 | 12.04 | 13.55 | 34.34 | 60.24 | 10.75 | 12.59 | 31.09 |\n| Multi-CLIP | 68.70 | 12.65 | 13.97 | 35.46 | 63.20 | 12.87 | 13.36 | 32.61 |\n| 3D-VLP | 70.18 | 11.23 | 14.16 | 35.97 | 63.40 | **15.84** | 13.13 | 31.79 |\n| 3D-VisTA | 76.6 | **16.0** | 15.2 | 38.6 | 62.6 | 11.9 | 12.9 | 32.8 |\n| 3D-LLM | 69.6 | 11.6 | 14.9 | 35.3 | - | - | - | - |\n| LL3DA | 78.16 | 13.97 | 16.38 | 38.15 | 70.29 | 12.19 | 14.85 | 35.17 |\n| **Ours** | **94.04** | 14.38 | **19.65** | **44.53** | **79.79** | 11.38 | **17.10** | **39.34** |\n\n## W3 & Q2: Ablations about LLMs and training schema.\n\n|  |  | ScanRefer | Multi3dRefer | Scan2Cap | ScanQA | SQA3D |\n|---|---|:---:|:---:|:---:|:---:|:---:|\n| **LLM** | **Training Schema** | **Acc\\@0.5** | **F1\\@0.5** | **C\\@0.5** | **CIDEr** | **EM** |\n| OPT-1.3B | one-stage | 47.6 | 48.3 | 75.7 | 86.2 | 54.3 |\n| Tiny-Vicuna-1B | one-stage | 49.5 | 50.1 | 76.2 | 86.7 | 54.5 |\n| Vicuna-7B | one-stage | 50.2 | 52.4 | 77.1 | 88.4 | 55.5 |\n| Vicuna-13B | one-stage | 49.9 | 51.7 | **78.2** | **89.1** | **55.8** |\n| Vicuna-7B | two-stage | **51.1** | **53.1** | 73.7 | 85.6 | 55.3 |\n\n1) **Size of LLM**: We compared various sizes of LLMs: Vicuna 1B, 7B, and 13B. The 7B model achieves the best performance on grounding tasks, while the 13B model excels in captioning and QA tasks. The 13B model does not show significant performance gains over the 7B model, suggesting that the current task scope and data scale do not challenge larger LLMs. \n\n2) **Type of LLM**: We tested the OPT-1.3B, which performs slightly worse than the Vicuna-1B of similar size. As for multimodal large models like LLaVA and LEO, basically they use the same LLM backbone (Vicuna-7B) and the projection layer (MLP) as ours. However, directly adapting their pre-trained weights to our model is unsuitable because we use different 2D or 3D representations.\n\n3) **Training schema**: We compare one-stage training with two-stage training (fine-tuning on each dataset). Two-stage training improves performance on grounding tasks but significantly decreases performance on captioning and QA tasks. The performance decline in captioning and QA tasks may be due to these tasks being easier to converge, leading to overfitting during further fine-tuning.\n\n## W4: Computation and time cost.\n\n Method | Data Preparation (per scene)  | GPU Usage | Training Time \n---|---|---|---\n 3D-LLM (BLIP-2) | ~ 15 min | 64 * V100 | ~ 1 day \n 3D-LLM (Flamingo) | ~ 15 min | 8 * A100 | ~ 7 days \n Ours | < 1 min | 4 * A100 | ~ 8 hours \n\nCompared to 3D-LLM[22], our model's simpler design significantly reduces computation and time costs.\n\n## W5: Discussion about object-centric representation.\n\nFirstly, it’s important to note that the object-centric representation is not claimed as a contribution in the paper. We state that our study is based on well-trained encoders, and our contribution is on how to incorporate a sequence of {object identifier, object features} into the LLM, which can solve various 3D scene-language tasks in unified formats. More comparisons among different object-centric representations can be left as a future work.\n\n## Q1: Importance of object identifiers.\n\n|  | ScanRefer | Multi3dRefer | Scan2Cap | ScanQA | SQA3D |\n|---|:---:|:---:|:---:|:---:|:---:|\n|  | **Acc\\@0.5** | **F1\\@0.5** | **C\\@0.5** | **CIDEr** | **EM** |\n| fixed order | 49.6 | 51.5 | 76.2 | **88.7** | 55.4 |\n| w/o obj identifiers | 22.8 | 25.6 | 48.9 | 79.8 | 50.0 |\n| **Ours** | **50.2** | **52.4** | **77.1** | 88.4 | **55.5** |\n\n**Order of Object Identifiers**: In our actual implementation, we randomize the order of object identifiers during training to minimize the influence of any inherent order distribution. Our results show that using a fixed order of identifiers yields slightly poorer performance compared to using a random order.\n\n**Removing Object Identifiers**: Object identifiers are crucial to our model's design. They allow users to reference objects for tasks such as 3D dense captioning (Scan2Cap), while the LLM uses these identifiers for grounding objects in both single object grounding (ScanRefer) and multiple object grounding (Multi3DRefer). Without object identifiers, it is necessary to adopt an alternative method for grounding and dense captioning. Following 3D-LLM[22], we add special location tokens to represent bounding boxes in the LLM. Thus, the model can output bounding boxes for grounding tasks and input bounding boxes for dense captioning. As shown in the table above, training the model without object identifiers reveals a significant decline in performance, particularly in grounding and captioning tasks.\n\n## Q3: Details about training data.\n\n Data type |  Size\n---|---\n 3D Scene | 1201 \n Point Cloud / Scene | 145K \n Images / Scene | ~ 80 \n Image Resolution | 640*480 \n Language | 155K \n Identifiers | 100\n\nAs a comparison, LEO[20] used 1.2M language training data, and 3D-LLM[22] used 700K, which are several times more than our training data (155K).\n\n## Q4: Details about the inputs at inference.\n\nAll objects detected by the 3D detector are combined as input. For each object, the extracted 3D and 2D representations are projected into the LLM’s embedding space, becoming 3D and 2D tokens. Consequently, each object is represented by three tokens: an identifier token, a 3D token, and a 2D token. As illustrated in Figure 2, these objects form a token sequence of length 3$n$, where $n$ is the number of objects. For more details, refer to Section 3.3 for the prompt template and Section 3.2 for feature extraction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y26WNQN9MO", "reviewer": "Reviewer_n84p", "summary": "This paper proposes a 3D MLLM that can understand the 3D environment at the object level. The proposed work designs object identifiers that are projected into language token space and can be understood by LLMs, which unifies several 3D scene understanding tasks into the same format. These object identifiers give a more natural way for humans and MLLM to reference specific objects of interest in a 3D scene. The paper extracts features from 3D scenes using several 2D and 3D pretrained models and finetunes the MLLM on 3D scene understanding tasks. Experiments were conducted on several benchmarks based on ScanNet and the results show the proposed method achieves state-of-the-art performance, surpassing prior arts by a large margin.", "questions": "1. Ln 183 mentions that DINOv2 has superior handling of local features within images. Why and how does this matter in the 3D scene understanding task? How does it influence the final performance? The relevant ablations study is currently lacking.", "limitations": "The authors provide a discussion on limitations and societal impact in the appendix. Yet they don't provide qualitative examples or discussion on the failure cases, which I highlighted in the weakness section.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper decomposes a 3D scene into objects and develops object tokens to represent them for interaction with LLMs. This is a more natural way for people and LLMs to refer to object entities in the scenes. \n2. The paper conducts extensive experiments across several 3D scene understanding tasks (Grounding, VQA, and Captioning), showing the versatility of the proposed method. The reported results show that the proposed method achieves state-of-the-art performance and outperforms the existing expert and LLm-based models by large margins. \n3. The paper also conducts a study on video input using a 2D video instance tracking model, showing that the proposed method can still work without the 3D model. \n4. The paper is well-written and the code is provided in the supplementary material.", "weaknesses": "1. The performance of the proposed method is highly reliant on the ability of the pretrained 2D and 3D models (Mask3D, DINOv2, DEVA). Specifically, since the scene is represented as discrete object tokens, some fine-grained information existing in the full 3D scenes is lost. For example, if the 3D segmentation model wrongly merges 2 chairs into one, there is no way for the proposed method to recover. In this regard, I encourage the authors to provide some case studies about how the model fails (or succeeds) in this way. \n2. Although multiple tasks are tested, they are all based on the same underlying dataset, ScanNet. Thus it's unclear how the proposed method works on other datasets. While this is maybe because the ScanNet has most established benchmarks on these 3D scene understanding tasks, I still want to see how the proposed method works on other datasets (e.g. some outdoor datasets from AV literature, or simply the next iteration of ScanNet, ScanNet++). Such evaluation can be quantitative or qualitative. It's also interesting to see how the model trained on ScanNet only can transfer to other datasets. \n3. I'm interested to see the zero-shot, open-vocabulary generalization of the proposed method. As the proposed method is based on several foundation models, it probably has the ability in such settings. I won't consider this as a major weakness though, as this is not claimed as a contribution in the paper. \n\n\nMinor issues, typos, and grammar errors:\n\n* Ln 156, \"due to due to\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## W1: Case study of the reliance on pre-trained detectors.\n\nPlease refer to Figure 1 the attached PDF in the “Author Rebuttal”.\n\nWe provide several qualitative cases where the detected objects are imperfect (such as incomplete point clouds or an object being separated into two or more parts). Despite the direct influence of incomplete masks on grounding quality, there are successful cases in captioning and QA tasks. The model's ability to perceive the surroundings of the objects allows it to infer the correct captions or answers.\n\nIt is worth emphasizing that previous SOTA models such LEO[20], 3D-VisTA[68], and M3DRef-CLIP[63] are also two-stage models reliant on pre-trained detectors, which face the same challenge as us when the detector fails. \n\n## W2: Zero-shot evaluation on other datasets.\n\n**Table A. Evaluation results on scene-language tasks 3RQA, 3RDialog, and 3RPlan, based on 3RScan.**\n| Method | 3RQA | 3RDialog | 3RPlan |\n|:---:|:---:|:---:|:---:|\n| LEO (zero-shot) | 35.8 | 25.5 | 23.4 |\n| **Ours (zero-shot)** | **36.2** | **32.7** | **30.9** |\n| LEO (fine-tuned) | 51.9 | 73.3 | 81.1 |\n| **Ours (fine-tuned)** | **55.8** | **82.1** | **93.5** |\n\nConsidering the lack of 3D scene understanding benchmarks built upon other datasets, we follow the precedent set by the 3D LLM method LEO[20] and conduct an evaluation on their proposed tasks: 3RQA, 3RDialog, and 3RPlan. These tasks are built upon the 3RScan[a] dataset, which belongs to a different domain than ScanNet. We use the same detection results as LEO and leverage our pre-trained weights (including the projection layer and LLM pre-trained on ScanNet). The results shown in the table above demonstrate our model’s zero-shot capabilities on 3RScan.\n\n[a] RIO: 3D Object Instance Re-Localization in Changing Indoor Environments. ICCV 2019.\n\n## W3: About zero-shot open-vocabulary generalization.\n\nFirstly, it's important to note that open-vocabulary is not claimed as a contribution in our paper, nor do the previous baselines include formal open-vocabulary evaluations. \n\nThe Table A above reveals some zero-shot open-vocabulary abilities of our model on novel scenes in 3RScan. For achieving real open-vocabulary capabilities, we can adopt open-vocabulary detectors such as SAMPro3D[b] and Open3DIS[c], and scale up the language data to train a robust MLLM. This remains a valuable direction in this field, and we tend to leave it for future work.\n\n[b] SAMPro3D: Locating SAM Prompts in 3D for Zero-Shot Scene Segmentation. arXiv 2023.\n\n[c] Open3DIS: Open-vocabulary 3D Instance Segmentation with 2D Mask Guidance. CVPR 2024.\n\n## Q1: How do DINOv2 features work in the 3D scene understanding task?\n\nAs described in Appendix A (Ln 531-541), we extract 2D representations from the area of projected masks in multi-view images of each object. DINOv2, trained with self-supervision objectives at both image and patch levels, captures detailed local information such as shape and texture, providing fine-grained perception abilities. This aligns with our design goal of extracting rich object-centric representations. The ablation results in Table 4 demonstrate the importance of these 2D representations in our final model.\n\n**Table B. Ablation study on the 2D encoder.**\n\n|  | ScanRefer | Multi3dRefer | Scan2Cap | ScanQA | SQA3D |\n|---|:---:|:---:|:---:|:---:|:---:|\n|  | **Acc\\@0.5** | **F1\\@0.5** | **C\\@0.5** | **CIDEr** | **EM** |\n| w/ CLIP | 46.3 | 48.7 | 73.1 | 85.0 | 53.9 |\n| w/ DINOv2 | **50.2** | **52.4** | **77.1** | **88.4** | **55.5** |\n\nWe also add an ablation study replacing DINOv2 with CLIP, which was trained with image-level contrastive learning and tends to neglect rich pixel-level details. The results show that using the CLIP encoder leads to lower performance, particularly on grounding tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "keqeVyUpp3", "reviewer": "Reviewer_KdPm", "summary": "The paper proposes a new representation for 3D multimodal LLMs, a family of foundation models that repurpose LLMs to receive multimodal (visual and linguistic) input. Specifically, the paper advocates for an object-centric representation, where objects are first discovered (detected or segmented) with an off-the-shelf model, then they are fed as tokens to the LLM.\nThis happens in the following way: i) objects are discovered in 3D using a 3D detector; ii) for every object we get a language identifier (e.g., <obj1>), a local point cloud and a 2D mask (by projecting the 3D mask back to multiple 2D views); the language identifier, featurized local point cloud and featurized 2D segment form tokens, that are fed to an LLM in the form of prompt.\n\nThis formulation unifies many visual-language tasks as text generation. As a byproduct, using task-specific prompts, the model can be jointly trained on several tasks jointly, leading to improved performance. The results show quantitative gains on visual grounding, captioning and VQA.", "questions": "Overall, I appreciate the presentation of the paper, its claims, good results and interesting technical contributions. At the same time, thinking broader of the field, I'm not sure whether this paper provides enough evidence towards the right direction; going through an in-domain bottleneck may drastically limit the generality modern LLMs/VLMs can offer. Moreover, the ablations lack some targeted experiments that would help us verify the proposed components.\n\nI would appreciate if the response can clarify/add some of the following:\n\n* Discussion on the scope of using object-bottlenecked 3D VL models for the general field (beyond evaluating on ScanNet).\n\n* Discussion on the relation of this work to previous object-centric tokenization approaches from the 2D VL domain.\n \n* Adding results on Nr3D/Sr3D.\n\n* Adding some more tergated ablations that switch off object identifiers.\n\nI will adapt my score after the discussion.\n\n---------------------------------------------------\n\nPost-rebuttal, some concerns are addressed and I'm increasing my score to from 4 to 6", "limitations": "Addressed", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. (Presentation) The paper is very clearly presented and easy to understand. The writing makes the right claims and all the useful details and questions are answered in the main paper.\n\n2. (Contribution) The submission addresses an important problem, which is to ground the knowledge of LLMs in the visual world. The proposed scene tokenization is not novel per se, but its combination with MLLMs is a nice feature. Also, several important details, such as appropriate featurization using both multi-view 2D and 3D are very useful to see, because MLLMs haven't shown such good results so far using different input representation.\n\n3. (Soundness) The results indeed show that the proposed approach manages to use MLLMs in an effective way. More importantly, the comparisons include baselines that also train multiple tasks simultaneously, so the gain doesn't seem to come more data alone. Although we cannot conclude that this architecture is better than baselines which may have trained on a single task only, it seems to be the strongest multi-task approach now.", "weaknesses": "1. (Contribution) The paper enters the debate of the right representation for visual-language (VL) understanding without giving a clear answer, unfortunately due to current benchmarks' limitations. Object-centric transformers for VL tasks have been proposed long ago (see VilBERT, VisualBERT, LXMERT, OSCAR, UNITER etc for 2D VL understanding), but then got superseded by one-stage approaches like MDETR. The issue with two-stage object-centric models is the definition of a vocabulary. Open-vocabulary methods cannot really rely on detector bottlenecks, since they, by design, have a limited vocabulary. We cannot possibly enumerate all the concepts a user may refer to.\n\nThat said, in 3D VL understanding and especially grounding, most approaches are indeed object-bottlenecked. This can be attributed to ScanNet being the base dataset for most 3D VL benchmarks. ScanNet limits the scope of in-domain approaches to few classes. One cannot for example detect the legs of a chair using ScanNet, since parts of objects are not annotated. This is true for both one-stage and two-stage approaches, as long as they are trained on ScanNet. But among the two directions, the one more promising to extrapolate on a broad domain seems to be the non-bottlenecked one.\n\nHowever, the object-centric tokenization is not bad, it provides a nice abstraction of the scene. My concern is that a detector will not cover some useful parts of the scene and, as a result, that part of the scene won't be visible at all to the model. This can be due to imperfect prediction or even limited concept vocabulary. Given how impressive the VLMs' generalization is, limiting them with in-domain detectors may be handicapping them.\n\n2. (Soundness) It seems that a lot of quantitative gain comes from using 2D features. While it is fair to use 2D features in the proposed approach, I'm not sure whether the main factor of good results is the use of object identifiers, 2D features or multi-tasking through the unification of the output space. For example, an approach that does the same unification (with different prompts of course) and uses scene tokens and multi-view features (for example by unprojecting 2D features from multiple views and performing some voxelization) would be the best baseline to validate the importance of object identifiers. Which baseline or ablation represents that direction? Right now, we can mainly conclude that 2D features are very important and that the format of object identifiers is better than other alternatives, but we cannot safely conclude that object identifiers are the most useful component.\n\nAnother interesting fact that I noticed in this paper is that the best-performing models on grounding (ScanRefer) are the ones trained only on ScanRefer. In fact, the proposed approach is the only multi-task approach that beats those models. I believe it would be useful to also see some results on Nr3D and Sr3D for several reasons. First, the margins on grounding are a bit narrow, so we don't know whether the most competitive approaches lack due to less data or architecture. Evaluating on more datasets could provide some more evidence. Second, Nr3D and Sr3D use ground-truth boxes. It's good to see what is the limit of the proposed approach if a part of perception is perfect.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## W1.1: Concerns about the recent trend of one-stage replacing two-stage methods in 2D.\n\nThanks for pointing out a promising future direction. However, one-stage models require large-scale training, for example, MDETR used 1.3M image-text pairs for pre-training. Given current limited 3D data (1200 scenes and 150K language annotations), our SOTA performance across various benchmarks highlights the effectiveness of our two-stage architecture. The data scaling and the exploration of end-to-end architectures should be left as a future work.\n\n## Q1.1 & W1.2: Discussion on the scope of using object-bottlenecked 3D VL models.\n\nFirstly, it's important to note that open-vocabulary is not claimed as a contribution in our paper, nor do the previous baselines (either one-stage or two-stage) include formal open-vocabulary evaluations. \n\nAs discussed in the Limitation section (Lines 594-599), we acknowledge the limitation of relying on object detectors. Based on object detectors, our object identifiers can refer to object-level instances or clustered objects but fail to represent part-level concepts. We will clearly state this in the revised version.\n\nAlthough part-level detectors such as PointNeXt[a] could be adopted to extract part-level instances, we still lack well-trained part-level encoders and related benchmarks. It is important to highlight that current large-scale 3D datasets are object-level, such as Objaverse[b] and OmniObject3D[c], and a large-scale, high-quality concept-level dataset is lacking. Therefore, the exploration of open-vocabulary/concept-level evaluations should be left for future work.\n\n[a] PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategies. NeurIPS 2022.\n\n[b] Objaverse: A Universe of Annotated 3D Objects. CVPR 2023.\n\n[c] OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generation. CVPR 2023.\n\n## Q1.2: Zero-shot evaluation on other datasets.\n\n**Table A. Evaluation results on scene-language tasks 3RQA, 3RDialog, and 3RPlan, based on 3RScan.**\n\n| Method | 3RQA | 3RDialog | 3RPlan |\n|:---:|:---:|:---:|:---:|\n| LEO (zero-shot) | 35.8 | 25.5 | 23.4 |\n| **Ours (zero-shot)** | **36.2** | **32.7** | **30.9** |\n| LEO (fine-tuned) | 51.9 | 73.3 | 81.1 |\n| **Ours (fine-tuned)** | **55.8** | **82.1** | **93.5** |\n\nTo assess the generalizability of our model, we follow the precedent set by the 3D LLM method LEO[20] and conduct an evaluation on their proposed tasks: 3RQA, 3RDialog, and 3RPlan. These tasks are built upon the 3RScan[d] dataset, which belongs to a different domain than ScanNet. We use the same detection results as LEO and leverage our pre-trained weights (including the projection layer and LLM pre-trained on ScanNet). The results shown in the table above demonstrate our model’s zero-shot capabilities on 3RScan.\n\n[d] RIO: 3D Object Instance Re-Localization in Changing Indoor Environments. ICCV 2019.\n\n## Q2: Discussion on the previous object-centric tokenization approaches.\n\nCompared to previous object-centric tokenization methods, our approach introduces a novel way to incorporate a sequence of {object identifier, object features} into the LLM, enabling it to solve different tasks in a unified question-answering format. \n\nFor instance, ViLBERT uses an additional task head to predict matching scores for object grounding, while VisualBERT ranks entities by comparing attention weights. These methods require task-specific designs and heads for different tasks, which is impractical for real-world human-assistant interactions. \n\nIn contrast, our method establishes a direct link between the object and its identifier, allowing the LLM to respond with an object identifier as the grounding result. This approach can naturally extend to more complex tasks, such as multiple object grounding (outputting several identifiers for multiple grounding results) and grounded captioning (producing complex captions interleaved with identifiers as the grounding result).\n\n## Q3 & W2.2: Results on Nr3D/Sr3D.\n\n**Table B. Evaluation results on Nr3D/Sr3D.**\n\n|  | Nr3D |  |  |  |  | Sr3D |  |  |  |  |\n|---|:---:|---|---|---|---|:---:|---|---|---|---|\n|  | **Overall** | **Easy** | **Hard** | **View Dep** | **View Indep** | **Overall** | **Easy** | **Hard** | **View Dep** | **View Indep** |\n| 3DVG-Trans | 40.8 | 48.5 | 34.8 | 34.8 | 43.7 | 51.4 | 54.2 | 44.9 | 44.6 | 51.7 |\n| TransRefer3D | 48.0 | 56.7 | 39.6 | 42.5 | 50.7 | 57.4 | 60.5 | 50.2 | 49.9 | 57.7 |\n| MVT | 59.5 | 67.4 | **52.7** | **59.1** | 60.3 | 64.5 | 66.9 | 58.8 | 58.4 | 64.7 |\n| 3D-VisTA | 57.5 | 65.9 | 49.4 | 53.7 | 59.4 | 69.6 | 72.1 | **63.6** | 57.9 | 70.1 |\n| **Ours** | **63.9** | **75.7** | 52.6 | 53.8 | **69.2** | **73.1** | **78.3** | 60.8 | **66.6** | **73.4** |\n\nThe results show that our model achieves state-of-the-art performance compared to previous expert models. \n\n## Q4 & W2.1: Importance of object identifiers.\n\n**Table C. Ablation study on object identifiers.**\n\n|  | ScanRefer | Multi3dRefer | Scan2Cap | ScanQA | SQA3D |\n|---|:---:|:---:|:---:|:---:|:---:|\n| w/o obj identifiers | 22.8 | 25.6 | 48.9 | 79.8 | 50.0 |\n| **Ours** | **50.2** | **52.4** | **77.1** | **88.4** | **55.5** |\n\nObject identifiers are crucial to our model's design. They allow users to reference objects for tasks such as 3D dense captioning (Scan2Cap), while the LLM uses these identifiers for grounding objects in both single object grounding (ScanRefer) and multiple object grounding (Multi3DRefer). \n\nWithout object identifiers, it is necessary to adopt an alternative method for grounding and dense captioning. Following 3D-LLM[22], we add special location tokens to represent bounding boxes in the LLM. Thus, the model can output bounding boxes for grounding tasks and input bounding boxes for dense captioning. As shown in the table above, training the model without object identifiers reveals a significant decline in performance, particularly in grounding and captioning tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cKS7pKoL4L", "reviewer": "Reviewer_G2or", "summary": "This paper proposes a 3D Multi-Label Language Model (MLLM) designed to perceive and represent 3D scenes at the object level. To interpret individual object instances, the authors develop object identifiers to convert the 3D scene into a series of distinguishable object tokens and present object-centric representations using foundation models. Experiments are conducted on various 3D scene-language tasks.", "questions": "Please refer to the weaknesses section.", "limitations": "The authors have discussed the limitations and potential societal impact of the proposed method.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. Introducing Large Language Models (LLMs) into 3D perception and representation is a valuable and innovative research direction.\n\n2. Leveraging foundation models to extract 3D and 2D embeddings shows significant potential for enhancing the performance and capabilities of the 3D MLLM model.", "weaknesses": "1.\tThe concept of object identifiers is not new, as similar methods have been previously introduced, such as in \"Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V.\"\n2.\tThe authors claim the proposed model enables \"efficient object referencing and grounding abilities,\" but this efficiency is not evaluated in the experiments. Furthermore, experiments on 3D referring expression segmentation are not provided, making this claim hard to substantiate.\n3.\tIt is unclear how the proposed object-centric representations address the problem of 3D data scarcity.\n4.\tTable 2 lacks comparisons with several notable works, such as the state-of-the-art method CORE-3DVG (NeurIPS 2023) on the ScanRefer dataset. More existing methods should be included for a comprehensive evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## W1: Difference with Set-of-Mark.\n- **Different ways to introduce object identifiers.** Set-of-Mark attaches object Identifiers directly onto the image, relying entirely on the multimodal LLM’s OCR capability to perceive the identifiers from the image. This method is indirect and can introduce ambiguity, especially when there are many objects in the image. Our method explicitly assigns the identifiers to each object in the language prompt. Intuitively, it is easier for the LLM to understand the link between the object and its identifier from the text.\n\n- **Inefficiency of adapting Set-of-Mark to 3D.** Adapting Set-of-Mark for 3D perception requires cross-view clustering of the segmented masks (from SAM) and labeling object identifiers on multi-view images. However, currently there is no MLLM that can handle a sequence of images with a good OCR capability.\n\n- **Prompt-based vs Training-based.** Set-of-Mark is a prompt-based method, which could produce uncontrollable outputs. Our training-based method can stably handle a series of tasks including grounding, captioning, and QA. \n\nAt last, our SOTA performance across various 3D benchmarks demonstrates the effectiveness of our proposed method to use object identifiers. Thus, the similar concept of object identifiers in Set-of-Mark does not diminish the contribution or novelty of our work at all.\n\n## W2.1: Evidence of object referencing and grounding abilities.\nSection 3.3, along with Figures 2 and 3, illustrates how object identifiers are utilized during interactions with LLMs. Users can reference objects with identifiers for tasks like 3D dense captioning (Scan2Cap), while the LLM responds with identifiers to ground objects for both single object grounding (ScanRefer) and multiple object grounding (Multi3DRefer). The superior performance of our model on benchmarks such as Scan2Cap, ScanRefer, and Multi3DRefer, demonstrates its efficient object referencing and grounding abilities.\n\n## W2.2: Results of 3D referring expression segmentation.\n**Table A. Evaluation results of 3D referring expression segmentation on ScanRefer.**\n\n|  | Nr3D |  |  |  |  | Sr3D |  |  |  |  |\n|---|:---:|---|---|---|---|:---:|---|---|---|---|\n|  | **Overall** | **Easy** | **Hard** | **View Dep** | **View Indep** | **Overall** | **Easy** | **Hard** | **View Dep** | **View Indep** |\n| 3DVG-Trans | 40.8 | 48.5 | 34.8 | 34.8 | 43.7 | 51.4 | 54.2 | 44.9 | 44.6 | 51.7 |\n| TransRefer3D | 48.0 | 56.7 | 39.6 | 42.5 | 50.7 | 57.4 | 60.5 | 50.2 | 49.9 | 57.7 |\n| MVT | 59.5 | 67.4 | **52.7** | **59.1** | 60.3 | 64.5 | 66.9 | 58.8 | 58.4 | 64.7 |\n| 3D-VisTA | 57.5 | 65.9 | 49.4 | 53.7 | 59.4 | 69.6 | 72.1 | **63.6** | 57.9 | 70.1 |\n| **Ours** | **63.9** | **75.7** | 52.6 | 53.8 | **69.2** | **73.1** | **78.3** | 60.8 | **66.6** | **73.4** |\n\nTable A shows that our method surpasses previous baselines for 3D referring expression segmentation on ScanRefer. It is important to note that we did not provide referring expression segmentation results on ScanRefer simply because it is not a common evaluation metric for this dataset. Most previous baselines assess accuracy based on the IoU between the predicted boxes and the ground-truth boxes. The comparison based on box IoU can already demonstrate our model’s grounding ability.\n\n## W3: Why proposed object-centric representations alleviate the problem of 3D data scarcity.\n\nAs discussed in Section 1 (Lines 72-84), training robust scene-level representations typically requires a large amount of paired scene-language data, which is difficult to obtain. To overcome this challenge, we represent scenes using object-centric representations derived from well-trained 3D and 2D encoders. Benefiting from at least million-level pre-training, the 3D encoder excels at extracting spatial and shape attributes from point clouds, while the 2D encoder adeptly extracts rich features of object’s appearance from multi-view images. Intuitively, we then propose to combine the well-trained 3D and 2D representations explicitly at the object level, along with the object identifiers, to comprehensively represent a whole scene.\n\nUnlike previous 3D LLMs such as LEO[20] and 3D-LLM[22], which require constructing additional data (near million-level) for pre-training or alignment, our method achieves state-of-the-art performance without the need for additional alignment data. Considering that we adopt similar instruction tuning techniques as theirs, the biggest difference between our model and their models is the design of representations. Thus, our superior performance with less data usage can indicate that our proposed object-centric representations alleviate the problem of 3D data scarcity to some extent.\n\n## W4: Lack of comparison with SOTA methods.\nIn Table 2, we've included previous SOTA methods for each dataset:\n- **ConcreteNet [45]** with the highest Acc\\@0.5 on **ScanRefer**,\n- **M3DRef-CLIP [63]** with the highest F1\\@0.5 on **Multi3DRefer**,\n- **Vote2Cap-DETR++ [10]** with the highest CIDEr\\@0.5 on **Scan2Cap**,\n- **Scene-LLM [17]** with the highest CIDEr on **ScanQA** and the highest EM on **SQA3D**.\n\nIn Tables 6–10 in the appendix, we provide comprehensive comparisons on each dataset by including additional SOTA models.\n\nCORE-3DVG is a missed reference on ScanRefer. However, although it surpasses our method by 1.25% on Acc\\@0.25, our method still demonstrates superior grounding performance with a 6.39% higher Acc\\@0.5. We will include CORE-3DVG as a baseline for comparison in the future version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
### **[Our Contributions]**
We are glad to find out that the reviewers generally acknowledge our contributions:

(Contribution)
- The combination of object-centric representations and MLLMs is a nice feature [KdPm], a valuable and innovative research direction [G2or], and it is a natural way for people and LLMs to refer to object entities using object identifiers [n84p].
- Appropriate featurization using both multi-view 2D and 3D are very useful to see, because MLLMs haven't shown such good results so far using different input representation. [KdPm, G2or]

(Soundness) 
- The training schema of the model is single-stage yet effective on downstream tasks. [2kgC]
- The experimental results show our state-of-the-art performance across various tasks, indicating the effectiveness of our method. [2kgC, n84p, KdPm]
- The paper also conducts a study on video input using a 2D video instance tracking model, showing that the proposed method can still work without the 3D model. [n84p]
- The code is provided in the supplementary material. [n84p]

(Presentation) 
- The paper is very clearly presented and easy to follow. [2kgC, n84p, KdPm]

### **[New Experiments and Results]**
In this rebuttal, we have added more supporting results to address reviewers' concerns.
- Details about multi-modal inputs, detectors, encoders, and LLMs used in other methods. [2kgC]
- Results on ScanQA test set. [2kgC]
- Ablation of object identifiers. [2kgC, KdPm]
- Details about training data. [2kgC]
- Zero-shot evaluation on 3RScan datasets. [n84p, KdPm]
- Ablation of 2D encoder. [n84p]
- Results on Nr3D/Sr3D datasets. [KdPm]
- Results of 3D referring expression segmentation on ScanRefer dataset. [G2or]

Thank you again for your constructive comments. We would be happy to answer and discuss if you have further concerns.

Meta-review: The paper proposes to unify different tasks involving 3D scenes and language into question answering by 1) introducing object identifiers for detected objects, 2) formulating the task as a question-answer task with object identifiers and object embeddings and 3) using a fine-tuned LLM to generate answers.  Objects are detected using Mask3D, and object features are composed of 3D features from Uni3D and 2D features from DINOv2.  The LLM (Vicuna) is fine-tuned using LoRA.  The proposed method is used to unify visual grounding in 3D scenes (ScanRefer, Multi3DRef), dense captioning of 3D objects (Scan2Cap), and 3D QA tasks (SanQA, SQA3D).  In addition, the authors demonstrate they can adapt their model to 2D videos by using a 2D video instance tracking model (vs a 3D object detector).  

Three of the reviewers advocate for acceptance (2kgC, n84p, KdPm) while one reviewer is negative (G2or).

Reviewers initially had questions about additional experiments and ablations (2kgC,n84p,KdPm), discussion of time and resource cost (2kgC), and requested additional details and clarification regarding training and inference (2kgC) as well as more discussion on prior work that uses object-centric token representations (n84p, KdPm, G2or) and reliance on performance of pre-trained feature detectors and extractors (n84p, KdPm).  The strong response has encourage two of the reviewers (KdPm, G2or) to increase their ratings (although G2or remains negative)

Given the positive rating of the majority of reviewers, and the extensive experiments, the AC recommends acceptance as the proposed method provides a interesting and promising approach to unify 3D vision and language tasks.  The authors are encouraged to include the additional experiments for the rebuttal (including ablation on the importance of object identifier) and add discussion and clarifications from reviewers in their camera ready.
Decision: Accept (poster)