DenseGrounding: Improving Dense Language-Vision Semantics for Ego-centric 3D Visual Grounding
Authors: Henry Zheng
Hao Shi
Qihang Peng
Yong Xien Chng
Rui Huang
Yepeng Weng
zhongchao shi
Gao Huang
Venue: ICLR 2025 Poster (2025)
Keywords: 3D Visual Grounding
Embodied AI
Egocentric Vision
Primary area: applications to robotics, autonomy, planning
TL;DR: DenseGrounding enhances visual and textual semantics to significantly improve ego-centric 3D visual grounding, outperforming existing methods.

Abstract: Enabling intelligent agents to comprehend and interact with 3D environments through natural language is crucial for advancing robotics and human-computer interaction. A fundamental task in this field is ego-centric 3D visual grounding, where agents locate target objects in real-world 3D spaces based on verbal descriptions. However, this task faces two significant challenges: (1) loss of fine-grained visual semantics due to sparse fusion of point clouds with ego-centric multi-view images, (2) limited textual semantic context due to arbitrary language descriptions. We propose DenseGrounding, a novel approach designed to address these issues by enhancing both visual and textual semantics. For visual features, we introduce the Hierarchical Scene Semantic Enhancer, which retains dense semantics by capturing fine-grained global scene features and facilitating cross-modal alignment. For text descriptions, we propose a Language Semantic Enhancer that leverage large language models to provide rich context and diverse language descriptions with additional context during model training. Extensive experiments show that DenseGrounding significantly outperforms existing methods in overall accuracy, achieving improvements of **5.81%** and **7.56%** when trained on the comprehensive full training dataset and smaller mini subset, respectively, further advancing the SOTA in ego-centric 3D visual grounding. Our method also achieves **1st place** and receives **Innovation Award** in the 2024 Autonomous Grand Challenge Multi-view 3D Visual Grounding Track, validating its effectiveness and robustness.

Published as a conference paper at ICLR 2025
DENSEGROUNDING: IMPROVING DENSE LANGUAGE- VISION SEMANTICS FOR EGO-CENTRIC 3D VISUAL GROUNDING
Henry Zheng', Hao Shi- Qihhang Peng', Yong Xien Chng', Rui Huang', Yepeng Weng", Zhongchao Shi?, Gao Huangl
DDparrmeen of Automation, BNRist, Tsinghua University AT Lab, Lenovo Research
(jh-zheng22, shi-h23, pgh22, chngyx100 0, hr20] i ls. tsinghua edu. on {wengypl, shiz:c)eleeenooooo com (aachanngettsingee
ABSTRACT
Enabling intelligent agents to comprehend and interact with 3D environments through natural language is crucial for advancing robotics and human-computer interaction. A fundamental task in this field is ego-centric 3D visual grounding, where agents locate target objects real-world 3D spaces based on verbal descrip- tions. However, this task faces two significant challenges: (1) loss of fine mne-grained visual semantics due to sparse fusion of point clouds with ego-centric multi- images, (2) limited textual semantic context due to arbitrary language descriptions. We propose DenseGrounding. a novel approach designed to address these issues by enhancing both visual and textual semantics. For visual features, we introduce the Hicrarchical Scene Semantic Enhancer, which retains dense semantics by cap- turing fine-grained global scene features and facilitating moos- alignment. For text descriptions, we propose a Language Semantic Enhancer that leverages large language models provide rich context and diverse language descriptions with additional context during model training. Extensive experiments show that DenseGrounding significantly outperforms existing methods in overall accuracy, with improvements of 5.81 % and 7.56% when trained on the comprehensive full dataset and smaller mini subset, respectively, further advancing the SOTA in ego- centric 3D visual grounding. Our method also achieves 1st place and receives Innovation Award in the CVPR 2024 Autonomous Grand Challenge Multi-view 3D Visual Grounding Track, validating its effectiveness and robustness, Code
INTRODUCTION
Recent years have seen increasing attention in the field of embodied AI, with the introduction of
3D visual grounding benchmarks (Achlioptas al. 2020a Chen al. 2020 Wang al.) 2024)
prompting a surge of research works (Guo et al. 2023 Wu et al. 2023 Zhao et al.) 2021 Jain 2022, Yang et al.) 2024 Huang et al.] 2022] Wang
2024). The task of 3D visual grounding, which aims to locate target objects in real-world 3D environments based
natural language descriptions, is a fundamental perception task for embodied agents. This capability is crucial for enabling agents to interact with and understand their surroundings through language, facilitating applications robotics and human-computer interaction. For instance, accurate 3D visual grounding empowers robots to perform tasks such object retrieval and manipulation based on verbal instructions, enhancing their functionality in service and assistive scenarios.
Despite these advances, significant challenges continue hinder the performance of 3D perception systems. One major challenge lies how embodied agents perceive their environment, as they typ- ically rely ego-centric observations from multiple views while moving around, lacking holistic, scene-level perception. Some methods attempt to enhance scene-level understanding using recon-
structed scene-level 3D point clouds (Wu al. 2023
Zhao al.) 2021
Jain et al. 2022 Yang
"Equal contributions. "Corresponding author.

---

Published as a conference paper at ICLR 2025
Description: box that near the printer
Augmented Description:
View
Cloud (onty -2%]
Select the box that is in proximity the printer, situated near the
Vhew
View
Denae Semantics
Spares Ceometrics
Box
Printer
Sparce Fusion
Oven
Limited points objects
Additional Anchers
Intended Target
Distractor
Anchors
(4) Insufficient language semantics due limited context.
(b) Semantics sparse fusion.
Figure 1: (a) illustrates how limited context due to arbitrary descriptions leads to insufficient lan-
guage semantics. (b) highlights the issue of losing fine-gree grained semantics in sparse fusion.
al. 2024, Guo et al.] 2023]] Huang et al. 2022 2024b]e) following the previous 3D perception methods (Wu et al. 2024b Jiang et 20200, but these approach are impractical in real-world appli- cations where such comprehensive scene-level information is not readily available. this context, EmbodiedScan (Wang et al. 2024) have emerged, utilizing multi-view ego-centric RGB-D scans to address the need for models to understand scenes directly from sparse views. decouples the encoding of RGB images and repthhreconstruuted point clouds from ego-centrie views to extract both semantic and geometric information. Semantic features are then projected onto 3D space using intrinsic and extrinsic matrices to create a semantically enriched sparse point cloud for bounding box regression. However, due to the high number of points in the reconstructed point cloud and computational limitations, only sparse subset (around 2%) is sampled. This sampling results in significant loss of visual semantics, limiting the model's 's ability to retain fine-graine object details, especially for smaller objects, and ultimately hindering grounding performance.
Another challenge is the ambiguity in natural language descriptions found in existing datasets (Chen al. 2020 Achlioptas et al., 2020a Wang et al. 2024). Humans naturally provide concise instruc- tions, often lacking critical reference points or anchor objects, which leads to inherent ambiguities. For example, command like "Please get the water bottle on top of the table" does not specify which table or provide additional context in environments with multiple similar objects. This conciseness makes difficult for models to disambiguate between similar objects within complex scenes (Wang et al. 2024p. Furthermore, the increased complexity of scenes in new datasets- similar objects instances in a single scene exaee-exaerbbttss this issue. In result, training on these data may cause model to have suboptimal performance. While previous attempts have been made to enrich descriptions using large language models with predefined templates (Guo et al.] 20233 or by concatenating related descriptions (Wang et al., 2024), these methods have not fully mitigated the ambiguities present in the annotations and have not utilize the available informations in the dataset.
more
Addressing these challenges crucial for advancing the field of embodied AI perception and en- hancing the practical utility of 3D visual grounding in real-world applications. In response to these challenges, we propose DenseGrounding, a novel method for multi-view 3D visual grounding that alleviates the sparsity in both visual and textual features. Specifically, to address the loss of fine- grained visual semantics, we introduce the Hierarchical Scene Semantic Enhancer (HSSE), which enriches visual representations with global scene-level semantics. HSSE operates hierarchically, starting with view-level semantic aggregation that integrates multi-scale features from RGB images within each view to capture detailed view-level semantics. Then, HSSE performs scene-level se- mantic interaction to combine these aggregated multi-view semantics with language features from descriptions, promoting cross-modal fusion and hoslistic understanding of the scene. Finally, HSSE conducts semantic broadcast that infuses the enriched global scene-level semantics into the multi-scale feature maps of reppttrrccotttuuttee point cloud, improving their fine-grained semanties.
2

---

Published as a conference paper at ICLR 2025
To mitigate the ambiguity in natural language descriptions used for training, for the text modality, propose the Language Semantic Enhancer (LSE) that constructs a scene information database based on the existing dataset. EmbodiedScan. This database captures crucial information about object relationships and locations within the scene. During training, the LSE leverages L.M to enriches input descriptions with additional anchors and context, reducing confusion among similar objects and providing more robust representations by incorporating external knowledge. This enrichment increases the semantic richness of the language descriptions and reduces inherent ambiguities.
By addressing the ambiguities in annotated text and enhancing the model's capacity to capture scene- level global visual semanties from ego-centric multi-view inputs, DenseGrounding, advances the oftte-of-thh-arr in 3D visual grounding. Extensive experiments demonstrate the effectiveness of our approach, highlighting the importance of enriched textual descriptions and bidirectional interaction between text and visual modalities in overcoming the limitations of previous methods.
Our main contributions are as follows:
We develop a novel Language Semantic Enhancer (LSE) that employs LLM-assisted de- scription augmentation to increase semantic richness and significantly reduce umbiguities in natural language descriptions. By leveraging an LLM grounded in scene information database, our approach enriches the diversity and contextual clarity of the textual features.
We introduce a Hierarchical Scene Semantic Enhancer (HSSE) that effectively aggregates ego-centric multi-view features and captures fine-grained scene-level global semantics. HSSE enables interaction between textual and visual features, ensuring the model captures both global context and detailed object semanties from ego-centric inputs.
Our method achieves state-of-the-ar performance on the EmbodiedScan benchmark, sub- stantially outperforming existing approaches. We secured first place in the CVPR 2024 Autonomous Driving Grand Challenge Track on Multi- View 3D Visual Grounding (Zheng ct al. 2024), demonstrating the practical effectiveness and impact of our contributions.
2 RELATED WORKS
3D Visual Grounding. The integration of language and 3D perception for scene understanding is crucial in embodied AI, enabling agents to navigate, understand, and interact with complex real-
world environments. Key tasks (Chen et al.
2020, Achlioptas et al., 2020b] Azuma et al. 2022
Chen et al. 2021) in this domain, 3D Visual Grounding (3DVG) (Wu et al. 2023] Jain et al. 2022
Zhu et al. 2023b Huang et al. 2022 2024a Cai al. 2022). 3D Captioning (Huang et al. 2024a
Chen et al. 2023 Cai et al. 2022) and 3D Question Answering (3DQA) (Huang et al. 2024a),
combine natural language understanding with spatial reasoning to advance multimodal intelligence
in 3D spaces. Our work focuses on ego-centric 3D visual grounding tasks that integrate multimodal
data to localize objects in 3D space.
Major 3D visual grounding methods can be divided into ote-stage and two-stage architectures. One- stage approaches (Chen et al., 2018 Liao al. 2020 Luo et al. 2022; Geng & Yin 2024) He & Ding 2024) fuse textual and visual features in a single step, enabling end-to-end optimization and faster inference. Recent works have optimized single stage methods in various ways: 3D- SPS [Luo et al., 2022) progressively selects keypoints guided by language; ViewInfer3D (Geng & Yin 2024) uses large language models (LLMs) infer embodied viewpoints; and ReiMask3D (He & Ding 2024) integrates language with geometrically coherent sub-clouds via cross moosl group- word attention, producing semantic primitives that enhance lision-language understanding.
Two-stage approaches Yang et al.,
2019; Achlioptas et al. 2020a Huang et al. 2022 Guo et al.
2023 Wu et al. 2024a Chang et al. (024)) first use pre-trained object detectors (Jiang et al. [020
Wu et al., 2024b) to generate object proposals or utilize ground truth bounding boxes, which are
then matched with the linguistic input. Advances this framework include the Multi- Trans-
former (Huang et al. 2022), projecting the 3D scene into holistic multi-view space; ViewRe- fer (Guo et al.] 2023). leveraging LLMS to expand grounding texts and enhancing cross-view in- teractions with inter-view attention; and MiKASA Transformer (Chang et al., 2024), introducing a scene-awe aweee object encoder and a multi-key-ae key-hn technique to improve object recognition and spatial understanding. In paper, we target the more challenging setting of single-stage methods.
3

---

Published as a conference paper at ICLR 2025
LLMs for Data Augmentation. Recent advances in large language models (LLMs) (Achiam al. Touvron et al. 2023 Du et el. 2021) have demonstrated remarkable capabilities. Fully lever- aging the power of LLMs beyond language processing, some recent studies have successfully in- tegrated them with other modalities, leading to the development of highly effective multi-modal
2023
methods (Moon et al. 2022 Guo al.) 2023 Feng al. 2024) for vision-language tasks.
However, directly utilizing LLM or VLMs in robotics perception tasks remains a research ques-
tion (Kim et al., 2024] Zhen al. 2024; Phan et al. 2024). Therefore, leveraging LLMs directly
implement data augmentation and maximizing their potential to enhance text data diversity become new trend (Peng et al. 2203 Khan et al. 2023 Dunlap al. 2023 Guo et al., 2023 Ding al. 2024). Among them, ALIA (Dunlap et al. (202]) leverages LLMs to generate image descriptions and guide language- image modifications, augmenting the training data. This approach surpasses traditional data augmentation techniques in fine-grained classification, underscoring the value of LLM--diven augmentation in visual tasks. In 3D visual grounding, although ViewRefer (Guo al.) 2023) employs LLMs to augment textual data through rephrasing view-dependent descriptions from varying perspectives, however the data samples are augmented individually and suffer from limited textual semantic information. To obtain high-quality description augmentations that disambiguate among similar items and more robust representations, we utilize LLMs to enrich input descriptions utilizing readily available information in the dataset as additional context.
PRELIMINARIES
3.1 EGO-CENTRIC 3D VISUAL GROUNDING
In real-world scenarios, intelligent agents perceive their environment without any prior scene knowl- edge. They often rely on ego-centric observations, such as multi-view RGB-D images, rather than pre-established scene-level priors like pre-reconstructed 3D point clouds of the entire scene, com- monly used in previous studies (Wu et al. 2024b Huang et al. 2024b Chng et al., (202).
Following Wang al. 2024), we formalize the ego-centric 3D visual grounding task follows: Given a language description L E TR.", together with V views of RGB-D images {(T,, Dr))k= where L. E x represents the RGB image and D. IR HxW denotes the depth im- age of the v-th view along with their corresponding sensor intrinsics {(K,KKD)) KD) and ex- trinsics {(T), Te) TD)) ==1, the objective is output 9-degree-of ofrereom (9DoF) bounding box B = (z. y, 2, 1, w, h, O, Φ. v). Here, (z, 2)2 are the 3D coordinates of the object's ss center, (1, w, h) are its dimensions, and (0, Φ, v) are its orientation angles. The task is to determine B such that accurately corresponds to the object described by L within the scene represented by {(Te, D D
3.2 OVERVIEW OF THE ARCHITECTURE
We adopt and improve upon the previous SOTA in ego-centric 3D visual grounding (Wang et al. 2024). To prevent geometric information from interfering with semantic extraction and fully lever- age each modality's strengths, [Wang et al. (2024) decouple the encoding of input RGB and depth signals from each ego-centric views. following Liu et al. (202)). Specifically, the depth data from each view is transformed into a partial point cloud, which is then integrated into a holistic 3D point cloud P E R Nx3 using global alignment matrices, thereby preserving precise geometric details.
Next, a semantic encoder and a geometric encoder extract multi-scale semantic and geomet- ric features from {LoJ and P. respectively, denoted as { 1| XVVxHxXW,Xx and [F E RRN.C. Here, N is the total number of points sampled from all perspective views, and S is the number of scale for the encoded feature maps.
The semantic features from RGB data are then lifted to the 3D space using intrinsic and extrinsic matrices {(K), T and concatenated with geometric features at multiple scales, followed by fusion through Feature Pyramid Network (FPN). However, due to the large number of points in the depth-rea reaphhrrconstuuceed point cloud and computational limitations, only a sparse subset (about 2%) is sampled, and the semantic features at unsampled locations discarded, resulting in significant semantic information loss. To address this issue, we introduce a Hierarchical Scene Semantic En- hancer module (detailed in Sec. 4.1 to improve scene-level semantics. The enhanced features
4

---

Published as a conference paper at ICLR 2025
lnbie
FFN
Head
1nn the chair besce the tablo, nearest to the Inlevision, and furthest the windows
Text-Guided Cross Attention
Hisrarchical Scone Semantic Enhancer
Language Semantic Enhancer Text Encoder
K,
Vilton-Guidded Crozs Attantion
Q
K
Seledion
Fusion
Multi-modal Decoder
Geometric Encoder
(a) Overview of DenseGrounding
Language Semantic Enhancer (LSE)
(c) Hierarchical Scene Semantic Enhancer (HSSE)
Object Location Branch
Predicted Bbox
Bbox Labels
pool
pool
Modill
Vieswlovell
View level
Samantic Aggiaasto
....
Semaniia Aggresgate
Text Feature
Object Relation
Select K
Language
Scene-Iooe Semantic Interaction
X Laaam
VK.V
K.
Faccing front the qui choose the curteiin the right of Input
With baok
curtain the of near the bedi betweon the picture and ciock, and away from biryele
Sematic Broadicat
Sermalic Broadcast
View
Augmented Text
Figure 2: (a) shows overall framework, while (b) details the Language Semantic Enhancer (LSE)
module, and (c) describes the Hierarchical Scene Semantic Enhancer (HSSE) module.
then fed into the decoder, which remains unchanged from the previous work, where the visual tokens most similar to the text features are selected as queries.
4
METHOD
In this section, we present our proposed method, DenseGrounding, for ego-centric 3D visual ground- ing. As shown in Figure 2 our method consists of three key components: Hierarchical Scene Se- mantic Enhancer (Sec. LLM- Language Semantic Enhancer (Sec. [4.2). and an Enhanced Baseline model (Sec. 4.3) We will describe each component in detail in the following subsections.
4.1 HIERARCHICAL SCENE SEMANTIC ENHIANCER
To mitigate the semantic loss inherent in sparse point clouds, we introduce the Hierarchical Scene Semantic Enhancer (HSSE) module, which extracts individual view semantics and enriches visual representation with global scene-level semantics. This enriched information is then unprojected to the depth reconstructed point cloud during fusion, minimizing the semantic loss.
Our proposed HSSE module works hierarchical manner, starting with view-level semantic ag- gregation to capture view-level semantics from individual egocentric perspective views. Then, it fuses these aggregated view semantics with language semantics, facilitating scene-level multi-view semantic interaction and cross-modal feature fusion. Moreover, broadcast module integrates these scene-level semantics into multi-scale semantic feature maps, enriching the fine- mne-griined semantics in the sparse point cloud. Finally, the HSSE employs aggregation- broadcass mechanism to filter out irrelevant information, ensuring efficient and streamlined semantic transfer to the point cloud for enhanced performance.

---

Published as a conference paper at ICLR 2025
View-Level Semantic Aggregation. With the encoded 2D features from each view, HSSE performs view-level semantic aggregation to capture view-level global semantics within each view. For the v-th view, given the multi-scale semantic features ( s== E " x2,,,,,xC. extracted from RGB image, we aim to capture the salient semantic cues in each view.
First, these semantic features are processed through Feature Pyramid Network (FPN) to integrate information across different scales. A specific scale of the feature map is then selected as the reference feature Ret BHH,W,xC,
Ref
Conv
Upsample
RRH....
Next, adaptive average pooling is applied to the reference feature to reduce its spatial dimensions, yielding the initial query features Fü E Rh XM,
F = Pool Ret) E R Xmx
Subsequently, we compute the rross-attention between the pooled queries and the reference feature to aggregate semantic information at the view level.
Fo CrossAttm( FØ K
Refs V
FR
where FV E K xC, encapsulates the global semantics of the view. We then apply self attention layer, further refine the features and model intra-view relationships.
F SelfAttt( Q = Po: K Po V = =F))
Scene-Level Semantic Interaction. Moreover, HSSE conducts scene-level semantic interaction by integrating the aggregated multi-view semantics 1/ with the language semantics Fi derived from the language description. To promote cross-modal fusion and scene-level interaction among multi-view semantics, concatenate (ET)V= with Long and apply Leaene layers of self-attention.
E H 1m Fngg SelfAttn frxan ((( V FLang)
(5)
where (Fo V captures the scene-level global semantics, and F'agg represents the refined language semantics after the interaction.
Semantic Broadcast. Finally, HSSE broadcasts the scene-level global semantics Py back to the multi-scale semantic feature maps ( E &,,C. for each view. Specifically, we com- ==1 and the scene-level semantics Fgy. treat-
pute cross-attention between the spatial features (
ing the spatial features as queries.
[ S s=1 s=1 CrossAttnQQ = ( #=1: K = P =
(6)
where {F 1 1== represents the enhanced feature maps enriched with scene-level global semantics.
4.2 LLM-BASED LANGUAGE SEMANTIC ENHANCER
Human interactions with intelligent agents often involve casual and vague language descriptions, resulting in input that lacks clear anchors, limited textual context, and ambiguities. To address this, we propose a Language Semantic Enhancement (LSE) pipeline based on Large Language Models (LLMs) to enhance the training data. This pipeline enriches input language descriptions by lever- aging L.Mss, which are grounded with Scene Information Database (SIDB) containing object locations and relationships, providing more contextual details of the corresponding scene.
Construct Scene Information Database (SIDB). Visual grounding datasets are typically built on indoor 3D detection datasets, where collected RGB-D images or point clouds are used to represent scene. Each object within the scene is annotated with 3D bounding boxes, which visual grounding datasets extend by adding language descriptions detailing the object's relationship to its surround- ings. To ensure accurate contextualization for L.MSs augmenting these language descriptions,

---

Published as a conference paper at ICLR 2025
propose to construct Scene Information Database (SIDB) based the annotated training set that is also used by all baseline methods for model training.
For each 3D scene, follow the design of the datasets and assume that every language description targets a specific bounding box. These descriptions are grouped according to their corresponding scenes, allowing the SIDB to store detailed object relationships within each scene. This approach enables LLMs to better understand the spatial and relational dynamics of objects in a given scene, leading to a more holistic comprehension of the environment.
Moreover, to provide location information about the object instances in the seene, we enrich each of the descriptions in SIDB with position information from the original dataset annotations or pseudo- labels from the detection model, associating each described object with its spatial location. These descriptions, combining both spatial and semantic information, form the backbone of the SIDB. By providing this structured and detailed knowledge, the SIDB allows LLMs to generate more contextually informed and reliable text descriptions grounded the specific dynamies of each scene.
Prompt for LLM-based Enhancement. To effectively enrich the input descriptions, we design a prompt for the LLM that leverages the SIDB to generate more detailed and contextually rich de- scriptions. By instructing the LLM to utilize positional relationships among objects in the scene, the prompt ensures sufficient and reliable anchors, encouraging the description of the target object using multiple reference objects to minimize ambiguity and distractions. For instance, it draws on spatial context from the database more accurately reflect the object's relationships to surrounding items. Moreover, the prompt guides the LLM to maintain consistency between the original and augmented text by preserving the original content while adding additional information, which reduces the risk of the generated text deviating from the initial prompt.
Enhance Description with LLM and SIDB. For each description to be augmented (raw descrip- tion), we select the & most related context descriptions in the corresponding scene based on object names from the SIDB and provide them to the L.M. Details on the context description selection can be found in Appendix [..522 The LLM leverages the database to incorporate spatial and semantic details, aiming enrich the text with additional contextual anchors that can help prevent confusion with similar-looking objects.
4.3 ENHANCED BASELINE
In this work, we introduce a strong baseline by building upon the state-of-the-art method estab- lished by EmbodiedScan for ego-centric multiview 3D visual grounding and further boosting its performance. More details about the proposed improvements can be found the Appendix A.I.1
5 EXPERIMENTS
Dataset and Benchmark. The EmbodiedScan dataset (Wang al. 2024), used in our experiments, large-scale, multi-modal, ego-centric dataset for comprehensive 3D scene understanding. consists of 5,185 scene scans sourced from well-known datasets such as ScanNet (Dai al. 2017). 3RScan (Wald et al. 2019), and Matterport3D (Chang et al. (017). This extensive dataset pro- vides diverse and rich foundation for 3D visual grounding tasks, offering broader scene coverage compared to prior datasets. This makes the dataset significantly larger and more challenging than previous ones, providing a more rigorous benchmark for 3D visual grounding tasks.
For benchmarking, the official dataset maintains non-pute non-public test set for the test leaderboard and divides the original training set into new subsets for training and validation. In this paper, we refer to these as the training and validation sets, while the non-public test set is called the testing set.
Experimental Settings. Due to resource limitations, we reserve the full training dataset for baseline comparisons on the test set and leaderboard submissions to ensure a fair and comprehensive evalua- tion. For the "mini" data in the Data column of Table 1 and analysis experiments in Sec. 5.22 use a smaller subset of the data as a proxy task in performing experiments. The subset is referred to as mini sets, available through the official release by Wang al. (2024).
7

---

Published as a conference paper at ICLR 2025
Table I: Validation Result: Accuracy performance of the models on the official full validation set. We follow the experimental setting proposed by Wang et al. (2024) and use the RGB-D as visual input. The 'Data' column indicates the data split used. 1 denotes the improved baseline discussed in Sec. 4.3 The results reported for EmbodiedScan are obtained by re-evaluating the validation set using the officially provided weights.
Easy
Hard
Indep
Dep ACC25
Overall ACC25
Method
Data
ACC25
ACC25
ACC25
ScanRefer (Chen al. 2020
Full
13.78
9.12
13.44
10.77
12.85
BUTD-DETR (Jai et al.
2022
Full
23.12
18.23
22.47
20.98
22.14
L.3 (Zhu al. (2023a)
Full
24.01
18.34
23.59
21.22
23.07
EmbodiedScan [Wang et al. 2024
Full
39.28
30.88
38.23
39.30
38.60
DenseGrounding
Full
45.31
34.23
44.40
44.42
44.41
(+6.03)
(+3.35)
(+6.17)
(+5.12)
(+5.81)
EmbodiedScan (Wang et al. EmbodiedScan'
2024
Mini
34.09
30.25
33.56
34.20
33.78
Mini
36.23
30.51
35.89
34.30
35.77
(+2.14)
(+0.26)
(+2.33)
(+0.10)
(+1.99)
DenseGrounding
Mini
41.95
34.38
40.89
42.19
41.34
(+7.86)
(+4.13)
(+7.33)
(+7.99)
(+7.56)
We report accuracy using the official metric, considering instances where IoU exceeds 0.25. Addi- tionally, we present results for "Easy" and "Hard" scenes, classifying a scene as "Hard" if it contains 3 objects of the same class. The "Dep" and "Indep" metries further challenge spatial under- standing ability by assessing its performance with and without perspective-specife descriptions.
Implementation Details. We follow EmbodiedScan (Wang et al. 2024) by using feature encoders, sparse fusion modules, and a DETR- (Carion al.] (020) decoder. Specifically, we use ResNe150 (He et al., 2016) and MinkNet34 (Choy et al. 2019) as the 2D and 3D vision encoders, respectively. Moreover, as mentioned in Sec. 4.3] we replace RoBERTA with CLIP (Radford et al.) 2021) text encoder as language feature encoders. We first trait our image and point cloud en- coders on the 3D object detection task, adhering to EmbodiedScan's 's training settings and integrating CBGS (Zhu et al.) 2019). The L.E--uugmentdd text is generated using GPT4-o mini and is utilized exclusively during training. For inference, our model processes descriptions directly, without any enhancement, aligning with our baseline methods for fair comparison. More details in App. A.1.2
5.1
MAIN RESULTS
Evaluation. We evaluate the 3D visual grounding performance of our proposed method, DenseCrrunding, and report the results in Table where we compare against established SOTA methods from the dataset benchmark. The table divided into two sections: the upper half presents models trained on the full training set, while the lower half showcases performance using mini training set, which is approximately 20% of the full dataset.
On the full training set, DenseGrounding achieves significant improvement of 5.81% % over the pre- vious strongest baseline, EmbodiedScan. This substantial gain highlights the effectiveness of our approach leveraging extensive training data to capture complex risual-linguise correlations in 3D environments. It demonstrates our method's 's superior ability to understand and ground linguistic expressions in rich visual scenes. When trained on the more manageable mini-training set, we intro- duce an enhanced baseline, denoted as EmbodiedScan", which achieves a 1.99% performance gain over the original EmbodiedScan. This stronger baseline ensures a more rigorous and fair compar- ison with our method. Remarkably, even against this enhanced baseline, DenseGrounding attains a substantial 5.57% improvement in overall accuracy, culminating in a total performance gain of hard samples, where scenes contain more
7.56% over the previous -fate-of-the-ant Specifically, than three objects of the same class, we observe a 4. 4.4 increase in accuracy. This demonstrates method's proficiency in accurately identifying the correct target among similar-looking objects.
8

---

Published as a conference paper at ICLR 2025
Table 2: Ablation on LSE. R and L refers to object relationship and object location infor- mation in SIDB, respectively.
Table Ablation of Proposed Methods. The reported values are Accuracies for predic- tions greater than 25% IoU with groundtruth.
Method
Easy
Hard
Overall
LSE
HSSE
Easy
Hard
Overall
Concat Samples
36.80 31.44
36.37
35.77
30.51
35.77
LLM
37.69 31.97
37.23
V
39.58
32.60
39.02
LLM+DB(R)
38.48 33.54
38.08
39.67
34.49
39.25
LLM+DB(R+L)
39.29 33.55
38.82
/
41.95
34.38
41.34
As detailed in Table | DenseGrounding consistently outperforms the baselines across various evaluation categories, including both casy and hard samples, as well in view-independent and view-dependent tasks. These consistent gains across different metrics underscore the robustness and generalizability of our approach in 3D visual grounding tasks. Moreover, achieves the highest performance among all methods but also secured 1st place in the CVPR 2024 Autonomous Driving Grand Challenge Track on Multi-View 3D Visual Grounding.
model not only
5.2 ANALYSIS EXPERIMENTS
Effectiveness of Each Component. We conduct an ablation analysis assess the effectiveness of each component, as shown in Tab. Our baseline is an enhanced model that excludes text augmentations and semantic enhancements. Introducing the augmented data led to a remarkable accuracy improvement of 3 3.48%, highlighting its significant impact. The further integration of the HSSE module provided an additional performance boost of 2.09%. Furthermore, the combined use of LSE and HSSE resulted in 5.57% accuracy improvement on hard samples compared to the baseline, underscoring our model's 's enhanced capacity for both visual and language understanding, particularly in disambiguation of challenging cases.
Ablation on LSE. To evaluate the effectiveness of our proposed LSE, conducted experiments comparing several text augmentation methods. The "Concat Samples" method from Embodied- Scan disambiguate descriptions by concatenating multiple annotations. "LLM" refers to our re- implementation of template-based LLM augmentation used by Viewrefer (Guo et al.) 2023). Our method, "LLM+DB(R)," employs LLM with SIDB incorporating object relationships only, while "LLM+DB(R+L)" extends this approach by adding location information from 3D bounding boxes. It is worth noting that since "Concat Samples" method only has about 25% data, we limit other meth- ods to use 25% data for fair comparisons. The results demonstrate that "LLM+DB(R+I) RL)"" achieves the notable over all improvement of 2.45% against naive baseline, confirming the effectiveness of incorporating both object relationships and location data in augmentation process.
Ablation HSSE Components. We further explore the design of the HSSE module through comprehensive experiments, as presented in Tab. 4 and [S] In Tab. we conduct an ablation study determine the optimal number of self-attention layers needed for effective learning of the scene- feature representation. Additionally, Tab. Tab.5lexaminns the impact of varying feature map sizes for the pooled Fü feature. The results indicate that small feature map size may be insufficient to capture the necessary information, while an excessively large feature map may include redundant details. Based on these findings, we adopted the best configuration for our final model.
5.3 QUALITATIVE ANALYSIS
We present qualitative results to illustrate the effectiveness of DenseGrounding in improving ego- centric 3D visual grounding performance. baseline model, EmbodiedScan. It can be clearly seen that method outperforms the baseline in correctly identifying the target objects based on ambiguous descriptions. In cases where the baseline model struggles disambiguate between multiple similar objects, DenseGrounding successfully de- tects the correct target by leveraging its enriched textual deseriptions and robust cross-modal interac- tions. For instance, in environments where descriptions such "select the keyboard that is close to
a comparison between our model and our

---

Published as a conference paper at ICLR 2025
Table 4: Ablation on the number of self tention layers for HSSE.
Table 5:
Ablation on the view feature map size after pooling for HSSE.
Lccene
Easy
Hard
Overall
Pooled Size
Easy
Hard
Overall
1
40.15
35.44
39.77
1
40.61
34.91
40.15
2
40.76
32.70
40.11
3
41.92
33.75
41.26
3
41.95
34.38
41.34
5
41.95
34.38
41.34
41.06
33.96
40.49
7
41.92
34.28
41.30
6
40.66
33.23
40.06
9
40.05
34.60
39.61
Text Input
Ground Truth
Baseline
Ours
choose the chair that is for from the copier
select keyboard that is close to the fan
select the lamp that in front of the door
Figure 3:
Qualitative Analysis. Comparison of Ground Truth, our baseline and DenseGrounding. Ground truth boxes are shown in green, baseline in red, and DenseGrounding's predictions in blue.
the fan" are provided, our model accurately localizes the keyboard, while the baseline misidentifies nearby objects. This improvement is attributed to HSSE, which captures both fine-grained details and provides better alignment with text, enabling better object identification. These qualitative re- sults demonstrate the enhanced performance of DenseGrounding, especially in complex scenes with multiple distractors, solidifying its robustness and precision in real-world applications.
5.4 LIMITATIONS
While the DenseGrounding significantly improves the ego-centric 3D visual grounding task perfor- mance, it has limitations. In real-life applications, vague or ambiguous descriptions from human instructions pose challenges, the model struggles without the necessary information resolve ambiguities. The lack of clear instructions limits the effectiveness of the embodied agent. Future research should explore integrating human-agent interaction, allowing the model to query users for clarification, and improving adaptability and robustness in real-world scenarios.
CONCLUSION
In this work, we propose DenseGrounding, novel approach to address the challenges of ambiguity in natural language descriptions and loss of fine- ane-grained semantic features in multi-view 3D visual grounding. By leveraging LLMs for description enhancement and introducing the HSSE to enhance fine-grained visual semantics, our method significantly improves the accuracy and robustness 3D visual grounding in ego-centric environments. Extensive experiments on the EmbodiedScan bench- mark demonstrate that DenseGrounding outperforms existing state-of of-the-art models, securing first place in the CVPR 2024 Autonomous Grand Challenge. These results highlight the importance of enriched language descriptions and effective cross-modee feature interactions for advancing embod- ied AI perception and enabling real-world applications in robotics and human- commuter interaction.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
The work is supported in part by the National Key R&D Program of China under Grant 2024YFB4708200 and the National Natural Science Foundation China under Grant U24B20173.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint (rXii:2303.08777 2023.
Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. In ECCV, 2020a.
Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas J. Guibas. ReferIt3D: Neural listeners for fine-grain fine-grained 3d object identification in real-world scenes. In ECCV, 2020b.
Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question an- swering for spatial scene understanding. In CVPR, 2022.
Daigang Cai, Lichen Zhao, Jing Zhang, Lu Sheng, and Dong Xu. 3djeg: A unified framework for joint dense captioning and visual grounding 3d point clouds. In CVPR, 2022.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and
Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV. Springer, 2020.
Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint arXiv: 709.06158.. 2017.
Chun-Peng Chang, Shaoxiang Wang, Alain Pagani, and Didier Stricker. Mikasa: Multi-key-anchor & scene-aware transformer for 3d visual grounding. In CVPR, 2024.
Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In ECCV. Springer, 2020.
Sijin Chen, Hongyuan Zhu, Xin Chen, Yinjie Lei, Gang Yu, and Tao Chen, End-to-end 3d dense captioning with vote2cap-detr. In CVPR, 2023.
Xinpeng Chen, Lin Ma, Jingyuan Chen, Zequn Jie, Wei Liu, and Jiebo Luo. Real-time referring expression comprehension by single-stage grounding network. arXiv preprint arXiv:1812.03-422 2018.
Zhenyu Chen, Ali Gholami, Matthias Nießner, and Angel Chang, Scan2cap: Context-aware dense captioning in rgb-d scans. In CVPR, 2021.
Yong Xien Chng, Xuchong Qiu, Yizeng Han, Yifan Pu, Jiewei Cao, and Gao Huang. Explor- ing contextual modeling with linear complexity for point cloud segmentation. arXiv preprint arXiv:2410.21211, 2024.
Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In CVPR, 2019.
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nicßner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 2017.
Bosheng Ding, Chengwei Qin, Ruochen Zhao, Tianze Luo, Xinze Li, Guizhen Chen, Wenhan Xia, Junjie Hu, Anh Tuan Luu, and Shafiq Joty. Data augmentation using Ilms: Data perspectives, learning paradigms and challenges. arXiv preprint arXiv:2403.02999 2024.

---

Published as a conference paper at ICLR 2025
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. arXiv preprint arXiv:2103.10360, 2021.
Lisa Dunlap, Alyssa Umino, Han Zhang, Jiezhi Yang, Joseph E Gonzalez, and Trevor Darrell. Di- versify your vision datasets with automatic diffusion-based augmentation. NeurIPS, 36, 2023.
Chun Feng, Joy Hsu, Weiyu Liu, and Jiajun Wu. Naturally supervised 3d visual grounding with language-reggllaizzed concept learners. In CVPR, 2024.
Liang Geng and Jianqin Yin. Viewinfer3d: 3d visual grounding based on embodied viewpoint inference. IEEE RA-L, 2024.
Zoey Guo, Yiwen Tang, Ray Zhang, Dong Wang. Zhigang Wang, Bin Zhao, and Xuelong Li. Viewrefer: Grasp the multi-view knowledge for 3d visual grounding. In ICCV, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Shuting He and Henghui Ding. Refmask3d: saaguage--uideed transformer for 3d referring segmen- tation. arXiv preprint urXiv:2407.182444 2024.
Haifeng Huang, Yilun Chen, Zehan Wang. Rongjie Huang, Runsen Xu, Tai Wang, Luping Liu, Xize Cheng, Yang Zhao, Jiangmiao Pang, et al. Chat-scene: Bridging 3d scene and large language models with object identifiers. NeurlPS, 2024a.
Rui Huang, Songyou Peng, Ayca Takmaz, Federico Tombari, Mare Pollefeys, Shiji Song, Gao Huang, and Francis Engelmann. Segment3d: Learning fine-grained class-agnostic 3d segmen- tation without manual labels. In ECCV, 2024b.
Rui Huang, Henry Zheng, Yan Wang, Zhuofan Xia, Marco Pavone, and Gao Huang. Training an open-vocabulary monocular 3d detection model without 3d data. NeurlPS, 2024c.
Shijia Huang, Yilun Chen, Jiaya Jia, and Liwei Wang. Multi-view transformer for visual ground- ing. In CVPR, 2022.
Ayush Jain, Nikolaos Gkanatsios, Ishita Mediratta, and Katerina Fragkiadaki. Bottom up top down detection transformers for language grounding in images and point clouds. In ECCV. Springer, 2022.
Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi- Chi-Wing Fu, and Jiaya Jia. Pointgroup: Dual-set point grouping for 3d instance segmentation. In CVPR, 2020.
Zaid Khan, Vijay Kumar BG, Samuel Schulter, Xiang Yu, Yun Fu, and Manmohan Chandraker.
Q: How to specialize large vision-language models to data-scarce vqa tasks? a: Self-train on unlabeled images! In CVPR, 2023.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source laaoon-annnaggeeetti model. arXiv preprint XXXi:24406009246 2024.
Yue Liao, Liu, Guanbin Li, Fei Wang, Yanjie Chen, Chen Qian, and Bo Li. A real-time cross- modality correlation filtering method for referring expression comprehension. In CVPR, 2020.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019. URL htllhttps://aaaee wwwpvv/..ggaabbb//...oa/
11692
Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela I. Rus, and Song Han. Bevfusion: Multi-task multi-sensor fusion with unified bird's-eye view representation. In ICRA. IEEE, 2023.
Junyu Luo, Jiahui Fu, Xianghao Kong, Chen Gao, Haibing Ren, Hao Shen, Huaxia Xia, and Si Liu. 3d-sps: Single-stage 3d visual grounding via referred point progressive selection. In CVPR, 2022.
12

---

Published as a conference paper at ICLR 2025
Jong Hak Moon, Hyungyung Lee, Woncheol Shin, Young- Hak Kim, and Edward Choi. Multi-modal understanding and generation for medical images and text via vision- saiinnlanguage pre-training. IEEE Journal of Biomedical and Health Informatics, 26(12), 2022.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4, arXiv preprini urXiv:2304.0322 2023.
Thinh Phan, Khoa Vo, Duy Le, Gianfranco Doretto, Donald Adjeroh, and Ngan Le. Zeetad: Adapt- ing pretrained vision-language model for zero-shot end-to-end temporal action detection. In WACV, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée
Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. /3971, 2023.
Johanna Wald, Armen Avetisyan, Nassir Navab, Federico Tombari, and Matthias Nießner. Rio: 3d object instance re- re-localization in changing indoor environments. In ICCV, 2019.
Tai Wang, Xiaohan Mao, Chemming Zhu, Runsen Xu, Ruiyuan Lyu, Peisen Li, Xiao Chen, Wenwei Zhang, Kai Chen, Tianfan Xue, Xihui Liu, Cewu Lu, Dahua Lin, and Jiangmiao Pang. Embod- iedscan: A holistic multi-modal 3d perception suite towards embodied ai. In CVPR, 2024.
Tung- Yu Wu, Sheng- Yu Huang. and Yu- Yu-Chiang Frank Wang. Dora: 3d visual grounding with order- aware referring. arXiv preprint arXiv:2403. 165.9 2024a.
Xiaoyang Wu, Li Jiang, Peng-Shuai Wang, Zhijian Liu, Xihui Liu, Yu Qiao. Wanli Ouyang, Tong
He, and Hengshuang Zhao. Point transformer v3: Simpler faster stronger. In CVPR, 2024b.
Yanmin WU, Xinhua Cheng, Renrui Zhang. Zesen Cheng, and Jian Zhang. decoupling and dense alignment for 3d visual grounding. In CVPR, 2023.
Eda: Explicit text-
Li Yang, Ziqi Zhang, Zhongang Qi, Yan Xu, Wei Liu, Ying Shan, Bing Li, Weiping Yang, Peng Li, Yan Wang, et al. Exploiting contextual objects and relations for 3d visual grounding, In NeurlPS, 2024.
Sibei Yang, Guanbin Li, and Yizhou Yu. Dynamic graph attention for referring expression compre- hension. In ICCV, 2019.
Lichen Zhao, Daigang Cai, Lu Sheng, and Dong Xu. 3dvg- tdwg-transformer: Relation modeling for visual grounding on point clouds. In ICCV, 2021.
Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 3d-vla: 3d vision-language-actin generative world model. arXiv preprint
arXiv:2403.09631, 2024.
Henry Zheng, Hao Shi, Yong Xien Chng, Rui Huang, Zanlin Ni, Tan Tianyi, Peng Qihang, Weng Yepeng, Zhongchao Shi, and Gao Huang. Denseg: Alleviating vision-language feature sparsity in multi-view 3d visual grounding. In CVPRW, 2024.
Benjin Zhu, Zhengkai Jiang, Xiangxin Zhou, Zeming Li, and Gang Yu. Class-balanced grouping and sampling for point cloud 3d object detection. arXiv preprint arXiv: 2019.
Chenming Zhu, Wenwei Zhang, Tai Wang, Xihui Liu, and Kai Chen. Object2scene: Putting ob- jects in context for open-vocabulay 3d detection, 2023a. URL htt.h/tpss///exi..orr////// 0309909453
Ziyu Zhu, Xiaojian Ma. Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 3d-vista: Pre- trained transformer for 3d vision and text alignment. In CVPR, 2023b.
13

---

Published as a conference paper at ICLR 2025
A
APPENDIX
CONTENTS
A.1
More Implementation Details
14
A I. Enhanced Baseline
14
1.2 Hyperparameters
14
A.2
Performance on Individual Classes
14
A.3
Analysis Limited Data Scenario
15
A.4
Robustness Analysis
15
A.5
Implementation Details of LSE
15
A.5.I
Prompt for LSE module
15
A.5.2
Details SIDB
17
A.1
MORE IMPLEMENTATION DETAILS
A.1.1 ENHANCED BASELINE
First, we replace the RoBERTa (Liu et al. 2019) language encoder with the CLIP (Radford et al., 202T) encoder. Given that visual grounding requires deep understanding of both linguistic in- structions and environmental visual features, strong cross-modal alignment is critical for success. CLIP, specifically designed for such tasks, provides superior alignment between language and vi- sion, making it natural fit for this application. Moreover, visual grounding often involves identify- ing a diverse range of objects, and the original EmbodiedScan model relies on a detection pipeline to pretrain the visual feature encoders. However, this approach can be hampered by the long- loigetailed distribution of objects in the pretraining dataset, leading to suboptimal detection performance. To ad- dress this, incorporate the 2019) during pretraining, which has proven effective in mitigating data imbalance and enhancing detection accuracy across rare and common object categories.
Grouping and Sampling (CBGS) method (Zhu al.)
A.1.2 HYPERPARAMETERS
Our multi-view visual grounding model, DenseGrounding, is trained with the AdamW optimizer using a learning rate of 5e-4, weight decay of 5e-4, and batch size of 48. The model is trained for 12 epochs, with the learning rate reduced by 0.1 at epochs 8 and 11. All other settings align with EmbodiedScan.
A.2 PERFORMANCE ON INDIVIDUAL CLASSES
We show the performance of our method on the individual classes in the validation in Fig.
Figure 4: Performance of our method on each class in validation of EmbodiedScan
14

---

Published as a conference paper at ICLR 2025
A.3 ANALYSIS ON LIMITED DATA SCENARIO
In Figure [ we present a comparative analysis of the performance between our method and the baseline under varying amounts of training data. The results clearly demonstrate that method consistently outperforms the baseline, especially in scenarios with limited data availability. Notably, when trained on just 40% of the training data, our method surpasses the baseline model that was trained on 80% of the data. This highlights the data efficiency and robustness of our approach, indicating its effectiveness even when training data is scarce.
Scaling Asalysis on Limited Data Scenario
0.38
Accursoy:
0.54
70
100
Amount
Figure 5: Comparison of DenseGrounding and EmbodiedScan on limited data scenario.
A.4 ROBUSTNESS ANALYSIS
Table 6: Zero-shot Cross Dataset Performance
Method
Training Scenes
Testing Scenes
Easy
Hard
Overall
EmbodiedScan
SSScan+MMtttrpoor
Scannet
9.95
7.94
9.81
DenseGrounding
3R 3RScan+Matterport
Scannet
18.57
12.42
18.14
Cross-dataset evaluation is essential for testing the generalization capabilities and robustness of egocentric 3D visual grounding methods. This process is particularly challenging due differences in camera settings, scene layouts, and object characteristics in visual data across various datasets, which can significantly impact model performance. To further assess the zero-shot generalization of our approach, we reorganized the EmbodiedScan dataset to create zero-shot setting. Specifically, we trained our model exclusively on scenes from 3RScan and Matterport3D) and then evaluated its performance ScanNet scenes, which were entirely unseen during training. This setup tests the model's s ability to adapt to new environments with different camera settings and scene characteristics. As shown in Table 6 our method significantly outperforms the baseline in this zero-shot setting, demonstrating superior robustness and the ability to generalize effectively to new and diverse scenes despite the inherent cross-d challenges.
A.5 IMPLEMENTATION DETAILS OF LSE
In this section, we provide the implementation details of the Language Semantic Enhancer (LSE) module, focusing on how the LLM prompted. We delve into the specific prompts used and explain how information is selected to provide context to the LLM.
A.5.1 PROMPT FOR LSE MODULE
In this section, we present the prompts used in the Language Semantic Enhancer (LSE) module, along with example input and output, illustrated in Fig.
15

---

Published as a conference paper at ICLR 2025
[System]
You AI spatial reasening ossistont specialized in rephrasing expressions related to 3D indoor scenes. Your task is to rephrase the given expression, enhancing the accuracy of spatial relationships based on the provided context.
Below are some expressions (visual grounding descriptions) that describe the scene and one specific expression that needs rephrasing. The text after LOCA TION INFO provides the 3D position of the objects centers (unit: meters) the lidar coordinate system. Enhance this specific expression accuracy and diversity by using anchor points.
NOTE:
1. Input:
Several visual grounding descriptions that provide reliable positional information. Each description includes the 3D coordinates of objects, the coordinates are given as (x, y. z) with units in meters.
A visual grounding description that needs rephrasing.
2. Output: A rephrased visual grounding description.
3. Ensure that the target object remains unchanged.
4. Use the pasitional relationships of more objects in the scene to describe it, ensuring higher accuracy and without any potential ambiguity. Only describe spatial relationships with high confidence: avoid adding information or assumptions.
6. Understand the spatial relationships between objects based their 3D coordinates supplement
to spatial information.
7. DO NOT provide the LOCATION TION INFO in the rephrosed expression.
8. Just output the rephrased sentence directly, prohibit outputting any other statements.
Example 1 (maybe not this scene): Before rephrase:
select the pillow that is the decoration. LOCATION INFO: pillow located at (2.5, 2.8, 0.7). decoration located at (1.25, 3.5, 0.5)
After rephrase:
Choase the pillow near the decoration, it also situated beneath the mirror and positioned between the fireplace the decoration
Example 2 (maybe not in this scene):
....
[User]
Visual Grounding Descriptions:
choose the clock that is front the towel. LOCATTIO INFO: clock located at (0.04, -1885, 1.52).
towel is located at (0.15, 1.97, 0.55)
select the clock that is farthest from the pitcher. LOCATION INFO: clock is located at (0.07, -2088,
1.85). pitcher located at (-1.44, 1.50, 0.85)
the clock that is closer the chandelier. LOCATION clock is located at (0.04, -1.85, 1.52).
chandelier is located at (0,24, 2,38, 2,20)
Visual grounding description that needs rephrasing: the clock that is closer the refrigerator. LOCA INFO: clock is located at (0.04, -1.85, 1.52). refrigerator located (-1.05, 0,50, 0.76) Now, rephrase according the above requirements.
[Assistant]
Choose the clock that is closer to the refrigerator, positioned beneath the chandelier and in front of the towel
Figure Sample Input and Output LLM for Language Enhancement
16

---

Published as a conference paper at ICLR 2025
A.5.2 DETAILS ON SIDB
Context Description Selection. Given a text description to be augmented (i.e., the raw description) and a SIDB containing text descriptions (i.e., context descriptions), we select relevant context descriptions (we used k=50) to provide the LLM with scene and object relationship information. Specifically, when selecting descriptions as context to be utilized by the LLM for augmentation, we find descriptions in the SIDB that both belong to the same scene as the raw description and contain the target or anchor class the raw description in their text. If more than 50 descriptions qualify, randomly select 50 descriptions from among them. Conversely, if fewer than 50 descriptions qualify, we randomly select from the descriptions in the same scene. This strategy simplifies the process of context selection and allows for diversity in the context provided to the LLM.
17


Reviews:
[{"review_id": "Y5Nsrll3kt", "reviewer": "Reviewer_WKF8", "summary": "This paper introduces DenseGrounding, designed to address the issues of sparse fine-grained visual semantics and ambiguous language descriptions in ego-centric 3D visual grounding tasks. DenseGrounding includes two main modules: the Hierarchical Scene Semantic Enhancer (HSSE) and the Language Semantic Enhancer (LSE), each focused on enhancing semantics in the visual and language domains, respectively. The HSSE effectively captures fine-grained global semantics and promotes alignment between visual and textual inputs, while the LSE utilizes large language models (LLMs) to enrich language descriptions, increasing semantic richness and reducing inherent ambiguities. DenseGrounding achieves SOTA performance on the EmbodiedScan benchmark and secured first place in the CVPR 2024 Autonomous Driving Grand Challenge Track on Multi-View 3D visual grounding, underscoring its potential for real-world applications.", "questions": "The proposed model performs excellently on the ego-centric 3D visual grounding task. However, I am curious whether it could also be applied to common 3D visual grounding tasks, such as the ScanRefer benchmark.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed LSE module applies LLM to enrich text semantics, introducing a novel perspective to this task.\n2. The proposed HSSE module effectively preserves fine-grained semantic features by first extracting semantic features from individual views and then broadcasting them across the entire scene, thus addressing the issue of semantic sparsity.\n3. The paper provides a clear definition of the novel ego-centric 3D visual grounding task, making it accessible and easy to follow. The design of the method’s modules and the implementation of the ablation studies are also well-explained, allowing readers to easily understand the structure of DenseGrounding and the improvements achieved.\n4. The proposed method achieves SOTA performance on the ego-centric 3D visual grounding task and earned first place in the CVPR 2024 Autonomous Driving Grand Challenge Track for multi-view 3D visual grounding, highlighting its potential for real-world applications.", "weaknesses": "1. The proposed method lacks descriptions for certain model details, such as the Multi-scale Fusion and Query Selection components in Figure 2.\n2. The paper does not provide a sufficiently clear explanation of how the LLM leverages the SIDB to enrich the semantics of descriptions. For instance, what is the specific format of the location information in the SIDB, and how does the LLM utilize this location information?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rHLUzczenN", "reviewer": "Reviewer_1bvp", "summary": "This paper aims to address the challenges of loss of fine-grained visual semantics and limited textual semantic context in ego-centric 3D visual grounding. It proposes a hierarchical scene semantic enhancer to enrich visual semantics, while leveraging a large language model (LLM) to enhance context and provide diverse textual semantics. Experimental results demonstrate that the proposed DenseGrounding achieves state-of-the-art results.", "questions": "See weaknesses for details.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-organized and facilitates ease of comprehension.\n- The paper introduces an improved baseline, and DenseGrounding achieves a significant performance enhancement.\n- The idea of enhancing visual and linguistic features is reasonable; however, there is a potential risk of data leakage (which will be discussed later).", "weaknesses": "- Potential unfair comparisons and prior information leakage. From the ablation study in Table 3, it can be observed that the primary performance improvement of the method derives from the LSE module, which constructs a database and utilizes an LLM for enhancement. However,\n    1. The construction of the database may leak crucial dataset priors, as it involves prior knowledge of other contextual objects in the scene and their relationships and localization information, even within the training set. This information should ideally be learned and acquired automatically through ego-centric multiview images.\n    2. The database essentially constructs local contextual information between objects, which can significantly enhance the performance of 3D object detection. However, this information is extracted offline and then used as a shortcut to assist DenseGrounding. \n    3. Minor: the use of the LLM introduces additional costs and lacks flexibility.\n- The performance improvement of the HSSE module is modest, particularly in hard cases, where it may even have a negative impact.\n- The novelty of the HSSE module is somewhat limited, as utilizing multi-scale features and employing vision-language fusion for enhancing visual features are common techniques in the vision-language domain.\n- The description of the construction and selection of scene information in the dataset is not sufficiently specific. It is necessary to clarify what types of contextual information are being utilized, such as how each contextual object is defined or how specific objects are selected to consider their relationships.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jCkC8oEhVf", "reviewer": "Reviewer_RUeb", "summary": "This paper proposes **DenseGrounding**, an interesting approach aimed at enhancing ego-centric 3D visual grounding with focus on two challenges\n\n- Loss of Fine-Grained Visual Semantic: Sparse fusion of point clouds and multi-view images leads to semantic loss, which can hinder object detection in 3D space.\n- Limited Texture Semantics: Ego-centric 3D visual grounding often has sparse or ambiguous language descriptions, which can lead to issues in object localization.\n\nAnd this paper introduces two components:\n\n- **Hierarchical Scene Semantic Enhancer (HSSE)**: This module enhances visual semantics by incorporating multi-view, scene-level information and facilitating cross-modal (text and visual) alignment.\n- **Language Semantic Enhancer (LSE)**: Leveraging large language models (LLMs) with a structured Scene Information Database (SIDB), this component augments language descriptions with more contextually rich, diverse inputs.\n\n\nThey obtained strong results and the experiments are solid.", "questions": "- Could you evaluate the model on additional benchmarks to assess its generalization, and report on its zero-shot performance?\n- Could you provide an analysis of the model’s performance across varying data sizes, compared to the baseline?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper's writing and motivations are both strong.\n\n2. HSSE module for visual enhancement and LSE for text enhancement addresses two crucial limitations and make the proposed algorithm  effective towards real-world ego-centric tasks.\n\n3. This proposed methods achieved competitive results on the EmboidiedScan benchmark.", "weaknesses": "- I am not too surprised to see DenseGrounding shows improved results on EmbodiedScan, a domain-specific dataset. However, the performance drop on the “hard” split in the HSSE ablation suggests limitations in handling complex scenes, raising doubts about whether it genuinely enhances spatial understanding or is just tuned to EmbodiedScan’s data characteristics. Testing on diverse datasets could clarify its robustness.\n- Lack of Scaling Analysis: While the paper compares models trained on different data sizes, it lacks a detailed scaling law analysis. Understanding how performance scales with varying data sizes is crucial for practical applications, especially in data-scarce environments. A scaling study would show if DenseGrounding efficiently learns from limited data.\n- Dependency on Complete Scene Graphs: The Language Semantic Enhancer (LSE) assumes access to a complete, accurate scene graph, which is rarely available in real-world settings. While using a predicted scene graph could mitigate this, the potential noise and errors in such predictions could degrade performance. Exploring the model’s robustness with imperfect scene graphs would enhance its practical relevance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Vww3rPoclk", "reviewer": "Reviewer_3qfg", "summary": "In this paper, they propose a model for the ego-centric 3Dvisual grounding task where  the model finds a 9DoF 3D bounding box from multiple RGB-D images and a language description for the specified object. Based on the EmbodiedScan, they introduced two modules: Language Semantic Enhancer (LSE) and Hierarchical Scene Semantic Enhancer (HSSE). In LSE, they use the predicted bbox and label and a Scene Information Database (SIDB) for augmenting the input text. HSSE uses view-level semantic aggregation. It is notable that SIDB depends on the annotated dataset (see L. 312) during the construction. They achieved the best results in the EmbodiedScan dataset, although the detailed analyses are limited, compared with those performed in the previous EmbodiedScan paper.", "questions": "1. Is it possible to construct SIDB without using the dataset annotations? DId you perform ablation for this? \n2. Can you clarify whether you use real or rendered images for the experiments? Also can you provide the ablation study of real vs rendered images as of Table 7 of  EmbodiedScan paper?\n3. How do you treat augmentation errors in LSE?\n4. Is it possible to report the detailed results, e.g., object category-wise result, as of EmbodiedScan?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Proposing Language Semantic Enhancer (LSE) for the text input augmentation. LSE depends on Scene Information Database (SIDB), which uses the scene annotation.\n2. Proposing Hierarchical Scene Semantic Enhancer (HSSE).\n3. The best results in the EmbodiedScan dataset although I am not sure the experimental settings are consistent with the existing models.", "weaknesses": "1. It seems that SIDB depends on the annotated dataset as explained in the L.312. Isn’t this cheating in the experiments with the EmbodiedScan dataset because the model can reach rich symbolic textual information? Did you perform some ablation study for this?\n2. The model improvement from EmbodiedScan seems incremental: the improvement of \n3. Some experimental settings seem still unclear from the paper. For example, it is not clear whether authors use rendered or real images in EmbodiedScan.\n4.  It is also expected that authors explain what are the inputs of the previous models in the main result table of Table 1. For example, ScanRefer depends on the 3D point cloud (pcd) and text. BUTD-DETR uses RGB-D as noted in Table 5 of the EmbodiedScan paper.\n5. The experimental setting of ego-centric 3D visual grounding is not fully explained in this paper. For example, it is not clear how they obtain the camera pose, e.g., intrinsic and extrinsic matrices are obtained from the writings of Section 3.1. If models depend on them, it is recommended that authors update manuscripts for explaining the experimental settings.\n6. The experiments are limited in the EmbodiedScan dataset. It is also interesting whether this method can be expanded for related datasets and tasks.\n7. Limited related studies. It is recommended that authors include a wide branch of 3D visual groundings and question answering tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GUG6u0dMj3", "reviewer": "Reviewer_9Djk", "summary": "This paper introduces a method for ego-centric 3D visual grounding, where agents are able to locate target objects in real-world 3D spaces based on verbal descriptions. The main goal of this paper is to address two critical challenges: 1. how to enhance the fine-grained visual semantics by improving the sparse fusion between ego-centric multi-view images, and 2. how to improve the textual semantic context in arbitrary language descriptions. To tackle these issues, the authors first introduce the Hierarchical Scene Semantic Enhancer, which effectively captures fine-grained scene-level global semantics. Secondly, they introduce the Language Semantic Enhancer, which uses LLM-assisted description augmentation to enhance natural language descriptions. The method presented in this paper achieves the SOTA result on the EmbodiedScan benchmark, demonstrating its effectiveness.", "questions": "1. The details of selecting the top-k most related descriptions used in LSE are not listed in the paper;I am still confused about how to rank these descriptions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The motivation is clear, the improvement is substantial, and the paper is easy to follow.\n2. They enhance the language description quality on EmbodiedScan. By releasing the improved annotations, they can benefit future projects.", "weaknesses": "1. Section 4.2, 'Prompt for LLM-based Enhancement,' lacks implementation details, such as prompts, and needs more examples to demonstrate the enhanced descriptions as well as the statics about the enhanced annotations including amount, average length.\n2. The methods used in the hierarchical scene semantic enhancer lack novelty. Similar attention-based methods, such as BUTD-DETR [1], have been widely used in previous works. The author should emphasize the difference.\n3. Table 1 lacks proper citations.\n\n[1] https://arxiv.org/pdf/2112.08879", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)