LOCATE 3D: Real-World Object Localization via Self-Supervised Learning in 3D
Authors: Paul McVay
Sergio Arnaud
Ada Martin
Arjun Majumdar
Krishna Murthy Jatavallabhula
Phillip Thomas
Ruslan Partsey
Daniel Dugas
Abha Gejji
Alexander Sax
Vincent-Pierre Berges
Mikael Henaff
Ayush Jain
Ang Cao
Ishita Prasad
Mrinal Kalakrishnan
Michael Rabbat
Nicolas Ballas
Mido Assran
Oleksandr Maksymets
Aravind Rajeswaran
Franziska Meier
Venue: ICML 2025 spotlightposter (2025)
Keywords: self-supervised learning
object localization
referring expressions
3D language grounding
Primary area: applications->computer_vision
TL;DR: A model that can localize objects in 3D from textual referring expressions.

Abstract: We present LOCATE 3D, a model for localizing objects in 3D scenes from referring expressions like "the small coffee table between the sofa and the lamp." LOCATE 3D sets a new state-of-the-art on standard referential grounding benchmarks and showcases robust generalization capabilities. Notably, LOCATE 3D operates directly on sensor observation streams (posed RGB-D frames), enabling real-world deployment on robots and AR devices. Key to our approach is 3D-JEPA, a novel self-supervised learning (SSL) algorithm applicable to sensor point clouds. It takes as input a 3D pointcloud featurized using 2D foundation models (CLIP, DINO). Subsequently, masked prediction in latent space is employed as a pretext task to aid the self-supervised learning of contextualized pointcloud features. Once trained, the 3D-JEPA encoder is finetuned alongside a language-conditioned decoder to jointly predict 3D masks and bounding boxes. Additionally, we introduce LOCATE 3D DATASET, a new dataset for 3D referential grounding, spanning multiple capture setups with over 130K annotations. This enables a systematic study of generalization capabilities as well as a stronger model. Code, models and dataset can be found at the project website: locate3d.atmeta.com

For AI systems to effectively assist us in the physical world, such as through robots or smart glasses, an understanding of LOCATE 3D: Real-World Object Localization via Self-Supervised Learning in 3D
Paul McVay", "1
Sergio Arnaud
Ada Martin
Arjun Majumdar'
Krishna Murthy Jatavallabhula'
Phillip Thomas'
Ruslan Partsey
Daniel Dugas'
Abha Gejji
Alexander Sax' Vincent-Pierre Berges Mrinal Kalakrishnan
Mikael Henafr
Ayush Jain 12
Ang Cao 13
Ishita Prasad
Michael Rabbat
Nicolas Ballas
Mido Assran
Oleksandr Maksymets Aravind Rajeswaran
Franziska Meier
Abstract
the 3D world grounded in human natural language is essen- tial. Towards this goal, we study the task of 3D localization via referring expressions (Chen et al., 2020; Achlioptas et al., 2020), or simply 3D-REFEXP. It requires localizing an object (in 3D) from a textual expression that may include combination of attributes (e.g.. "red chair") and/or spa- tial relationships (e.g.. "the small coffee table between the sofa and the lamp"), "). In this work, we develop LOCATE 3D (see Figure 1), a state-of-the-an model for 3D-REFEXP. It builds on two key components: (1) 3D-JEPA, a novel SSL algorithm to learn contextualized scene representations, and (2) a novel language 3D localization decoder.
We present LOCATE 3D, model for localizing objects in 3D scenes from referring expressions like "the small coffee table between the sofa and the lamp." LOCATE 3D sets new state-of-the- art on standard referential grounding benchmarks and showcases robust generalization capabilities. Notably, LOCATE 3D operates directly on sen- sor observation streams (posed RGB-D frames), enabling real-world deployment on robots and AR devices. Key to our approach is 3D-JEPA, a novel self-supervised learning (SSL) algorithm applicable sensor point clouds. It takes as in- put 3D pointcloud featurized using 2D founda- tion models (CLIP, DINO). Subsequently, masked prediction in latent space employed as a pre- text task to aid the self-supervised learning of contextualized pointcloud features. Once trained, the 3D-JEPA encoder is finetuned alongside a language-condittoned decoder to jointly predict 3D masks and bounding boxes. Additionally, we introduce LOCATE 3D DATASET, a new dataset for 3D referential grounding, spanning multiple capture setups with over 130K annotations. This enables a systematic study of generalization capa- bilities well as a stronger model. Code, models and dataset can be found at the project website: vccate3d.atmmta.com
The task of 3D-REFEXT is challenging date. At one end of the spectrum are methods that train specialized models for this task (Jain et al., 2021; Chen et al., 2022) on small benchmark datasets. They often require human annotation at inference time in the form of detailed 3D meshes or object instance segmentation, making them difficult to deploy on real-world devices. At the other end are methods that try to leverage 2D VLMs for 3D tasks (Xu et al., 2024; Zhu et al., 2023). While these methods can encode rich linguistic structures by leveraging LLMs, they employ a simplistic and hand-crafted representation of the 3D world.
LOCATE 3D operates in three phases. In the first pre- processing phase, we leverage the underlying sensor ob- servation stream lift features from 2D foundation mod- els (Radford et al., 2021; Oquab et al., 2023) into 3D point clouds (Jatavallabhula et al., 2023). Subsequently, transformer encoder, pre-trained with our 3D-JEPA SSL algorithm, to transform the "lifted" foundation features into contextualized features that can provide better scene-level understanding. Finally, we use a language conditioned 3D decoder (Cheng et al., 2021: Kamath al., 2021) to localize the object of interest. Notably, LOCATE 3D operates directly sensor observation streams without requiring manual post post-processing (c.g., 3D mesh refinement or ground-truth instance segmentations), making it readily deployable on robots and AR devices.
use
1. Introduction
Equua contribution 'FAIR at Meta CCarnegie Mellon Univer- sity 'University of Michigan, Ann Arbor. Correspondence to: Sergio Arnaud <sergioarnaud @ meta.com>, Paul Vay <pme- vay@ meta.com>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).
We outline our contributions this work below.
1. 3D-JEPA A novel SSL method applicable to 3D point

---

LOCATE 3D; Real-World Object Localization via Self-Supervised Learning in 3D
RGB-D Data
Locate 3D
Text Query
*The chair by the entrance door"
Lift features from 2D models DINO)
3D-JEPA Encoder
Decoder
Fine-tuned
Predictions
RGB-D Sequence, Point Cloud
2D Lifted Features
3DJEPA Festures
Masks, Bounding Boxes
1. Preprocessing
2. Contextualized Representation
Localization
Figure I: Overall Architecture of LOCATE 3D, which operates in three phases. In Phase 1: Preprocessing. we construct point cloud with "lifted" features from 2D foundation models, which provide local information. In Phase 2: Contextualized Representations, these lifted features are passed through pre-trained 3D-JEPA encoder, which provides contextualized representation whole scene. Finally, in Phase 3: 3D Localization, a 3D decoder head uses the text query and 3D-JEPA features to localize the referred object.
clouds that can learn contextualized representations of 3D scenes. It takes inputs 3D point clouds with fea- tures lifted from 2D foundation models. The SSL. pretext task involves predicting the latent embeddings of ran- domly masked regions in the featurized point cloud (Ass- ran et al., 2023). We show that the resulting 3D-JEPA fea- tures are contextualized for the scene, while the features lifted from 2D foundation models only provide local understanding. Conceptually, this is analogous to the dif- ference between contextualized token embeddings (De- vlin et al., 2019) and word embeddings (Mikolov et al., 2013) in NLP. 3D-JEPA pre-training provides signif- icant performance gain to our LOCATE 3D model, for both in-domain (59.8% + 61.7 %) and out-of-domain (51.5% 56.7%) evaluations.
ization capabilities on held-out scenes and annotations in ScanNet++. Finally, it also successfully localizes ob- jects in multi-room test environment enabling robotic mobile manipulation in an unseen environment.
3. LOCATE 3D DATASET (L3DD) A new dataset for 3D RefExp which spans ScanNet (Dai et al., 2017), Scan- Net++ (Yeshwanth et al., 2023), and ARKitScenes (De- hghan al., 2021). L3DD covers 1,346 scenes and over 130K language annotations. It allows us to study the robustness of LOCATE 3D to variety of capture setups and independent samplings of indoor environments. It also serves as a source of training data for RefExp mod- els; while LOCATE 3D already achieves SoTA perfor- mance using only standard benchmark training datasets (61.7%) ), L3DD training data further improves perfor- mance of our approach on these benchmarks (61.7% 63.7%). We call this model LOCATE 3D+.
2. LOCATE 3D- a model for 3D RefExp (see Figure 1) that achieves SoTA benchmark results and strong out-of- domain generalization. LOCATE 3D leverages our 3D- JEPA and fine-tunes tie-ttnes it for the 3D RefExp task using Lunguage-oonditioned 31D decoder. The decoder performs interleaving cross-attention between the 3D features and text queries, and employs joint mask and bounding box prediction strategy. On standard 3D RefExp bench- marks (SeanRefer, SR3D, NR3D), LOCATE 3D achieves SoTA results compared to prior work (58.5% 61.7%) Crucially, LOCATE 3D achieves these impressive results with fewer assumptions compared to prior models. Lo- CATE 3D does not require ground-truth region proposals, meshes, surface normals at inference time, making it suitable for real-world deployment. When compar- ing to prior work under similar assumptions (Jain et al., 2021), LOCATE 3D presents a significant advancement (40.7% + 61.7%). further exhibits strong general-
2. LOCATE 3D: Model Overview and Training
The overall architecture of LoCATE 3D is presented in Fig- ure 1. LOCATE 3D is designed to operate RGB-D sensor observations of static environments (e.g.. homes in which objects remain stationary over short intervals). LOCATE 3D contains three modules: (1) Preprocessing: using 2D foundation models to construct a featurized 3D point cloud (Section 2.1). (2) Contextualized Representations: pro- duced by SainTraanformer-v3 (PTv3) (Wu et al., 2023) encoder that operates on the featurized point cloud to gen- erate a contextualized 3D representation. The encoder is pre-trained with our novel SSL algorithm, 3D-JEPA (Sec- tion 2.2). (3) Localization: using a language- language-conditionee conditiionddi 3D object localization decoder that jointly predicts masks and bounding boxes (Section 2.3.1). The decoder is trained
2

---

LOCATE 3D: Real-World Object Localization via Self-Supervised Learning in 3D
from scratch, jointly with the 3D-JEPA initialized PTv3 encoder for the referential grounding task (Section 2.3.2).
Imput
Masking
Masiad Inpul
2.1. Preprocessing: Lifting 2D Foundation Model Features into 3D Point Clouds
We begin by preprocessing the inputs (posed RGB-D im- ages) by constructing 3D pointcloud to encode geome- try, and featurizing the pointcloud with off-the-shelf 2D foundation models to encode semantic information. We compute vision-only features from DINOv2 (Oquab et al., 2023), which are dense (patch-level). We also extract vision- language features from CLIP (Radford et al., 2021). Since CLIP features are global (i.e., one feature per input image) and not dense, we extract 2D instance masks from the in- put images using SAM (Kirillov et al., 2023) and compute per-mask CLIP features instead. These features are mapped back to the pixels containing the masks. We concatenate the CLIP and DINO features, along with a harmonic encoding of the RGB pixel intensities to obtain dense 2D feature maps. These feature maps are lifted to 3D similarly to (Jataval- labhula et al., 2023). We begin by unprojecting the RGB-D image to obtain a pointcloud, followed by voxelization (we use a voxel size of 5 cm across our experiments). We then compute a single feature per voxel by weighted averaging of all contained features. Weights are calculated using tri- linear interpolation based on distance to voxel boundaries. This process results in a point cloud with "lifted" features, PtCun= = {2,,, Vis 24, f.) ) where f, E Rd is the feature vector of the 5th point. This point cloud PtCin can be directly used for object localization. However, we find that refining the features further with an encoder, pre-trained with self-supereised learning (as described in Section 2.2), results in substantially better localization.
E,, Target Encoder
Eg Context Encoder
Mask (f meny)
Feutures
Mesitt %
IKs,y) Lossiin in Embeddiin Space
Predicted Festures
Masked Contextua Ized Features
Figure 2: 3D-JEPA training framework: The context encoder computes latent features from a masked point cloud. Subsequently, predictor operates on these latent features to predict the features of masked regions. The target encoder has the same architecture as context encoder with weights being the exponentated moving average of context encoder over course of training. The loss is computed per point in the embedding space and averaged across all points that were masked.
contains lifted 2D foundation features). 3D-JEPA trains two transformer models E-(-) and Po(-) using the following objective,
min |IPo (Eo(PiC), - sg (Eo ||. 0.6
(I)
where PiC denotes a masked view of the point cloud, M is a mask variable describing regions that were masked, and sg(.) is the stop-grad operator. Finally, E-)-) is the exponentiated moving average version of the encoder, which is important to prevent the representation from collapsing.
2.2. 3D-JEPA: Learning Contextualized
The loss is computed per-point for all points in the masked region and then averaged. After training, E-(-) is used as the contextualized representation.
Representations via SSL
3D-JEPA takes as input the lifted features Pt. from above, and learns contextualized representation for the scene. Unlike the features in PtC which are local just the objectmaask/pachh, we seek learned features that attend to different parts of the scene to obtain representations that capture the entire context of the scene. This is analogous to learning contextualized embeddings (Devlin et al., 2019;TB Pe- ters et al., 2018) as opposed to word-embeddings (Mikolov et al., 2013) in language.
Performing masked prediction in the latent space is advan- tageous over doing in the ambient/input space for two reasons. First, unlike standard MAE in 2D vision, our input space is high-dimensional. In fact, since we use lifted 2D foundation features, our input dimension is equivalent to the ViT feature dimension. We found directly reconstruct- ing such fine-grained and digh-dimensional features to be difficult. Second, methods that leverage masked prediction in a teacher-student framework have produced the strongest results recently (Assran et al., 2023; Oquab al., 2023).
To learn such representation, we take inspiration from the Joint Embedding Predictive Architectures (JEPA) frame- work (Assran et al., 2023; Bardes et al., 2024) and develop 3D-JEPA. It is an prcoder-predictor framework that per- forms masked prediction (He et al., 2022; Devlin et al., 2019) in the learned latent space. Concretely, let PtC be the input point cloud with some features (in our case, it
To adapt the JEPA approach to 3D we developed a bounded Aadius-based masking strategy and efficient 3D native en- coder and predictor architectures.
Encoder and Predictor Architectures. Unlike images or voxels that contain a regular grid structure, point clouds
3

---

LOCATE 3D; Real-World Object Localization via Self-Supervised Learning in 3D
Encoder
Linear
3D-JEPA
NeJ
Query to Pointeloud Cross-Attention
Dok
Mask Predictiions
Features
MLP
Add
Repeated Decoder Blocks
2-Layer
Pointcloud to Query Cross-Attention
XYZ
Vidk
Box Predictions
Qx6
(n times)
Query Self- Self-Attention
Text Query
Ioef
Linear
C
MLP
Taken Predictions
OKT
Learned
Inputs
Queries
(
Concat
Point Features NxE
Text Features
Query Features QXE
Outputs
Figure 3: In our language-oonditioned.... 3D mask and bounding box decoder, 3D-JEPA features are jointly processed with text and learned query embeddings by decoder blocks and specialized prediction heads that generate mask, token, and hox predictions. N the number points in imput pointcloud. T the number of tokens the imput text, Q the number of generated model queries, E the decoder feature dimension, F is the input text feature dimension, is the 3D JEPA feature dimension.
are order invariant and set-valued. U-Net (Çiçek et al., 2016), PointNet (Qi et al., 2016), DeepSets (Zaheer al., 2017), and PointTransformer (Engel al., 2020) have all emerged as promising architectures for point clouds. For our implementation of 3D-JEPA, we use Point Transformer v3 (PTv3) (Wu et al., 2023) for the encoder. In each layer, it first serializes the point cloud based on local proximity using bijective space filling curves. Subsequently, points are grouped together and they attend to each other within the group, which is loosely analogous convolutions. For the predictor, we use similar serialization step, but then use standard transformer with sparse attention pattern. This allows for faster mixing of information from the start, due to lack of an explicit grouping. However, we found sparse attention to be crucial for training throughput and memory. as well as for training stability. See Appendix A. for further details about the architecture.
2.
3D DECODER
As illustrated in Figure 3, the decoder processes two inputs: the 3D-JEPA features Eg Eg (PtC))) and a text query t. These imputs are iteratively refined through transformer and then fed into parallel prediction heads that generate 3D mask and bounding box predictions for all referenced objects. The details of this architecture are described below.
Decoder Input Embedding, We first project the 3D-JEPA Eg (PtC representation into the model working dimen- sion E and add learned 3D positional embeddings. Simi- larly, we project the per-word CLIP (Radford et al., 2021) embeddings of the text query t. Next, we initialize set of learnable object queries and concatenate them with the language embeddings along the sequence dimension. In the following, we refer to the projected 3D-JEPA features as "point features" and the concatenated object queries and language tokens collectively as "queries."
Masking Patterns. We found the choice of masking pat- tern to play a crucial role in the quality of representations learned, consistent with prior literature (Assran et al., 2023; Bardes et al., 2024). In particular, among the variants we tried, the serialized percent masking pattern shown in Fig- ure 4 worked best. contains two salient components: (1) masking out "regions" (points close in proximity) instead of random points encourage better spatial understanding than simple local interpolation; and (2) masking out per- cent of the scene instead of a fixed size allowing for training on point clouds of varying spatial sizes.
Decoder Blocks. We apply sequence of self-attention and cross-attention operations between the point features and queries. Specifically, each decoder module consists of three attention blocks: (1) a self-attention block that enables queries to refine their representations through mutual in- teraction, (2) a cross-attention block where queries extract relevant information from point features enhance their contextual understanding, and (3) final cross-attention block that updates the point features, ensuring they in- formed by the refined queries. The first two blocks are stan- dard from prior work (Vaswani et al., 2023; Carion et al., 2020; Cheng et al., 2021), while the final cross-attention block is inspired by (Jain et al., 2025), which demonstrates the impact updating visual features for detection tasks in 3D. The decoder blocks are repeated for iterative refinement of the point features and queries. In LOCATE 3D, the num- ber of decoder blocks E = 768. We observe positive correlation between model scaling and performance, as discussed in Section 4.
2.3. Object Localization from Referring Expressions
To solve the 3D referring expressions task, we design Lunguage-condition 3D object localization decoder (Fig- ure 3) that operates on the contextualized representations produced by our 3D-JEPA encoder (Section 2.2). This sec- tion describes the decoder architecture (Section 2.3.1) and end-to-end training procedure (Section 2.3.2).
8 and model dimension is

---

LOCATE 3D: Real-World Object Localization via Self-Supervised Learning in 3D
Decoder Heads. Our decoder consists of three parallel prediction heads (Figure 7) that process the refined learned queries Q independently as object proposals. Specifically, for each query, we jointly predict a 3D bounding box and 3ID mask, using dedicated prediction heads. Additionally, following (Kamath et al., 2021), we predict an alignment matrix that grounds each query by determining which noun in the referring expression it corresponds to, if any.
itScenes (Dehghan et al., 2021). This section describes the dataset, and Section 4 discusses the impact of using L3DD) to train our 3D referring expressions model.
3.1. Dataset Statistics
In total, our dataset contains 13 131,641 samples. Decomposed by scene dataset, L3DD contains:
1. ScanNet: 30,135 135 new language annotations covering 550 venues and 5,527 objects for training. 4,470 new language annotations covering 130 venues and 1038 objects for validation.
The mask head follows the approach of (Cheng et al., 2021), by processing the queries with an MLP and computing dot product with the point features to predict per-point mask logits. The text alignment head is composed of an MLP that receives the queries and directly predicts the alignment ma- trix. For bounding boxes, we developed a novel architecture (Figure 7). First, we concatenate linearly projected x, y, 2 coordinates to the refined point features along the feature dimension. Then, we perform cross-attention between these concatenated point features and a linear projection of the refined queries. Finally, for each query, we use an MLP to regress bounding boxes.
2. ScanNet++: 91,846 new language annotations cover- ing 230 venues and 13,359 objects for training. 3,774 new language annotations covering 50 venues and 1,303 objects for validation.
3. ARKitScenes: 991 new language annotations covering 293 venues and 1,862 objects covering scenes used for pretraining. 425 new language annotations covering 93 venues and 460 objects for validation.
All validation split samples were validated by least 1 human annotator. Over 80 percent of ARKitScenes and ScanNet++ validation split samples were validated at least three times, and samples were only included if a majority of annotators agreed that the sample was unambiguously correct.
2.3.2. TRAINING LOCATE 3D
LOCATE 3D trains the language-conditiones 3D decoder from scratch and fine-tunes the 3D-JEPA pretrained PTv3 encoder. It does so by using supervision signals from both object masks and bounding boxes to explicitly combine spatial constraints (boxes) and dense semantic supervision (masks); which leads to better localization performance (see experiments in Section 4.3).
3.2. Comparison with prior datasets
As shown in Table 9, L3DD significantly increases the scale of existing 3D RefExp data along two key axes when com- pared to prior data - language annotation quantity and scene coverage. Our language annotations approximately double the available quantity of training data, and after adjusting for the size of scenes, we approximately quintuple the num- ber of size-adjusted venues with dense RefExxp supervision. These annotations span multiple scene datasets, allowing principled study of scene generalization while holding the annotation process fixed. We show in Table 10 that this additional scene diversity is key to the value of L3DD as training data holding the number of annotations fixed, training on ScanNet and ScanNet++ together significantly outperforms training only on ScanNet annotations (61.8% + 63.2% recall 0.225 on DRSD//NR3DDssaaReeenn Dataset visuals, collection procedure, and further analysis of L3DD is available in Appendix D.
Specifically, LOCATE 3D optimizes composite loss func- tion, which includes: (1) a mask loss, combining Dice and cross-entropy loss terms (Cheng et al., 2021); (2) a bounding box loss, composed of LI distance and generalized IoU (Car- ion et al., 2020) terms; and (3) a text alignment focal loss with alpha balancing. Following (Carion et al., 2020), we de- fine a matching cost and use Hungarian Matching to assign object query predictions to ground truth objects. We apply progressively weighted deep supervision at every decoder layer and maintain Exponential Moving Average (EMA) of the model weights to use for evaluation and inference (Izmailov et al., 2018). In order to not destroy the pretrained features we use a stage-wise learning rate scheduler (Kumar et al., 2022); specifically we start by training the decoder with frozen encoder features, then fine-tune both jointly with lower learning rate for the encoder. Further training details are provided in Appendix C.
4. Experiments and Analysis
3. LOCATE 3D DATASET Overview
In this section, we report results for our trained models. LoCATE 3D is trained and evaluated the standard 31) ref- erential grounding benchmarks SR3D, NR3D (Achlioptas et al., 2020), and ScanRefer (Chen al., 2020). LOCATE 3D+ additionally incorporates our newly collected L3DD
LOCATE 3D DATASET (L3DD) is a new human-annotated referring expression dataset covering ScanNet (Dai al., 2017), ScanNet++ (vl) (Yeshwanth et al., 2023), and ARK-
5

---

LOCATE 3D: Real-World Object Localization via Self-Supervised Learning in 3D
Table 1: Results on 3D language grounding in 3D mesh and sensor point clouds (PC). We evaluate top-1 accuracy on the validation set without any assumption of ground-truth proposals. We find that model, LOCATE 3D, achieves mate-of-the--ae results across all three benchmarks for top-1 accuracy @5.. "VLMM-aaselies localization decoder LOCATE 3D was trained on training splits from the benchmarks, while that LOCATE 3D+ uses henchmark data as well as L3DD.
evaluated 2,000 samples per benchmark (6,000 total). "Thhe
Joint Evaluation
SR3D
NR3D
ScanRefer
Acc @ 25
Acc
Acc
Acc
Acc @25
Acc
Acc @25
Acc
Method
@ 50
@ @25
@:50
@ 50
@50
Mesh PC
Referlt3DNet Achlioplas et al. (2020)
26.6
27.7
24.0
26.4
ScanRefer (Chen et al., 2020)
35.5
22.4
InstanceRefer (Yuan et al., 2021)
33.6
31.5
29.9
40.2
32.9
LanguageRefer (Roh et al., 2022)
39.5
28.6
SAT-2D (Yang et al., 2021)
37.14
35,4
31.7
44.5
30.1
BUTD-DETR (Jain et al., 2021)
50.28
52.1
43.3
52.2
39.8
3D- VisTA (Zhu et al., 2023)
53.1
48.1
56,5
51.5
47.7
42.2
51.0
46.2
PQ3D (Zhu al., 2024)
58.5
52.5
62.0
55.9
52.2
45.0
56.7
51.8
Sensor PC + Proposals from Mesh PC
3D-VisTA (Zhu et al., 2023)
45.9
41.8
47.2
43.2
42.1
37.4
46.4
42.5
ConcreteNet (Unal et al., 2024)
56.12
49.50
Sensor PC
3D-LLM (Hong ct al., 2023)
30.3
BUTD-DETT (Jain et al., 2021)
40.7
26.6
43.3
28.9
32.2
19.4
42.2
27.9
Llama VLM Baseline (Ours)*
28.8
18.3
21.3
13.9
28.0
16.9
37.3
24.2
GPT-4o VLM Baseline (Ours)*
38.6
25.5
29.2
18.9
38.2
25.1
48.2
32.5
LOCATE 3D (Ours)
61.7
49.4
65.8
52.9
53.7
40.5
59.9
49.6
LOCATE 3D (Ours)"*
63.7
51.3
68.2
54.8
56.1
43.2
61.1
50.9
dataset for training. We evaluate the validation split of the benchmarks and report top-1 accuracy without assum- ing ground-truth object proposals. Notably, we evaluate our methods on sensor point clouds obtained directly from lifting RGB-D observations, rather than the cleaned, post- processed point clouds sampled from mesh reconstructions provided by ScanNet. This choice better represents real- world deployment scenarios though it typically results in performance degradation due to sensor noise, missing re- gions, and registration errors, as discussed in (Jain et al., 2024). Section 4.1 compares Locate 3D to prior methods on standard benchmarks. Section 4.2 analyzes the impact of 3D-JEPA pre-training. Section 4.3 presents ablation studies various components of architecture, and Section 4.4 evaluates generalization capabilities novel environments and robotic deployment.
observations with modular pipeline composed of three stages. In stage 1, a VLM - either Llama-3 (Meta AI, 2024) or GPT-4o (OpenAI, 2024) - is used to select a single 2D frame from the observation stream. In stage 2, a VLM selects an object in the selected frame, by choosing from 2D object masks generated with GroundingDINO (Liu al., 2023) and SAM 2 (Ravi al., 2024). In stage 3, the 2D mask for the selected object is propagated in time using SAM 2 (Ravi et al., 2024), and lifted to 3D using depth and camera information, to generate predicted 31) bounding box. Further details of the Llama-3 and GPT-4o VLM baselines are provided in Appendix
We present the overall results in Table 1, Most prior work assumes access refined meshes and mesh (object) region proposals at training and inference time. We instead choose to evaluate our model under more realistic conditions where only the sensor observation stream is available. Despite eval- uating in this more stringent setting, our model (LOCATE 3D) achieves SoTA results, even when compared to prior work that operates under refined mesh point clouds. Further- more, when also trained with L3DD dataset (LoCATE 3D+), the model demonstrates even stronger performance, improving across all metrics while maintaining the same architecture and training methodology. We now discuss
4.1. How does LOCATE 3D compare to prior methods for 3D referential grounding benchmarks?
First, we study the performance of LOCATE 3D on three standard referential grounding benchmarks: SR3D, NR3D, and ScanRefer (Achlioptas et al., 2020; Chen et al., 2020). compare with prior work and two vision-language model (VLM) baselines. The VLM basclines process the RGB-D
6

---

LOCATE 3D: Real-Worid Object Localization via Self-Supervised Learning in 3D
Table 2: Ablation study of input features and encoder con- figurations. We compare different input modalities and encoder architectures. RGB refers raw RGB features, while Concept- Fusion (CF) leverages CLIP+DINO-v2 features. We evaluate accuracy (525 and @ 50 IoU) on the combined SR3D, NR3D, and ScanRefer evaluation sets. Results demonstrate that CF features consistently outperform RGB, and encoder initialization with 3D- JEPA yields the best performance. Note that in all experiments with an encoder, the encoder is fine-tuned with the decoder.
entryway door") and features of single 3D point to classify if the point satisfies the description not. On this task, 3D-JEPA outperforms ConceptFusion 39% to 34%. On a re- laxed "noun correctness" measure (e.g. is this point part of any "chair") 3D-JEPA outperforms ConceptFusion 73% to 66%. These results illustrate the power of 3D-JEPA features and their usefulness in many 3D vision domains.
Table 3: Impact 2D foundation features on LOCATE 3D. We evaluate accuracy (@ and @50 IoU) on the combined SR3D, NR3D, ScanRefer evaluation sets. LOCATE 3D is pre-trained and finetuned using different 2D foundation features. We find clear trend: larger models HAM-)) as well combining CLIP and DINO-v2 leads better results. (SAM-) indicates masks computed with said model. (1 indicates higher is better)
Input
Encoder
Initialization
Acc@25 f
Ace 50
RGB no
n/a
28.9
17.4
RGB yes
PonderV2
42.2
30.1
RGB
yes
random
51.4
38.7
CF
no
n/a
53.9
39.7
CF
yes
random
59.8
47.0
CF
yes
3D-JEPA (frozen)
56.2
44.0
CF
yes
3D-JEPA (ours)
61.7
49.4
2D Foundation Features
Acc@25 25 1
Acc@50 50 t
DINO-v2
53.7
39.4
different components that lead to these performance gains.
CLIP-B (MobileSAM)
53.7
42.1
CLIP-L (SAM-H)
59.2
45.5
4.2. Understanding the impact of 3D-JEPA
(III-H. + DINOV2 (ours)
61.7
49.4
Is 3D SSL necessary? Are lifted 2D foundation features sufficient? We conduct a systematic ablation study examin- ing three key aspects: (1) the choice of input features, (2) the role of utilizing an encoder architecture as opposed to simply training decoder on lifted features and (3) the ben- efits of initializing the encoder with 3D-JEPA pre-training. For each configuration, we train the same type of decoder. The overall results are presented in Table 2.
4.3. LOCATE 3D ablations
Benefits from advances in 2D foundation features. Due to the massive amount of data available on the internet, 2D foundation models have been consistently improving. Do those advances also improve results in 3D? If so. this provides a powerful opportunity to leverage progress and ad- vances there. To tease this apart, we trained variants of Lo- CATE 3D using different 2D foundation features. This goes through the full pipeline of lifting 2D features, pre-training the 3D backbone using said features, and end-to-end fine- tuning for 3D-RefExp using the localization decoder. The results are presented in Table 3. We find that using larger models (CLIP-L, SAM-H) improves results over smaller variants (CLIP-B, Mobile suggesting benefits from scaling. Additionally, using CLIP-- and DINO-v2 improves results considerably over using CLIP-L alone. Thus, we find promising evidence that improvements in 2D founda- tion models translate to improved 3D object localization.
We first examine the impact of input features, comparing raw RGB point clouds with lifted 2D foundation features. The results clearly demonstrate the importance of strong 2D foundation features, with CF showing a substantial im- provement over RGB (28.9% + 53.9% Acc@22 25). When in- corporating an encoder architecture, we observe interesting patterns. Even with random initialization, the encoder pro- vides gains for both input types, though more pronounced for RGB (28.9% + 51.4%) compared to CF features (53.9% + 59..8))...8...... Using a frozen 3D-JEPA encoder improves per- formance over the baseline CF features by 3% and 4% at Acc@25 and 5cc@50 respectively, indicating that 3D-JEPA learns strong representations suitable for localization. Fi- nally, we find that fine-tuning the 3D-JEPA encoder yields the best performance at 61.7%, underscoring the importance of SSL pre- tra-trainig and task-speeific finetuning.
What is the optimal decoder head architecture and su- pervision strategy? Given our joint prediction task, we in- vestigate two key design choices: (1) the type of supervision signal (mask-only, box-only, or both), and (2) the architec- ture of the bounding box prediction head. Our experiments in Table 5 show that mask-only supervision achieves mod- erate performance (55.4%) but lags behind our approach with a dedicated box head (61.7%). While post-processie these masks with DBSCAN helps address noisy predic- tions (58.4%), it still underperforms; particularly at higher IoUs 44..6% vs 49.4% at loU@ 50) while introducing non- differentiable components into the pipeline. On the other extreme, box-only supervision leads to extremely poor per-
Does 3D-JEPA learn contextualized representations? The key promise of SSL methods is to learn general purpose representations that are useful in many applications. 3D- JEPA accomplishes this by contextualizing, smoothing, and extracting the predictable features from the 2D foundational model inputs. We analyze the 3D-JEPA features through woint-wise probing experiments for localization. We train a probe that operates on a text description (e.g., "chair near
7

---

LOCATE 3D: Real-World Object Localization via Self-Supervised Learning in 3D
formance (0.3%), we hypothesize this is due the lack of the more dense mask supervision signal. Finally, comparing box head architectures, we find that our transformer-based design significantly outperforms using a MLP (61.7% vs 35.6%), demonstrating the importance of properly incorpo- rating spatial information through cross-attention.
Table 4: Generalization of LOCATE 3D. We report accuracy @ @25 IoU. Using lifted 2D foundation features consistently im- proves results compared to RGB features, and 3D-JEPA pretraining further bolsters generalization. "GPT-4o VLM-Baseline evaluated on 2,000 samples SN++ and 6,000 samples for SN Joint.
Method
Evaluation Dataset
What is the impact of scaling the decoder? We evaluate three different decoder sizes. As detailed in Appendix C.4, larger decoders consistently improve performance, partic- ularly at higher IoUs. For all decoder sizes, 3D-JEPA pre- training provides consistent 3-4% improvement in perfor- mance compared to randomly initializing the encoder.
ScanNet Joint Eval
LX3D
SN+
ARKitScenes
FRE
Baselines
GPT-4o VLM
37.6"
60.5
26,8
18.9
CF + 3D-Decoder
53.8
46.1
21.8
48.9
Ablations
How to fine-tune 3D-JEPA? We evaluate whether and how to fine-tune 3D in order to adapt it for the referential grounding task. We find that fine-tuning is necessary, as the frozen encoder underperforms (56.2% Acc@25), and dedicate stage-wise learning rate schedule achieves the best results (61.7%), surpassing a baseline single-stage scheduler (59.5%); more details in Appendix C.4.
RGB, Random, SN
51.3
37.5
11.3
39.9
CF, Random, SN
58.8
51.5
41.7
54.1
Ours
LOCATE 3D
61.7
56.7
46.2
52.0
Deployment on Robot. As outlined earlier, our model is capable of working with sensor streams and does not require human intervention test time (c.g., for mesh refinement or instance segmentation). We deployed our LOCATE 3D model on a Spot robot in a held-out apartment scene. This scene is dustif--istiibuttinn by heing a multí-room test apart- ment and is larger in size compared to the training data. The task was to navigate a furniture object and pick up a "plush toy." Success was verified by grasping the toy. Our re- sults show that LOCATE 3D achieved a success rate of 8/10 trials, outperforming baselines with maximum success rate of 5.66/10 (see details in Table 11). Note that naviga- tion and pick-up used pre-trained skills, while localization relied on our model. Additional details are provided the supplementary video and Appendix E.
4.4. Evaluating LOCATE 3D in novel environments
Performance on L3DD. We evaluate LOCATE 3D's ability to generalize by testing it on the unseen scene dataset splits of our L3DD dataset (Section 3), which spans three scene datasets. We also evaluate on 412 samples on our held- out environment (FRE) used for robot testing. Even when trained only on ScanNet, LOCATE 3D demonstrates strong performance on L3DD's ScanNet++ and ARKitScenes. This is despite significant domain gaps. The linguistic distribution is different due to a different annotator pool and task instructions. The scenes were captured with dif- ferent hardware by unique research efforts and span larger scenes and partial scans, both absent in ScanNet. And the object distribution is different, as ScanNet++ has denser open- vocabulary instance annotations. Notably, LOCATE 3D outperforms both baselines across most metrics, show- casing the robustness of our approach.
4.5. Computational Analysis
Run-time Analysis For computational efficiency, we cache the environment's 2D features for each view, as well as the featurized point cloud. For ScanNet experiments, we compute this cache offline; and for robot experiments compute it while doing the initial environment exploration phase. With this feature cache, forward pass of our model takes second for a scene with 100k feature points and utilizes 8 GB of VRAM on an A100 GPU.
Our ablation studies reveal the key components enabling this strong generalization. First, replacing raw RGB inputs with lifted foundation features (CF) significantly improves cross- dataset performance across all benchmarks (SN++: 37.5%
+ 51.5%, ARKitScenes: 11.3% - 41.7%, FRE: 39.9% + 54.1%). The introduction of 3D-JEPA initialization (LOCATE 3D) further enhances generalization capabilities, boosting performance on SN+ to 56. 56.7% and ARKitScenes to 46.2%. Finally, Table 7, we show that incorporating additional training data from L3DD (LOCATE 3D+) yields substantial improvements across all benchmarks (SN++: 56.7% + 83.9%, ARKitScenes: 46.2% 57.6%. FRE:
Limitations We can utilize such eaching because our bench- marks operate under static (ScanNet) or quasi-static (robot) environments. Extending our approach to dynamic scenes would require real-time 2D feature computation and con- tinuous updates to the featurized environment. We believe the former is a matter of engineering, while the latter is an active research area, explored by methods like Lifelong LERF (Rashid al., 2024)
52.0% + 73.5%). However, we observe that including in-
domain training data from L3DD reduces the impact of 3D-JEPA pre-training on L3DD evaluations.
8

---

LOCATE 3D: Real-World Object Localization via Self-Supervised Learning in 3D
5. Related Work
ferred to in the query (Luo al., 2022; Jain al., 2021). In the seaunding-b--eeecctii setting, pretrained vision backbone extracts region proposals (bounding boxes or seg- mentation masks), and models are trained to select the best proposal that matches the query (Chen et al., 2022; Zhu et al., 2023; 2024; Unal et al., 2024). These approaches have so far not been demonstrated on sensor point clouds or scaled up to multiple datasets, with the sole exception being ConcreteNet (Unal al., 2024), which trains a model to rank proposals from scratch. The primary limitation with this class of approaches is in the requirement of an external region proposal mechanism, which is often difficult in 3D, and are prone to failures.
Self Supervised Learning (SSL), and more broadly rep- resentation learning for 3D data like point clouds, is rel- atively under-explored compared 2D vision (Fei et al., 2023). Among the few explorations, most have focused learning representations for individual objects using object- level point clouds (Sauder and Sievers, 2019; Pang et al., 2022; Zhang et al., 2022). PointMAE (Pang et al.. 2022; Zhang et al., 2022) and PointBERT (Yu et al., 2022) use masked modeling techniques for object-level point clouds and show that the resulting representations can be effective for tasks like classification and segmentation. They differ crucially from our work on two fronts. First, our work is focused on scene-level point clouds which are significantly larger compared to single object point clouds used in prior work. Second, we operate on point clouds with high fea- ture dimensions, as opposed to RGB point clouds in prior work. In our experiments we found that a naive instantia- tion of masked modeling is ineffective due these factors. PointCLIP (Zhang et al., 2021a) leverages 2D foundation features for CLIP, but again looks only at object-level point clouds. Prior works have also explored contrastive learn- ing methods (Xie et al., 2020; Zhang et al., 2021b), which encourage representations to be invariant to different types of transformations. Other approaches have explored distill- ing 21) features directly to 3D, by differentiable rendering ( (Kobayashi et al., 2022); (Cao et al., 2025)) or constrastive learning (Peng et al., 2023). Ponder-v2 (Zhu et al., 2025) proposes an SSL pretext task based on differentiable render- ing into 2D. We refer readers to a recent survey on SSL for point clouds for additional references (Fei et al., 2023).
The second class of methods aim to leverage 2D founda- tion models (VLMs) for 3D tasks. While direct lifting of 2D features has shown promising performance lo- calizing objects from simple noun phrases (Jatavallabhula et al., 2023; Kobayashi et al., 2022; Wang et al., 2024), this does not translate to 3D referential grounding. Thefore, ap- proaches have either relied on using stronger models such as VLMS (Yang et al., 2023a; Xu et al., 2024), or by finetuning separate head for the task (Hong et al., 2023; Huang et al., 2024). While these methods inherit the strong generaliza- tion capabilities of foundation models, they rely tools for grounding and inherit their weaknesses. In the broader context, one could interpret LOCATE 3D as a powerful tool that such agents can leverage to perform better in the future.
6. Conclusion
In this work, we introduced LOCATE 3D, a model for lo- calizing objects in 3D from textual referring expressions. Our approach leverages 3D-JEPA, a novel self-supervised learning method for point clouds. It first projects features from 2D foundation models (like SAM, CLIP, DINO) into 3D point clouds. Subsequently, it performs SSL on top of these lifted features with a pretext task of masked predic- tion in latent space i.e. predict latent features of masked regions using the remainder of the scene. We showed that this enables the learning of contextualized representations i.e. features of particular point take into account the whole scene. We also showed that a backbone pre-trained with 3D- JEPA can be effectively finetuned for the task of RefFxp, using a mask-and-box decoder, that ultimately results in LOCATE 3D. We show that LOCATE 3D achieves state-of- the-art results in standard RefExp benchmarks. Furthermore, unlike prior work, LOCATE 3D only requires sensor point clouds, making it suitable for applications like robotics and smart glasses.
3D Referential Grounding: Recent years have seen re-
newed interest in tasks at the intersection of perception and language, such visual grounding (Zhang et al., 2023; Liu et al., 2023; Ren et al., 2024) and visual question answer- ing (Antol et al., 2015; Sermanet et al., 2023; Majumdar et al., 2024). Of particular relevance to this work are 3D referential grounding benchmarks, such as SR3D, NR3D, and ScanRefer (Achlioptas et al., 2020), which require lo- calizing objects mentioned in language utterance using observations of a 3D scene. We note that the original instan- tiations of these benchmarks provide access to ground-truth bounding boxes of all objects in the scenes as input (in- cluding at test time), and the task is to select the correct bounding box. In contrast, LOCATE 3D operates directly on sensor observation, without requiring any annotations. Few prior works operate under this setting.
Current approaches for 3D grounding fall under two classes of methods. First are end-to-end models designed explic- itly for this task. that operate in one of two modes. In the grounding- grounding-by-detectii -e-detectoon setting, models are trained to directly the bounding box corresponding to the object re-
9

---

LOCATE 3D: Real-World Object Localization via Self-Supervised Learning in 3D
Impact Statement
Springer- Verlag. ISBN 978-3-030-58451-1, doi: 10. 10077978-33-030-55555-& 13. https: htt///do..co orrg/ 10 20.1007//788 3- 000-000--500 58452- 1_....
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using ural language. 16th European Conference on Computer Vision (ECCV), 2020.
References
Ahmed Abdelreheem, Kyle Olszewski, Hsin- Ying Lee, Pe- ter Wonka, and Panos Achlioptas. ScanEnts3D: Exploit- ing phrase-to-3D-objeee correspondences for improved visio-linguistic models in 3D scenes. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 3512-3522, 2024.
Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Language conditioned spatial relation reasoning for 3d object grounding. In NeurIPS, 2022.
Bowen Cheng, Alexander G. Schwing, and Alexander Kir- illov. Per-pixel classification is not all you need for se- mantic segmentation. In NeurlPS, 2021.
Panos Achlioptas, Ahmed Abdelreheem, F. Xia, Mohamed Elhoseiny, and Leonidas J. Guibas. Referit3d: Neural listeners for fine-grained 3d object identification real- world scenes. In European Conference on Computer Vision, 2020. https: / / /pi. semanticscholar: org/CozpusID: :2:
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Hal- ber, Thomas Funkhouser, and Matthias Nießner. Scan- Net: Richly-annotater 3D Reconstructions of Indoor Scenes. In Proc. Computer Vision and Pattern Recogni- tion (CVPR), JEEE, 2017.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2425-2433, 2433, 2015.
Afshin Dehghan, Gilad Baruch, Zhuoyuan Chen, Yuri Feigin, Peter Fu, Thomas Gebauer, Daniel Kurz, Tal Dimry, Brandon Joffe, Arik Schwartz, and Elad Shul- man. Arkitscenes: A diverse real-world dataset for 3d indoor scene understanding using mobile rgb- d data. In NeurIPS Datasets and Benchmarks, 2021.
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bo- janowski, Pascal Vincent, Michael G. Rabbat, Yann Le- Cun, and Nicolas Ballas. Self-supervised learning from images with joint- eentedibeddingg predictive architecture. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15619-15629, 2023.
http://ap...emmtttcccccceooeoeee .rg/ CorpusID:237271377
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. In North Ameri- can Chapter of the Association for Computational Lin- guistics, 2019. https: / //api/apppap// wewartiissmatiiiiieieeeeeeii icscollar. Sxg//oorpuaID:52967335
Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael G. Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video. ArXiv, abs/2404.084711, 2024.
Nico Engel, Vasileios Belagiannis, and Klaus C. J. Di- etmayer. Point transformer. IEEE Access, 9:134826- 134840, 2020. https: //apii http://aapp............/. sew.nti iccholar...... azg/CorrpaaD::222270000
Ang Cao, Sergio Arnaud, Oleksandr Maksymets, Jian- ing Yang, Ayush Jain, Sriram Yenamandra, Ada Mar- tin, Vincent-Pierre Berges, Paul McVay, Ruslan Part- sey, Aravind Rajeswaran, Franziska Meier, Justin John- son, Jeong Joon Park, and Alexander Sax. Lift-gs: Cross-scene render-supervised distillation for 3d lan- guage grounding, 2025. https://arxxt 2ass/2502.003
Ben Fei, Weidong Yang, Liwen Liu, Tian jian Luo, Rui Zhang, Yixuan Li, and Ying He. Self-supervised learning for pre-training 3d point clouds: A survey. ArXiv, 2023.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEF/VV Conference on Computer Vision and Pattern Recognition (CVPR), pages 16000-16009, June 2022.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nico- las Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Com- puter Vision - ECCV 2020: 16th European Confer- ence, Glasgow, UK, August 23-28, 2020, Proceed- ings, Part I, page 213-229, Berlin, Heidelberg, 2020.
Yining Hong. Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3D-LLM: Injecting the 3D world into large language models. In
10

---

LOCATE 3D: Real- Object Localization via Self-Supervised Learning in 3D
Advances in Neural Information Processing Systems (NeurlPS), pages 18995-19012, 2023.
Shunya Kato, Shuhei Kurita, Chenhui Chu, and Sadao Kurohashi. ARKitSceneRefer: Text-based localiza- tion of small objects in diverse real-world 3D in- door scenes. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 784-799, Singapore, December 2023. Association for Computational Linguisties. doi: 10.18653/v1/ 2023. www..iiingseemnlp..... https:/ htttps://aclattthttt http///aaaaathtogg/oo/ org/2023. findings emnlp. 56/.
Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang. Qing Li, Song-Chun Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. In Proceedings of the International Conference on Machine Learning (ICML), 2024.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson. Aver- aging weights leads to wider optima and better gener- alization. In Amir Globerson and Ricardo Silva, edi- tors, Proceedings of the Thirty-Fourth Conference on Un- certainty in Artificial Intelligncce, UAI 2018, Monterey, California, USA, August 6-10, 2018, pages 876-885, AUAI Press, 2018. http://aual org/ www//orig/00 /uag//8// proceedi ngs wwwp/apppesssepp. / 31.3/113..pd..
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan- Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4015-4026, 2023.
Sosuke Kobayashi, Eichi Matsumoto, and Vincent Sitz- mann. Decomposing nerf for editing via feature field dis- tillation. In Advances in Neural Information Processing Systems, volume 35, 2022. pdf/2205. 15585,55855 pdf.
Ayush Jain, Nikolaos Gkanatsios, Ishita Mediratta, and Katerina Fragkiadaki. Bottom up top down de- tection transformers for language grounding in im- ages and point clouds. ArXiv, abs/2112.08879, 2021. bttpss://apl. http://///app.ssaattsscc.... wwwanticschoaaar.....o org/ CorpusID:250921811
Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pre- trained features and underperform out-of-distribution, 2022. https: /tatps://arxvvv /ttp///avvvvvvvv///v///////a.avv wwwp//ii.orr//oooooo abs 2202. 10054.
Ayush Jain, Pushkal Katara, Nikolaos Gkanatsios, Adam W. Harley, Gabriel Sarch, Kriti Aggar- wal, Vishrav Chaudhary, and Katerina Fragkiadaki. Odin: A single model for 2d and 3d segmenta- tion. 2024 IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 3564-3574, 2024. https: / /api. semanticscholar org/ CorpusID:26675500
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding DINO: Marrying DINO with Grounded Pre-training for Open-Set Object Detection. arXiv preprint wrXiv:2303.05499 2023.
Ayush Jain, Alexander Swerdlow, Yuzhou Wang, Ser- gio Arnaud, Ada Martin, Alexander Sax, Franziska Meier, and Katerina Fragkiadaki. Unifying 2d and 3d vision- languae understanding. 2025. https:// arxiv. or.orrrgg/ooo.. waas//5500 10745.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. https: https///axxvv..ooo abs//
05101.
Jun-Bin Luo, Jiahui Fu, Xianghao Kong, Chen Gao, Haibing Ren, Hao Shen, Huaxia Xia, and Si Liu. 3d-sps: Single- stage 3d visual grounding via referred point progressive selection. 2022 IEEE/CVF Conference Vision and Pattern Recognition (CVPR), pages 16433- 16442, 2022. https: / /ppi. htgg/CoopuuII::24155333
Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Shuang Li, Ganesh Iyer, Soroush Saryazdi, Nikhil Keetha, Ayush Tewari, Joshua B. Tenenbaum, Celso Miguel de Melo, Madhava Krishna, Liam Paull, Florian Shkurti, and Antonio Tor- ralba. Conceptfusion: Open-set multimodal 3d mapping. Robotics: Science and Systems (RSS), 2023.
Computer
Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Sil- wal, Paul Mevay, Oleksandr Maksymets, Sergio Ar- naud, Karmesh Yadav, Qiyang Li, Ben Newman, Mo- hit Sharma, Vincent- Pierre Berges, Shiqi Zhang, Pulkit Agrawal, Yonatan Bisk, Dhruv Batra, Mrinal Kalakrish- nan, Franziska Meier, Chris Paxton, Alexander Sax, and Aravind Rajeswaran. Openeqa: Embodied question an- swering in the era of foundation models. In CVPR, 2024.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Ishan Misra, Gabriel Synnaeve, and Nicolas Carion. Mdetr modulated detection for end-to-end multi-modal un- derstanding. 2021 IEEE/CVF International Confer- ence on Computer Vision (ICCV), pages 1760-1770, 2021. https://aat /apii http://app....aaiiie..... CorpusID:23339396:

---

LOCATE 3D: Real-World Object Localization via Self-Supervised Learning in 3D
Meta AI. Llama 3: The llaaa 3 herd of models. https:
Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
//ai foi..mm mwtaa.ccmm.. cww/coama...//oc.ccccmcccaom.. 2024. [Large language model].
Adam Rashid, Chung Min Kim, Justin Kerr, Letian Fu, Kush Hari, Ayah Ahmad, Kaiyuan Chen, Huang Huang, Marcus Gualtieri, Michael Wang, Christian Juette, Nan Tian, Liu Ren, and Ken Goldberg. Lifelong lerf: Local 3d semantic inventory monitoring using fogros2, 2024. mttps://arxiinn http:///wwwwwww.//////////////// /203. 10494.
Tomas Mikolov, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. In International Conference on Learning Representations, 2013.
Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Ar- avindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, Xiao Wang. Xiaohua Zhai, Thomas Kipf, and Neil Houlsby. Simple open-vocabulary object detec- tion with vision transformers, 2022. https:/ arxiv. org/abs/ 2205. 06230.
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Rong- hang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. SAM 2: Segment Anything in Images and Videos. arXiv preprint arXiv:24065 00714, 2024. https: http///axivvv/aa wwwexiiv.orgaaaao
OpenAl. GPT-4o System Card. arXiv:2410.21276, 2024.
arXiv preprint
org/ abs 2408 240800071..
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Q. Vo. Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojeiech Galuba, Russ Howes, Po-Yao (Bernie) Huang, Shang- Wen Li, Ishan Misra, Michael G. Rabbat, Vasu Sharma, Gabriel Synnaeve, Huijiao Xu, Hervé Jégou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bo- janowski. Dinov2: Learning robust visual features with- out supervision. ArXiv, 2023.
Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang. Hongyang Li, Qing Jiang, and Lei Zhang. Grounded sam: Assembling open-world models for diverse visual tasks, 2024.
Junha Roh, Karthik Desingh, Ali Farhadi, and Dieter Fox. Languagerefer: Spatial-language model for 3d visual grounding. In Aleksandra Faust, David Hsu, and Gerhard Neumann, editors, Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pages 1046-1056. PMLR, 08-11 Nov 2022. https: / http///ooeeeeeeee #164/roh24 html.
Yatian Pang, Wenxiao Wang, Francis E. H. Tay, Wei Liu, Yonghong Tian, and Li Yuan. Masked autoencoders for point cloud self-superviser learning. In Computer Vision -ECCV 2022, pages 604-621. Springer International Pub- lishing, 2022.
ml Srll.. press/
Jonathan Sauder and Bjarne Sievers. Self-supervised deep learning on point clouds by reconstructing space. In Neural Information Processing Systems, 2019.
Songyou Peng, Kyle Genova, Chiyu "Max" Jiang, Andrea Tagliasacchi, Mare Pollefeys, and Thomas Funkhouser. OpenScene: 3D scene understanding with open vocabu- laries. In CVPR, 2023.
Pierre Sermanet, Tianli Ding, Jeffrey Zhao Fei Xia, Debidatta Dwibedi, Keerthana Gopalakrishnan, Chris- tine Chan, Gabriel Dulac-Arold, Sharath Maddineni, Nikhil J Joshi, et al. Robovqa: Multimodal long-horizon reasoning for robotics. arXiv preprint arXiv:2311.0089) 2023.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gard- ner, Christopher Clark, Kenton Lee, and Luke Zettle- moyer. Deep contextualized word representations. ArXiv, 2018.
C. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. Point- net: Deep learning on point sets for 3d classification and segmentation. 2017 IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 77-85, 2016. https:/ / https////ap..smmaitiessssseee semanticscholel....... ott//CorpusD::5115938
Ozan Unal, Christos Sakaridis, Suman Saha, and Lue Van Gool. Four ways to improve verbo-visual fusion for dense 3d visual grounding. In European Conference on Computer Vision (ECCV), October 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. https: www.ai..woggabs///../eaa...aaeee 03762.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh. Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen
12

---

LOCATE 3D: Real-World Object Localization via Self-Supervised Learning in 3D
Peihao Wang. Zhiwen Fan, Zhangyang Wang, Hao Su, Ravi Ramamoorthi, al. Lift3d: Zero-shot lifting of any 2d vision model to 3d. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21367-21377, 2024.
Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-BERT: Pre-training 3D point cloud transformers with masked point modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19313- 19322, June 2022.
Xiaoyang Wu, Li Jiang, Peng- Peng-Shuai Wang, Zhijian Liu, Xihui Liu, Yu Qiao, Wanli Ouyang, Tong He, and Heng- shuang Zhao. Point transformer v3: Simpler, faster, stronger. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4840- 4851, 2023. https:/ /api org/CorpusID: ID:6333582:26
Zhihao Yuan, Xu Yan, Yinghong Liao, Ruimao Zhang, Zhen Li, and Shuguang Cui. Instancerefer: Co- operative holistic understanding for visual grounding on point clouds through instance multi-level contex- tual referring. 2021 IEEF/CVF International Confer- ence on Computer Vision (ICCV), pages 1771-1780, 2021. https://api.. wwtppp.ssmatttssschseeeeeeeeeeee CorpusTD:2320922
.aappis.eeaane semant cscholar.
Saining Xie, Jiatao Gu, Demi Guo, C. Qi, Leonidas J. Guibas, and Or Litany. Pointcontrast: Unsuper- vised pre-training for 3d point cloud understand- ing. ArXiv, 2bs/2007.10885 2020. https: //api, semanticscholoo ar. org Corg/CorrpsT: 220666215.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabás Póczos, Ruslan Salakhutdinov, and Alex Smola. Deep sets. In NeurlPS, 2017. https://aat
semanticscholar.co
Runsen Xu, Zhiwei Huang. Tai Wang, Yilun Chen, Jiang- miao Pang, and Dahua Lin. Vlm-grounder: A vlm agent for zero-shot 3d visual grounding. In CoRL, 2024.
Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xu- peng Miao, Bin Cui, Yu Jiao Qiao, Peng Gao, and Hongsheng Li. Pointclip: Point cloud understanding by clip. 2022 IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 8542-8552, 2021a. https: /api. weeaatiicccoollaa...e.. org/
Jianing Yang, Xuweiyi Chen, Shengyi Qian, Nikhil Madaan, Madhavan Iyengar, David F. Fouhey, and Joyce Chai. Lim- -on-grounder: Open-vocabulary 3d visual grounding with large language model an agent. 2024 IEEE Inter- national Conference on Robotics and Automation (ICRA), 2023a.
Renrui Zhang, Ziyu Guo, Peng Gao, Rongyao Fang, Bin Zhao, Dong Wang. Yu Qiao, and Hongsheng Li. Point- M2AE: Multi-scale masked autoencoders for hierarchical point cloud pre-training. In Advances in Neural Informa- tion Processing Systems, volume 35, pages 27061-27074 2022.
Jianwei Yang, Hao Zhang, Feng Li. Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, arXiv preprint 1:Xiv:2310.1144:: 2023b.
Youcai Zhang, Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xie, Yuzhuo Qin, Tong Luo, Yaqian Li, Shilong Liu, et al. Recognize any- thing: A strong image tagging model. arXiv preprint arXiv:2306.03514 2023.
Zhengyuan Yang, Songyang Zhang, Liwei Wang, and Jiebo Luo. Sat: 2d semanties assisted training for 3d visual grounding. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1836-1846, 2021. doi: 10.1 00110099CCC4499222210000000
Zaiwei Zhang. Rohit Girdhar, Armand Joulin, and Ishan Misra. Self-supervised pretraining of 3d features on any point-cloud. 2021 IEEE/CVF International Con- ference Computer Vision (ICCV), pages 10232-10243, 2021b. https: /atas://apt http://ppp..aaaaeaee www.ssisschhollla........ org/ Corpus 23:7:009
Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, and Angela Dai. ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes. In Proceedings of the International Conference on Computer Vision (ICCV), 2023.
Naoki Yokoyama, Alexander Clegg, Joanne Truong, Eric Undersander, Tsung- Yen Yang, Sergio Araud, Sehoon Ha, Dhruv Batra, and Akshara Rai. ASC: adaptive skill coordination for robotic mobile manipulation. IEEE Robotics Autom. Lett., 91):7799-78, 2024. doi: 10. (009/LRA.202333336111 https: htttps://doi.rre 1109 /LR.. 2023.332622 3396109.
Haoyi Zhu, Honghui Yang. Xiaoyang Wu, Di Huang, Sha Zhang, Xianglong He, Tong He, Hengshuang Zhao, Chun- hua Shen, Yu Qiao, and Wanli Ouyang. PonderV2: Pave the way for 3D foundation model with a universal pre- training paradigm. IEEE Transactions on Pattern Analy- sis and Machine Intelligence, 2025.
13

---

LOCATE 3D: Real-World Object Localization via Self-Supervised Learning in 3D
Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 3d-Vi::: Pre-trained transformer for 3d vision and text alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Ziyu Zhu, Zhuofan Zhang, Xiaojian Ma, Xuesong Niu, Yixin Chen, Baoxiong Jia, Zhidong Deng, Siyuan Huang, and Qing Li. Unifying 3d lision-language understand- ing via promptable queries. In European Conference on Computer Vision (ECCV), 2024.
Özgün Çişek, Ahmed Abdulkadir, Soeren S. Lienkamp, Thomas Brox, and Olaf Ronneberger. 3d u-net: Learning dense volumetric segmentation from sparse annotation. In International Conference on Medical Image Computing and Computer-Assiset Intervention, 2016.
14

---

LOCATE 3D: Real-World Object Localization via Sel--Supervised Learning in 3D
A. Model Details
A.I. Pre-training
We experiment with several masking strategies to achieve the best pre-training representation. Two strategies for computing masks were percent-based and fixed number of points. In pereent-based masking, we mask percentage of the scene. E.g. mask 8% of the scene. This means that depending the size of the pointcloud that range from 1k points 200k points, the number of points in the masks varies widely. In fixed number of points masks, keep the number of points in every mask constant and vary the number of masks based on the size of the pointcloud to ensure cover similar portion of the scene. We found percent-basee masking to perform best.
Additionally, we experiment with two methods for constructing masks: radius-based and serialized-curve based. In radius- based masking, for a mask size æ, we randomly select a point in the scene and select the æ closest points based on masking, first order the points based using the serialization methods. We then
Cartesian coordinates. In serialized-cum mask contiguous blocks in this serialized sequence. We find that using serialized-cung based masking works best as the mask regions have more diversity than the circular aadius-base' masking.
We evaluate the masking strategies using frozen representation element- wise probe. We show the performance of the element-wise probe on the referring expression benchmark throughout the pre-training in Figure 5.
Predictinn
Encoding
Percont rulue maziing
soseee smepp ansoo ress A
Figure 4: Overview of different masking types
A.2. Encoder architecture
The main backbone of the encoder architecture is the PoineTransformerV3 archicteture. For input to the transformer, have CLIP, DINO and RGB features. For RGB, we harmonically embed the RGB features before using small MLP to process the features. Given that we run CLIP on each of the SAM masks for an image, some pixels are not contained in any masks and have no CLIP features. We use a learnable parameter to represent these points without CLIP features. Finally, we concatenate the MLP processed harmonically embed RGB, learnable parameter masked CLIP features, and the DINO features together. We then tokenize these points with sparse sub-manifold stem convolution before using the
15

---

LOCATE 3D: Real-World Object Localization via Self-Supervised Learning in 3D
benchmark_eaa Beachmarr__eO/ioousss
1e-08
group: space maskkish sae.-kiissssn-akttt-ERRe
group: serrrerttmsssk--saaarttt--------
group: sptttily.oossssnnnmnkkksss-eeee-
0.32
0.31
0.3
0.29
0.28
Step
0.27
0
100
200
300
400
Figure 5: Element-wise probe results during pre-training with different masking strategies
PoinfTransformerV3 architecture.
A.3. Predictor Architecture
The predictor architecture takes the encoded context along with a learnable parameter mask tokens to represent the points to predict as well as global registers. These inputs are process in transformer with a block sparse attention pattern and rotary positional embedding. The rotary positional embeddings use the continuous Cartesian coordinates of the points to rotate sub-section of the vector for each x, y, and z directions. The sparse attention mask is constructed the mask tokens can all attend all of the encoded context, the encoded context can diagonally self-attend. An example sparse attention mask is shown in Figure 6.
Registers
Encoded Context
Prediction Mask Tokeens
Ragerees
1
1
1
1
1
1
1
1
1
0
0
Encodded Cortaot
1
1
1
1
0
1
1
1
0
0
1
1
1
1
1
0
0
0
O
1
1
1
1
0
0
0
1
1
Prodiction Mask Tokains
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
Figure 6: Left: Block sparse attention pattern
B. Object Localization
In this section, we discuss details regarding our Conditioned 3D Decoder and the end-to-end training for referential grounding.
16

---

LOCATE 3D: Real-World Object Localization via Self-Supervised Learning in 3D
B.1. Decoder architecture
In Figure 7, we provide detailed illustrations for our token prediction head (top left), mask prediction head (top right), and bounding box prediction head (bottom).
queryc
Decoder Query Features
MLP
Per query: [ popllss 2. 2.4p, (Pldeeeee forlin points)
Query- Query-Tokere Token Match Predictions
Bounding Box Coordinates
Que.
Q...
Decoder
C
MLP
Dot
Query
Features
Decoder
Point Features
Token Head
Mask Head
Decoder
Linear
Per queryc ye xe yo 2i 2. Vy, 31,22
Query
Features
Bounding Box Coordinates
MultiHead Attention
Decoder
Point Features
MLP
Linear
Stack
XYZ
Bounding Box Head
Figure 7: Prediction heads of the language-condittoned decoder (Top Left) Our Token Prediciton Head (Top Right) Our
Mask Head (Bottom) Our bounding Head
B.2. Decoder Ablations
We present the results of additional experiments on decoder supervision objectives and prediction head architectures in Table and decoder size Table
C. LOCATE 3D training
C.1. Details of training
LOCATE 3D is optimized using AdamW (Loshchilov and Hutter, 2019) with parameters 0.9, 82 0.999, weight decay of 0.01 and using learning rate scheduler as described in Appendix C.2. We optimize the following loss function:
Litice XceLoe + Nova Lrox + A Eigon + Aatiga falign
17

---

LOCATE 3D: Real-World Object Localization via Self-Supervised Learning in 3D
Table 5: Ablation study on decoder supervision and bounding box prediction head architectures. We evaluate accuracy 25 and @50 IoU) on the combined SR3D, NR3D, and ScanRefer evaluation sets (Joint ScanNet). We observe that without box supervision, DBSCAN can be used remove outliers and improve accuracy. However, using box supervision with our transformer architecture leads the best performance and removes the reliance on additional post-proeet pros-pscesssing (with DBSCAN).
Supervision
Architecture
Joint ScanNet
Mask
Box
Box Head
Acc@25
Acc@50
no
yes
Transformer
0.3
0
no
yes
MLP
29.4
10.0
yes
Naive
55.4
39.1
yes
DBSCAN
58.4
41.6
yes
yes
MLP
35.6
13.5
yes yes
Transformer (ours)
61.7
49.4
Table 6: Impact of decoder size performance. We evaluate accuracy (@25 and @50 IoU) the Joint ScanNet benchmark across different decoder sizes (Small, Base, and Large). Larger decoders consistently improve accuracy, while 3D-JEPA pre-training provides a stable performance boost across all scales.
Initialization
Decoder Size
Acc@25 t
Acc@
Random
Small
54.4
39.0
Random
Base
57.6
44.5
Random
Large
58.8
46.3
3D-JEPA
Small
58.1
43.3
3D-JEPA
Base
60.6
47.6
3D-JEPA
Large
61.7
49.4
With:
As
4.0
(Class weight)
Amask 6.0
(Mask cross entropy weight)
Adice 4.0
(Mask dice weight)
Abox 1.0
(Bounding box LI weight)
Agio 1.0
(Bounding box GloU weight)
C.2. Training Schedule Details
Fine-tuning pre-trained encoder alongside randomly initialized decoder presents significant challenges. The primary issue stems from the decoder's ' initially poor gradients potentially destabilizing the valuable pre-trained representations the encoder. To address this, we designed training schedule that carefully balances the learning dynamics of both components while preserving the pre-trained features.
Our approach (Figure 8 follows progressive training strategy where components introduced gradually into the optimization process. The schedule consists of four primary phases: initial decoder training with frozen encoder, transition period introducing encoder fine-tuning a reduced rate, an encoder adaptation phase, and finally joint optimization with component-spedeeeeeee learning rates. Throughout training, we maintain lower learning rates for the encoder (0.5x the decoder rate) and implement smooth transitions between phases to ensure stability.
The complete schedule spans epochs, with the first 17 epochs dedicated the careful initialization and adaptation phases.
18

---

LOCATE 3D: Real-World Object Localization via Self-Supervised Learning in 3D
Inenooder
iddecoder
454
505
5--3
341
4.5
12xc 265
opedh
Figure 8: Learning rate schedule for encoder and decoder. Fine-tuning a pre-trained encoder alongside a randomly initialized decoder requires careful balancing to prevent unstable gradients from disrupting pre-trained representations. Our progressive training schedule spans 40 epochs, with an initial phase freezing the encoder, followed by gradual adaptation and joint optimization. The encoder (left) maintains lower learning rate (0.5x the decoder rate) with smooth transitions, ensuring stable fine-tuning and improved performance.
C.3. Finding the Right Sensor Supervision for LOCATE 3D
A key challenge in training LOCATE 3D is the mismatch between our input data (raw sensor information) and the available annotations (clean mesh pointclouds) in ScanNet. While ScanNet provides RGBD+pose trajectories, allowing us to reconstruct uncleaned 3D pointclouds via SLAM that better match real-world conditions, the question remains: how do we effectively supervise a model operating on noisy sensor data using annotations from clean meshes?
We explored two approaches for bridging this gap:
Mesh-space supervision: Transfer model predictions from sensor pointcloud to mesh pointcloud by assigning each mesh point the distance- distance-weighted average predictions from its k-nearest neighbors (k=8) in the sensor pointcloud. Supervision occurs in mesh space.
Sensor-space supervision: Transfer mesh annotations sensor pointcloud by assigning each sensed point the distance- weighted average of ground truth labels from its k-nearest neighbors (I: 8) in the mesh. Supervision occurs directly in sensor space.
Approach (2) proved superior, delivering both practical benefits in terms of efficiency (single transfer operation vs. one per decoder layer) and significantly better performance. The key insight came from analyzing failure cases: in approach (1), about 50% of points were effectively ignored during training because they had no close neighbors in the clean mesh. These ignored points were precisely the sensor artifacts and noise that the model needs to learn to handle in real-world deployments. By supervising sensor space, we force the model to process all input points, including challenging cases like sensor noise and artifacts.
C.4. Training ablations
How to fine-tune 3D-JEPA? Given pre-trained 3D-JEPA encoder and randomly initialized decoder, we investigate different strategies for jointly fine-tuning these components. Specifically, compare four configurations: (1) a randomly initialized encoder trained end-to-end with naive learning rate schedule (warmup cosine decay with equal learning rates for both components), (2) frozen pre-trained 3D-JEPA encoder, (3) pre-trained 3D-JEPA encoder fine-tuned with the naive schedule, and (4) pre-trained 3D-JEPA encoder trained with our stage-wise training Appendix C.2.
The experiment results are detailed in Table 8, the frozen variant performs moderately (56.2% Acc 25) but lags behind the random initialization where the model is fine-tuned end-to-end (59.8%); naive fine-tuning of the 3D-JEPA initialized encoder marginally improves results (59.5%), and our stage-wiss wise schedule achieves the best result 61.7%.
What is the impact of scaling the decoder? We investigate the effect of decoder scaling by evaluating three model sizes (Small, Base, and Large) with both random initialization and 3D-JEPA pre-training. As shown in Table 6, increasing decoder size consistently improves performance, with each scaling step (Small->) Balll-BBae---) -Lagge) yielding approximately
19

---

LOCATE 3D: Real-World Object Localization via Self-Supervised Learning in 3D
Table 7: Impact of LX3D train data. We report accuracy @25 JoU. ARKitScenes column contains both pretrain and val split as we significant difference when split up. Adding LX3D training data significantly improves performance unseen LX3D scenes. 3D-JEPA's impact LX3D evaluations is reduced when adding LX3D train data. shows standard enror.
Training Data
Evaluation Dataset
Existing SN Benchmarks ScanRefer
LX3D
NR3D
SN+
ARKitScenes
FRE
ScanNet, JEPA (LOCATE 3D)
59.9 = 0.58
53.7 0.51
56.7 0.74
46.2 1.32
52.0 2.46
ScanNet, Random Init
56.7 = 0.58
5.75 5.56
41.7 1.31
+ 54.1
ScanNet + LX3D, JEPA (LOCATE 3D+)
61.1 + 0.57
56.1
83.9 0.55
57.6 1.31
73.5 + 2.17
ScanNet + LX3D, Random Init
61.0 # 0.58
0.51
83.5 = 0.56
59.3 1.31
73.8 + 2.17
1-3% absolute improvement in Acc@25 and 2-5% in Acc@50 on our joint ScanNet evaluation. The larger gains at higher IoU thresholds suggest that increased model capacity particularly benefits precise object localization. Notably, the benefit of 3D-JEPA pre-training remains relatively stable across all scales, providing an ~3344% improvement over random initialization for each decoder size. These results suggest that model capacity and initialization quality contribute independently performance, with both larger decoders and better pre-training being beneficial for the 3D localization task.
C.5. Visualizing LOCATE 3D
Figure visualizes the transformation of point cloud features as they pass through the model's encoder and decoder. The left column shows the original RGB scenes, while the middle and right columns present PCA-reduced embeddings of the 3D point features extracted from the fine-tuned 3D-JEPA encoder and the decoder's refined output, respectively.
The encoder processes the lifted point cloud representation, capturing global scene context and producing initial feature embeddings. The decoder then iteratively refines these representations through series of self-attention and cross-attention operations, incorporating language-conditioned object queries. As seen in the PCA visualization, the encoder seems to learn more general semantic features that look like a zero-shot segmentation; the decoder enhances the feature distinctiveness around the referenced objects, leading to sharper and more localized embeddings.
Table 8: Impact of learning rate schedule model performance. We evaluate accuracy (@25 and @50 IoU) the combined SR3D, NR3D, and ScanRefer evaluation sets. The stage-wise schedule prevents catastrophic forgetting of pre-trained features to enable effective fine-tuning.
Encoder Initialization
LR Schedule
Acc@25
Acc@50
Random
Naive
58.8
46.3
Random
Stage-wise
59.8
47.0
3D-JEPA
Frozen
56.2
43.7
3D-JEPA
Naive
59.5
47.0
3D-JEPA
Stage-wise (ours)
61.7
49.4
D. Annotation Details
D.1. Collection Procedure
We collect data using an off-the-shelf annotation interface which allows for selecting rectangular regions 2D videos and associating them with text. We project 3D instance masks into 2D and show them alongside the RGB camera view. For ScanNet (Dai et al., 2017) and ScanNet++ (Yeshwanth et al., 2023), we use ground truth instance masks provided by the dataset. For ARKitScenes (Dehghan et al., 2021) and our robot trial environment, we use instance masks produced by SAMPro3D (?). Annotators have the ability to conglomerate multiple instance masks into a single object - this proved
20

---

LOCATE 3D: Real-World Object Localization via Self-Supervised Learning in 3D
RGB
Encoder
Decoder
"Locate the box on top of the bookshelf"
"Locate the coffee table in front of the L-shaped couch"
"Locate the far right painting above the bed"
Figure 9: Visualization of point cloud features before and after decoding. We compute the PCA of the fine-tuned 3D-JEPA encoder's ' point features (middle column) and the refined point features after decoding (right column). The RGB images (left column) provide reference scenes, while the PCA projections reveal that the encoder seems to learn smooth semantic features while the decoder learns sharper localized features.
21

---

LOCATE 3D: Real-World Object Localization via Self-Supervised Learning in 3D
necessary for SAMPro3D segmentations, which tended break large objects into many subparts. This is in line with ScanNet's instance segmentation procedure in which annotators stitch together automatiicall--genraatee oversegmentation of 3D scene (Dai al., 2017).
In addition to grounding the target object, annotators were asked to provide object grounding for all other nouns in the phrase they provided. For SAMPro3D masks, annotators could select any object they wished as a target. In later iterations of the task, for ScanNet and ScanNet++,, we highlighted a mask in white with at least distractor to serve as the target object for description. This is in line with prior literature which observed that the 3D RefExp task is most challenging the high-distractor case (Chen al., 2020). We later rely on an LLM (Llama 3.2-90B) to identify which object is the target.
While we found that annotations ScanNet and ScanNet++ were relatively reliable from first pass, we observed considerable mask/bounding box noise when collecting annotations on SAMPro3D instance masks. To address this, we created a separate validation task. In this task, annotators saw video similar to the video provided in the generation task, but only the selected masks for a particular language sample were highlighted. The RefExp description was also shown. This improved the quality of SAMPro3D samples substantially, but it also drastically slowed the collection rate. As such, we do not use ARKitScenes for supervised training.
Additionally, we had the validation task set up. had annotators validate every eval sample at least once (with over 80 percent of ScanNet++ and ARKitScenes samples receiving 3+ validation labels). Annotators were instructed to only keep accept samples which were unambiguously correct, and we only samples which a majority of annotators marked as trustworthy.
D.2. Annotation setup
Figure 10: An example sample from the RefExp task. This sample is taken from the ScanNet++ (Yeshwanth et al., 2023) split of L3DD. The model receives the camera input (RGB shown on left) with additional depth and pose information and must produce the object masks (shown on the right). The text conditioning "indoor plant on the table."
Our annotation setup largely mirrors ScanKefer (Chen et al., 2020) annotators simply describe objects in one-step fashion rather than the two-player game format of NR3D (Achlioptas et al., 2020). We use 2D video interface for the task, projecting 3D instance masks to 2D. Annotators can write full referring expressions and associate particular instance masks with particular tokens in the expressions. They may also agglomerate two or more instances into one.
When ground-truth instance masks are available (as ScanNet and ScanNet++), we use these as the basis for annotation. When they are not available (as ARKitScenes and the demo FRE apartment scene), we generate instance masks using SAMPro3D (?).
We implemented two versions of the task. Initially, aimed to simplify the task as much possible so that high quantity of annotations on SAMPro3D masks would be feasible. As it was challenging achieve both speed and quality with this approach, we switched to a more challenging task to annotate existing instance masks.
Version 1: Annotators select only two objects and may choose any object the scene as a target.
Version 2: Annotators select an arbitrary number of objects. When comprehensive semantic instance annotations are available (as ScanNe(/ScanNettt)) we highlight an object with at least distractor as a prescribed target.
22

---

LOCATE 3D: Real-World Object Localization via Self-Supervised L Learning in 3D
Frquueicy Ty Nouns
Prosencc To 20 Spatial feletions
0.8
0383
0027
0001
Figure 11: Histogram of most common nouns and spatial relations in L3DD. Computed over 5000 samples using Llama
3.1-8B. Left: Top nouns, Right: Top spatial relations.
D.3. Comparison with existing datasets
We show in ablations that mask supervision is crucial for training. Voxel coverage is the number of 5em voxels occupied by at least one point in the sensor pointclouds of these datasets. When multiple scans available of a single venue, only use the voxel occupancy of the first scan so as to ensure this acts measure of size-adjusted venue coverage. Finally, we compute object coverage by enumerating the number of unique object instance masks covered by each dataset. For the case of the ARKitScenes split of our data, we divide the number of SAMPro3D segments by the average number of segments per object instance (2.25). Note that this object count includes both grounded targets and grounded anchors.
We showed in Table 5 that mask supervision is crucial for high-quality training data, so ARKitSceneRefer's commendable method. Of the other three datasets with grounded anchors and mask coverage,
object diversity is not easily applicable there is significant overlap in instance coverage; the three together cover 29,857 instances. All of L3DD's 14,662 new instances in ScanNet++ have not yet been covered by RefExp dataset.
Additional dataset statistics ("starred values were computed using Llama 3.1-8B instr AI, 2024) over 5000 samples)
Average query length (in words): 7.55
Average number of grounded instances per query: 2.28
Vocabulary size: 3058
Percent of samples containing a reference to color: 42.6"
Percent of samples containing a reference object shape: 38.22*
Table 9: A comparison of LOCATE 3D DATASET with other 3D RefExp datasets. L3DD advances the state-of-the-art by covering multiple scene datasets and a large number of unique venues with mask-grounded 3D referring expressions.
Dataset
Scene Datasets
Objects
Expressions
Venues
Vuxels Covered
Human-generatee
Musks Grounded Anchors
Seantmns3DDSSe (Abdelrcheem et al., 2024)
16,797
83,572
613
26.1
10"
ScmmEns3D- Abbdlrehoem al., 2024)
14,710
10"
ScanEnnx33 ScanRefer (Abdelreheem al., 2024)
18,989
51,583
800
28.5
10"
ARKalSeenReeer (Kato 2023)
109
LOCATE 3D DATASET (ours)
3
23,549
131,641
1.346
123.7
105
D.4. Why does the additional training data help?
We ran an additional experiment in which we find that scene diversity a key factor L3DD improvements. Specifically, we compare two conditions: (1) ScanNet training data + 30K L.3DD samples also from ScanNet and (2) ScanNet data
23

---

LOCATE 3D: Real-World Object Localization via Self-Supervised Learning in 3D
Train data
Joint Evaluation
SR3D
NR3D
ScanRefer
Base+ScanNett+
0.632
0.674
0,562
0.608
Basc+ScanNct++ (updated task) Sase+ScanNNt
0,631
0.668
0,568
0.612
0.618
0.664
0.524
0.603
Table 10: Impact of scene diversity. We train all models on SR3D+NR3D+ScanRefer and add 30K samples from L3DD., We ablate whether these extra samples come from ScanNet ScanNet++.. We also ablate whether we are using the first or second iteration of the annotation instructions.
+ 30K L3DD samples from ScanNet++ (i.e., same quantity, but better quality). Table 10 shows that training on better quality scenes (2) outperforms (1) by about 1.5 percent. The use of data collected from updated instructions of our task (high-distractor objects targets, multi-anchor) does not significantly impact performance.
Table 11: Robot Experiments
Model
Task Success
Concept Fusion
2/10
VLM (Llama-3)
5/10
VLM (GPT-4o)
5.66/10
LOCATE 3D+ (ours)
8/10
E. Woald Robot Experiments
E.1. Robot Experiment Details
LOCATE 3D is intended to work in the real world, the robot trials test the performance of the models in real setting. These trials serve as a benchmark of evaluating different models. We are using Boston Dynamics Spot robot to carry out these experiments. The scene, a model apartment, this differs considerably from our training conditions by being multi-room test apartment and much larger area.
We scan the environment using iPhone's Record3D app, which provides RGB-D data that is processed by our preprocessing pipeline into 5cm featurized voxel maps. The testing environment represents challenging out-of-distributio scenario, featuring rooms regions and object categories unseen during training. The full scan of this apartment was processed offline to get detected bounding boxes for 10 pre-selected reference expression. Each of these references to a unique furniture and the prompts were 1. kiithhenn island, 2. rectangular dining table, 3. sink, 4. night stand near the window, 5. bed with pillow, 6. dresser the office, 7. dresser the hallway, 8. coffee table in front the sofa, 9, desk office, 10, kitchen counter with stove
We evaluate against two baselines: (1) A VLM--assed pipeline using GPT-4/Llama and SAM2, and (2) ConceptFusion, 3D mapping system. Results in Table 11 show LOCATE 3D+ achieves 8/10 performance.
The trial had tasks like "navigate to the coffee table in from of the sofa and pick up the plush toy". A given episode had 3 parts: location, navigation then pick up. Out of these only localisation was expected to be done via the model being benchmarked. Our navigation method uses a heuristic that combines A* path planning with point cloud data to identify a point near to the object. Specifically, this involves generating a low-resolution 2D occupancy grid from the point cloud, which enables the determination of accessible free spaces. The pick-up part of the task has a Boston Dynamics API call to grasp and OWL-ViT (Minderer et al., 2022) object detector. After completing navigation a gaze policy inspired by (Yokoyama et al., 2024) is used to scan the scene and the given object e.g "plush toy" was detected with object detector and the grasping API call was initiated. A closed set of objects for which it was tested out that the robot had high successes of grasping was used.
To ensure reliability, each task was repeated three times, resulting in a total of 30 trials per model on the Spot robot. successful trial was defined as a sequence three consecutive tasks: accurate localization, followed by effective navigation,
24

---

LOCATE 3D: Real-World Object Localization via Self-Supervised Learning in 3D
Boston ynnaic
Figure 12: Examples of the Spot robot at the end of navigation task before the pick task (right) the output bounding boxes of LOCATE 3D+ model (left). The task (top) "navigate coffee table in front of the sofa and pick up the bottle" (middle) "navigate sink and pick up the pineapple plush toy" (bottom) "navigate to the dresser in the hallway and pick up pink plush toy"
and culminating in successful pick action. Each trial had binary outcome, with success contingent on the correct execution of all three tasks. We report the number of successful episodes averaged over the repetitions achieved by each model. Notably, our results show that correct localization consistently facilitated successful navigation and pick-up.
An interesting finding is that coarse localization (within 2m of target) often proved sufficient given the robust manipulation capabilities of the pipeline. While triplicating results the VLM had a different seed value while generating the localization bounding boxes. The VLM pipeline being non-deterministic this gave different results. A successful detection for a particular seed value might not be replicated for all tries. That the reason why with averaging of these values a non-whole number
25

---

LOCATE 3D: Real-World Object Localization via Self-Supervised Learning in 3D
part of the reported results.
We also observed that times the most obvious way to refer to objects can include the room they are located in (c.g., "dresser in the bedroom").. However, our training data lacked such language, highlighting need for future work to incorporate this type of spatial context.
F. Vision-Language Model (VLM) Baselines
This section describes two baseline models that leverage vision-language models (VLMs) Llama-3 (Meta AI, 2024) and GPT-4o (OpenAI, 2024) within modular pipeline designed to solve the 3D referring expressions task, which is similar to (Xu et al., 2024). Specifically, both baselines include three primary modules: (1) frame selection, (2) 2D object selection, and (3) 2D-to-3D lifting. Results for these VLM baselines are in Tables and 4 and method details are provided below.
Frame Selection. In our L (Meta AI, 2024) VLM baseline, frame selection is performed by captioning (K = 20) uniformly sampled RGB frames from the observation stream. Specifically, for each frame we use SAM-2 (Ravi et al., 2024) to segment the image, and set-of-mark prompting (Yang et al., 2023b) with Llama-3 to generate a list of objects in the frame, which we treat as the frame caption. Next, we past the K = 20 captions to Llama-3 along with the referring expressions query, and ask the model to select the top-3 frame that may include the object described in the query. Finally, for each of the top-3 frames, ask Llama-3 to verify if object described in the referring expressions query is visible. The first verified frame is used in the subsequent modules. If none of the top-3 frames are verified, one is selected at random.
In our GPT-4o (OpenAI, 2024) VLM baseline, frame selection is performed by passing the K = 20 uniformly sampled RGB frames along with the referring expressions to the VLM, and asking the model to select the frame the includes the object described the query. The selected frame subsequently passed to the next module for 2D object selection.
2D Object Selection. In both baselines, object selection is performed using an open-vocabulary object detector (Ground- ingDINO (Liu al., 2023)), promptable image segmentation model (SAM 2 (Ravi et al., 2024)), and VLM (Llama- 3 (Meta AI, 2024) or GPT-4o (OpenAI, 2024)). Specifically, the full user query is passed to the object detector generate
2D bounding boxes for candidate objects in the scene. Instance segmentation masks for each object generated by SAM 2 and used to create a set-of-marks prompt (Yang et al., 2023b) for the VLM, which selects the object and correspondingly 2D object mask.
2D-to-3D Lifting In the final module, the 2D object mask is "lifted"" (or unprojected) into 3D. Specifically, we first use SAM 2 (Ravi et al., 2024) to propagate the 2D object mask over time. Next, we filter the 2D propagated mask, then unproject them into a 3D object point cloud using the depth observations and camera information (pose and intrinsics). Finally, we use DBSCAN find the largest cluster in 3D object point cloud, and use the bounds the predicted bounding box.
G. Full Ablation table
Table 12 provides a complete set of results for all of the LOCATE 3D and LOCATE 3D+ experiments and ablations in this work. For all methods, we provide results on SR3D, NR3D, ScanRefer (Achlioptas et al., 2020; Chen et al., 2020). For most methods, we additionally provide results for the ScanNet++ (Yeshwanth et al., 2023) and ARKitScenes (Dehghan et al., 2021) splits from LOCATE 3D DATASET.
26

---

LOCATE 3D: Real-World Object Localization via Sele-Supervised Learning in 3D
Table 12: Comparison of various ablations and baselines. Metrics shown as (@25, @ @50) for SN Joint, SR3D, NR3D, ScanRefer, SN++ and ARKitScenes.
Input Features
Enceder Init
Training Data
Other
SN Joint
8R30
NR3D
ScanRefeer
SNe4
ARKitSeenes
6/25
68:00
6:25
0550
6/25
6/25
68:00
6:
6850
6:25
6550
Jepa
(LOCATE 3D+) CLIP-L DINOw2 SAM-H
3D-Jepa
SN + LXID
0.637
0,513
0.682
0,548
0.561
0,432
0.61
0,509
0.899
0,575
0.576
0.208
Input Fenturization Ablotions
CLIP D MobileSMM
3D 3D-Jepa
SN
0.537
0.394
0.566
0.417
0.475
0.328
0.529
0,403
ClipLL SAM-I
30-3epo
SN
0.592
0.455
0.627
0.477
0.521
0,353
0.582
0.469
Jepa
0.421
0.437
0.545
Initialization Ablations
CLIP L DINOV2 SAM-H
Random
SN
0,588
0.463
0.625
0.496
0.516
0.382
0.567
0.461
0.515
0.255
0.417
0,152
CLIP L+ DENOv2 + SAM II
Random
SN LX3D
0.607
0.475
0.644
0.506
0.544
0.399
0.586
0.473
0.746
0.476
0.556
0.240
RGB Emput Ablations
Random
SN
0.513
0,386
0.409
0.500
0.150
0,042
0.445
0:457
0,493
0.576
0.474
0.697
"Facoder-Free"
CLIP L DINOV2 SAM
NIA
SN
0.558
0.397
0.562
0.414
0.463
0,528
0.551
0.418
0.461
0.179
0.218
0.058
RGD
NIA
SN
0.289
0.174
0.278
0.174
0.27
0.133
0.342
0.205
0.158
0.046
0 0.009
0.000
Supersision and Dreeder Heads Ablations
CLIP DINOx2 SAM II
Jepa
SN
Naive Bbox
0.554
0.391
0.395
0.426
0.464
0.342
0.391
0.545
0,250
0.177
Jepa
C DINO22 SAM-II
3D-lepa
MLP Bbox
0.356
0.135
0.149
0.285
0.356
0.133
0.032
0.002
0.001
CLIP-I+ DIND2 SAM-H
ih-lepp
SN
No Mask (oun head)
0.003
0.000
0.003
0.000
0.002
0.000
0.005
0.000
0.001
0.000
0.000
0.000
CLIP-L + DINOv2 SAM-H
3D-lepa
SIN
No Mask (mlp head)
0.294
0.100
0.311
0.102
0.239
0,077
0.306
0.115
0.025
0.003
0.011
0,000
Desader Ablations
Jepa
Decoder
0.567
C DINOv2 SAM-II
Random
SN
Small Decoder
0.564
0.360
0.584
0:415
0.460
0.314
0.332
0,500
0.448
0.190
0.121
CLIP-I+ DINOv2 SAM-H
3D-3epo
SN
Base Trooder
0.605
0.476
0.682
0.511
0.524
0.360
0.583
0.476
0.582
0.267
0.450
0.165
CLIP-1 DINOv2 + SAM-H
Random
SN
Rosc Deoader
0.576
0.445
0.615
0.478
0.505
0.367
0.557
0.442
0.495
0.231
0.395
0.149
Truining Ablatioss (Pinetuning and Scheduler)
CLIP-
3D-lepo
CLIP-1 DINO22
3Dslqpa
Naive Schedoler
0.470
0.503
0.57%
0.475
0.534
0.348
0.200
CLIP-L + DINOW2 + SAM-H
Randons
SN
Our schedaler
0.598
0.470
0.642
0.502
0.524
0.393
0.571
00268
0.532
0.265
0.439
0.168
Other Baselines
RGB
Ponde/V2
SN
0-432
0.309
0.431
03312
0.366
0.362
0.469
0.340
0.297
0.130
0.048
0,012
27


Reviews:
[{"review_id": "HBmVmHQ8hL", "reviewer": "Reviewer_J9xs", "summary": "This paper introduces Locate 3D, a model for localizing objects in 3D scenes from referring expressions, achieving state-of-the-art performance on standard referential grounding benchmarks. The key innovation is 3D-JEPA, a self-supervised learning (SSL) algorithm to learn contextualized scene representations. Locate 3D outperforms existing methods on SR3D, NR3D, and ScanRefer with fewer assumptions. Additionally, the LX3D dataset (130K+ language annotations) boosts generalization. Experiments demonstrate the model's strong performance and potential for deployment in real-world systems.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough review and constructive feedback. Below we address each point in detail.\n\n**Clarification about AR application**\n\nIndeed, we do not demonstrate deployment on an AR device. Our intended claim is that Locate 3D's input format is directly compatible with data streams from AR devices and robots - it operates on raw RGBD and camera parameters, which is the standard output format of these platforms. We demonstrate this compatibility through evaluation on diverse sensor-captured datasets (ScanNet, ScanNet++, ARKitScenes) as well as robot deployment. We will revise our phrasing to clarify that we're referring to input compatibility rather than explicit AR deployment.\n\n**Inference speed, memory usage, computational cost**\n\nWe will add this discussion to the paper. Our inference pipeline operates in two phases. First, we compute and cache 2D features and a featurized pointcloud of the environment. This can be  done offline for static environments like ScanNet, or during an initial exploration phase for robots. With this cache in place, our model performs inference in approximately one second for a scene with ~100,000 feature points with an average peak VRAM of ~8.1 GB on one A100. \n\n**LX3D dataset analysis; data volume vs diversity**\n\nWe ran an additional experiment, in which we find that **scene diversity is a key factor in LX3D improvements**. Specifically, we compare two conditions: (1) ScanNet training data + 30K LX3D samples also from ScanNet and (2) ScanNet data + 30K LX3D samples from ScanNet++ (i.e., same quantity, but better quality). We find that training on better quality scenes (2) outperforms (1) by ~2%. We will include this additional experiment in a revision.\n\nThe annotation quality of every sample LX3D was validated by human annotators. Specifically, only samples marked as unambiguously correct are included in the final dataset to ensure reliability, more details about our annotation setup and dataset can be found in Appendix D.\n\nFurthermore, we will publicly release the LX3D dataset upon publication, providing a high-quality asset to support future research in 3D referential grounding.\n\n**Combining RGB, CLIP, DINO features**\n\nThe features are combined by concatenating CLIP and DINO features with a harmonically embedded representation of the RGB features. Specifically, CLIP features are generated for SAM masks, with a learnable token used when mask features are unavailable. Full details are provided in Appendix A.2 and we will add a summary to the main paper. For more details, we invite the reviewer to read the response to a similar question posed by reviewer *gKhM*.\n\n**Impact of different 2D image features**\n\nWe provide experimental analysis in Table 3. We observe that features extracted from larger models (CLIP-L, SAM-H) consistently outperform those extracted from models with fewer parameters (CLIP-B, MobileSAM), with CLIP-L achieving 59.2% Acc@25 compared to CLIP-B's 53.7%. Additionally, concatenating features from CLIP and DINOv2 significantly improves results compared to using either of these in isolation, reaching 61.7% Acc@25, indicating that each of the features provide complementary information that benefits 3D object localization.\n\n**Discussing related work that performs lifting of 2D features (OpenScene, Lift3D)**\n\nWe appreciate the suggestion. Approaches such as OpenScene (CVPR 2023), Lift3D (CVPR 2024), and prior ideas proposed in distilled feature fields (Neurips 2022), similarly lift 2D features to 3D via neural rendering or contrastive learning. However, we incorporate two key advances in addition to lifting 2D features. (1) our self-supervised 3D pretraining (3D-JEPA) approach contextualizes the lifted 3D features, (2) our specialized language-conditioned decoder enables precise referential object localization. Both (1) and (2) contribute significantly to our performance. As shown in Table 9, Locate 3D+ (8/10) substantially outperforms Concept Fusion (2/10), a zero-shot feature lifting method, on complex referential tasks. We'll add this discussion to the related work section.\n\n**Multi-view inconsistency of SAM masks**\n\nCorrect, SAM masks are not multi-view consistent. This limits the performance of prior methods that use 2D features lifted to 3D in a zero-shot fashion. However, our SoTA results (in Table 1) demonstrate that **3D-JEPA effectively learns to deal with such noise**. Qualitatively, we find that 3D-JEPA features are smoother and more consistent than 2D lifted features (as shown in Figure 1), demonstrating (again) that our method is robust to such issues.", "claims_and_evidence": "The claims made in the submission are generally well-supported by clear and convincing evidence. However, the claim about deployment could benefit from stronger evidence or additional clarification:\n- “... making it readily deployable on robots and AR devices” is supported by the experiments in Sec. 4.4. While the paper demonstrates deployment on a robot, the AR application is not explicitly evaluated.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria generally make sense for the 3D object localization task, but there are some areas for potential improvement.\n- Locate 3D is designed for real-world deployment, but the current version of submission lacks the discussion of inference speed, memory usage, or computational cost.\n- I understand LX3D improves performance, but it is still unclear whether the improvement is due to data volume or diversity?", "theoretical_claims": "The paper primarily focuses on algorithmic and empirical contributions, rather than formal theoretical claims with proofs. \n\nFor line 113-115, authors claim the feature dimension “combines RGB, CLIP and DINOv2 features”. It is unclear how to combine those features as this is essentially an explanation of  $f_i$ in the corresponding equation.", "experimental_designs_or_analyses": "The experimental designs and analyses are well-structured and easy to follow. I have mentioned some points related to experiments in the previous sections to help strengthen this submission. In this part, it is also interesting to discuss how different 2D image features influence the performance of located 3D as it is mentioned in line 115-116 without experimental analysis.", "relation_to_broader_scientific_literature": "The key contribution of the paper is closely related to some prior findings/results/ideas including (1) 3D object localization work, (2) Serf-supervised Learning (SSL) on 3D representations and (3) 2D foundation models. Different from previous 3D object localization work, Locate 3D operates directly on sensor observation streams, and works with fewer assumptions. 3D-JEPA is a SSL method inspired by previous SSL work like PointMAE, which is specially designed for 3D object localization in order to learn conceptualized representations of 3D scenes. Large-scale 2D foundation models like DINOv2 and CLIP are treated as tools by Locate 3D for a better capability of localization.", "essential_references_not_discussed": "One of the core ideas of this work is to lift the general features produced by 2D foundation models. Therefore, I suggest discussing more about the existing work which conveys similar ideas in broader 3D scene understanding aspects. For example, OpenScene (CVPR’ 2023), Lift3D (CVPR’ 2024), etc..", "other_strengths_and_weaknesses": "Locate 3D has the potential to be a useful tool for some important downstream applications regarding higher-level 3D understanding since localization could be fundamental operations for scene understanding.\n\nThough the overall experimental structure in this submission is easy to follow, there are still some incomplete aspects regarding the LX3D dataset, especially considering that it is claimed as a core contribution. In this version of the submission, there is a lack of sufficient quantitative analysis (such as the aspects mentioned above) and qualitative analysis (e.g., annotation quality and granularity) of the LX3D dataset, which raises concerns about its reliability. Moreover, in the preprocessing stage, the authors mention using SAM to process images and obtain masks. However, SAM-generated masks do not ensure multi-view consistency, which is confusing to me. Using such masks for subsequent processing carries the risk of introducing excessive noise.", "other_comments_or_suggestions": "Overall, I believe this paper presents a useful tool for the community, but the experiments and analysis are not sufficiently thorough, which weakens some aspects of its contribution. However, if the authors can address my concerns, I would consider adjusting my score accordingly.", "questions_for_authors": "I have the following key questions that, if addressed, could potentially change my evaluation of the paper:\n1. Clarification on LX3D Dataset Analysis\n2. Computational Efficiency and Scalability\n3. Multi-View Consistency in SAM-based Preprocessing\n4. Claimed AR related applications\n\nFor the main concerns listed above, details can be found in the previous sections. For other minor concerns mentioned, I believe addressing them could also help strengthen this submission.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nenCsfrRS9", "reviewer": "Reviewer_C37t", "summary": "The paper proposes Locate 3D, a model for 3D grounding which achieves SOTA performance and strong out-of-domain generalization.\nSpecifically, a novel self-supervised learning method, 3D-JEPA, is proposed, generating contextualized features for the scene. Also, a new dataset LX3D is proposed to test the robustness of the proposed model.\n\n## update after rebuttal\nPlease see the rebuttal comment below.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough feedback. Below we address each point in detail\n\n**Clarification on mesh PC vs sensor PC**\n\nBy \"Mesh PC\" we refer to pointclouds sampled from carefully reconstructed 3D meshes that undergo extensive post-processing. Most prior works use this format as it’s ScanNet’s default.\n\nIn contrast, \"Sensor PC\" refers to pointclouds obtained directly from RGB-D sensors. These contain raw depth measurements including noise, missing regions, and registration errors. This format better represents real-world deployment scenarios.\n\nMethods originally reported on Mesh PC suffer significant performance drops (8-10%) when trained and evaluated on sensor data (Table 1), 3D-VisTA drops from 53.1% to 45.9% Acc@25 and BUTD-DETR from 50.28% to 40.7%. This gap was also observed by Odin (Jain et al, 2024). Under the (more rigorous) evaluation setting comprising sensor pointclouds, Locate 3D achieves SoTA (61.7% Acc@25) performance. We will clarify this terminology in the paper.\n\n> For the mesh pc methods in Table 1, the reviewer believes most of them are using pointcloud as input, not mesh.\n\nThe reviewer is correct – while these methods use pointclouds as input, we use the term \"Mesh pointcloud\" to indicate that these pointclouds are sampled from cleaned, post-processed meshes.\n\n**3D-VisTA evaluation clarification**\n\n3D-VisTA was originally designed for mesh pointclouds with a two-stage architecture: first using Mask3D to generate object proposals, then processing these proposals with the pointcloud for grounding. Due to implementation constraints and pre-computed proposal dependencies, we report results using sensor PC for 3D-VisTA while maintaining the original Mask3D proposals from mesh PC.\nWe note that **this represents an upper bound for 3D-VisTA's performance** on sensor data, as using sensor PC for proposal generation would only decrease performance further. Even with this advantage, 3D-VisTA achieves 45.9% Acc@25, while Locate 3D significantly outperforms it at 61.7%.\t\n\n**Question about query selection**\n\nFor ScanNet benchmarks we select the query with the highest “token probability” for the target noun. In our robot experiments, we use an off-the-shelf LLM to parse the referring expression and identify the target noun before applying the same selection mechanism.\n\n> Authors mention that 3D grounding approaches [...] are prone to failures'. The reviewer is curious about the differences.\n\nThese approaches, such as 3D-VisTA (Zhu et al., 2023) and PQ3D (Zhu et al., 2024), rely on external object detectors to generate proposals without considering the language query and then train their model to select one. If the object proposal misses, the entire model is bound to fail. Single-stage approaches like ours and BUTD-DETR (Jain et al., 2022), directly predict the bounding box by jointly considering the scene and language tokens. We will incorporate this discussion into our paper.\n\n**LX3D out-of-domain evaluation clarification (ScanNet vs ScanNet++)**\n\nWe see key differences in scanning hardware, setting (partial scene vs. full scene), scene size, and distribution of queries between the train and eval settings. Further, our eval set contains both ARKitScenes and ScanNet++. While we agree that some aspects of the visual distribution remain constant (e.g. common object classes), we believe that the evaluation faithfully represents the domain gap present during a new deployment of our model to indoor scenes.\n\nAs for your latter point (Locate 3D not being the best in multiple subsets) – we believe the VLM approach is strongest on ScanNet++ because each input contains only 5 frames, and Locate 3D not achieving maximum performance on FRE is an artifact of the small eval set size (~250 annotations on 1 scene).\n\n**Better perf on Acc@25 but lower on Acc@50**\n\nThis discrepancy arises because PQ3D operates on mesh pointclouds, while our method works on sensor pointclouds. At lower IoU thresholds (Acc@25), our approach effectively predicts the bounding box despite sensor data imperfections. However, at higher thresholds (Acc@50), achieving precise alignment becomes more challenging, resulting in a more significant performance drop for sensor-based methods. In short, PQ3D is evaluated in a more relaxed setting than ours, and is at an advantage.\n\nDespite our best efforts, we could not re-train PQ3D with sensor pointclouds due to their use of multiple backbones and multi-stage training strategies, making a direct comparison difficult. \n\n**CLIP and DINO backbones in Table 3**\n\nWhenever we use CLIP features, we first obtain masks using SAM and then extract CLIP embeddings for these masks following Conceptfusion. **This approach is necessary because CLIP produces global image embeddings, not patch or pixel-level features**. For DINO, we directly use its dense patch-level features and map them to individual points without requiring a segmentation step. We will add this explanation to Section 2.1.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "The proposed methods and evaluation are reasonable.", "theoretical_claims": "No theoretical claims and proofs involved.", "experimental_designs_or_analyses": "The proposed new dataset LX3D contains data from ScanNet and ScanNet++, where the visual data are similar. This poses some question in terms of the effectiveness on evaluating the robustness for out-of-domain data.", "relation_to_broader_scientific_literature": "This paper proposes a framework, which enables 3D grounding on posed RGB-D frames. It shows the real-world deployment on robots, and could potentially extend to AR devices.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strength:\n1. The paper is well-written, with clear structure and illustrations.\n\nWeakness:\n1. It's unclear how the authors define mesh PC and sensor PC. The reviewer notice the section C.3 in the appendix, but still get confused about the categories in Table 1. The authors could further clarify the input format for different methods, and the key difference for this paper.", "other_comments_or_suggestions": "For table 1, please bold the best result for Acc@50 on ScanRefer for consistency.", "questions_for_authors": "1. It's unclear how the authors define mesh PC and sensor PC. For the mesh pc methods in Table 1, the reviewer believe most of them are using point cloud as input, not mesh. \n2. For 3D-VisTA, how do you evaluating under Mesh PC and Sensor PC + Proposals from Mesh PC settings? Why not Sensor PC and Sensor PC + Proposals from Mesh PC settings, which is more comparable.\n3. Based on Figure 3, how do you choose the final prediction from the Q generated box? In the related work section (line 404-right column), the authors mention that 3D grounding approaches that 'provide probabilities for region proposals' is 'often difficult in 3D, and are prone to failures'. The reviewer is curious about the differences.\n4. The proposed new dataset LX3D contains data from ScanNet and ScanNet++, where the visual data are similar. This poses some question in terms of the effectiveness on evaluating the robustness for out-of-domain data. Also from the results in table 4, the LOCATE 3D is not the best in multiple subsets.\n5. In table 1, it is interesting that the proposed methods outperforms the existing methods in Acc@25, but not on Acc@50. Do the authors have some intuition why it is the case?\n6. In table 3, it is unclear why the authors choose to use different visual backbones. Does CLIP+DINO indicates using different backbones for masks and image separately, or using both features for both masks and the image? If it is the former one, why using different backbones for masks and image? Why not directly using CLIP for both or DINO for both?\n\nThe reviewer may adjust the final rating after rebuttal based on the clarifications from the authors.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wwKC77o0Gw", "reviewer": "Reviewer_fbHG", "summary": "This paper focuses on object localization via referring expressions in 3D real-world scenes to understand the 3D physical world, which is a valuable task in 3D perception and understanding. It introduces an end-to-end 3D transformer network to get point cloud, 3D features, and text query as input, and output the potential masks, box, and predicted text tokens. To get better 3D feature representation, it uses a novel self-supervised learning algorithm for 3D point clouds, 3D-JEPA, which is inspired by the 2D JEPA framework. The motivation is clearly explained in the paper.\n1. A novel SSL method for 3D point clouds, 3D-JEPA. It demonstrates that by adopting the pre-trained features from 3D-JEPA, the global perception capability is greatly enhanced. It can also provide improvements for downstream locate3D tasks in both in-domain and out-of-domain scenarios.\n2. An end-to-end 3D transformer model for the 3D referring expression task, locate3D. It leverages the overall performance on public 3D benchmarks like ScanRefer, SR3D, and NR3D.\n3. An incremental 3D referring expression dataset, LX3D. The samples are originally captured in ScanNet, ScanNet++, and ARKitScenes, and expand the language annotations.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed feedback. Their comments have helped us identify areas where we can better explain and justify our technical contributions. We address each comment in detail below.\n\n**Deriving better 3D features from 2D features; 3D-JEPA**\n\nThe 3D nature of our approach comes from the architecture and learning objective rather than the individual pointwise inputs. Our model processes the entire featurized pointcloud using explicit 3D coordinates to produce per-point features. 3D-JEPA's objective forces the model to learn additional information beyond 2D features through masked prediction in 3D space, requiring it to infer features for occluded regions using contextual information from the surrounding visible 3D scene. In short, the \"3D\" in 3D-JEPA refers to the domain in which the representation learning occurs.\n\nOur approach has precedent in prior work like OpenScene, which demonstrated that distilling 2D vision-language features to 3D via contrastive learning results in 3D models that outperform the original 2D features. \n\nEmpirical evidence shows 3D-JEPA features encode valuable information beyond lifted 2D features:\n1. 3D-JEPA features demonstrate superior performance in linear probing experiments (Section 4.2) when compared to ConceptFusion features (34%→39% and 66%→73%)\n2. Frozen 3D-JEPA features provide ~5% improvement over ConceptFusion inputs to the decoder (Table 2)\n3. ConceptFusion (direct 2D-to-3D feature lifting) achieves 20% success on our robot evaluations compared to Locate 3D's 80% (Table 9)\n\n**Aggregating multi-view features**\n\nWe voxelize the pointcloud at 5 cm resolution and aggregate multi-view point features within each voxel using weighted averaging based on trilinear interpolation distances. This aligns with standard practice in prior works (e.g., ConceptFusion); given space constraints, please also refer to our response to reviewer gKhM.\n\n**Farthest point sampling for masking**\n\nYes, we tried farthest point sampling but did not see improvements. We additionally experimented with several other masking strategies as discussed in Appendix A.1. However, none of the methods improved over random masking (similar to the findings in Assran et. al., 2023). Thus, we use random masks for their computational efficiency.\n\n**Fairness of VLM baselines and usage of depth information**\n\nYes, the two VLM baselines use depth information. In fact, they use the exact same inputs as Locate 3D (RGB-D + camera pose and intrinsics) to ensure fair comparisons. The VLM baselines are detailed in Appendix F.\n\nAt a high level, these baselines use a modular pipeline that consists of first selecting an RGB frame using a VLM, then selecting an object in the 2D frame using GroundingDINO, SAM-2, and a VLM, and finally determining a 3D bounding box using depth and camera extrinsic and intrinsic information. \n\n**Comparison with 3D-LLM and LEO on ScanRefer**\n\nLocate 3D significantly outperforms 3D-LLM [1]. Specifically, 3D-LLM reports 30.3% Acc@0.25 on ScanRefer, while Locate 3D achieves 59.9% (Table 1). We will add results from 3D-LLM [1] to Table 1. LEO [2] is evaluated on ScanQA but not on ScanRefer (note that ScanQA assumes text outputs, while ScanRefer requires bounding boxes or instance masks). We will discuss LEO in related work. \n\n**Comparison with other 3D semantic alignment and scene-level features**\n\nPlease refer to our previous response for a comparison of our approach to 3D-LLM, which trains a model to consume the entire scene. We are also happy to discuss additional related work if you might have concrete suggestions. \n\n**PTV3: Scratch vs pre-trained in Table 2**\n\nIn our approach, PTV3 is pre-trained with 3D-JEPA. In Table 2, we study alternatives including “random” initialization (i.e., PTV3  is initialized from scratch), and initializing from “PonderV2” weights. In all three cases, the encoder is finetuned end-to-end for referential grounding. We will add these details to the Table 2 caption.\n\n**Table 2: Random initialization vs. 3D-JEPA (Frozen)**\n\nThe better performance of random initialization compared to frozen 3D-JEPA can be attributed to the significant number of extra trainable parameters (~250M) that are optimized specifically for the referential grounding task. When we freeze the 3D-JEPA encoder, we only train the decoder and significantly limit the model's ability to adapt the encoder parameters to the task at hand. However, it's important to note that fine-tuning the 3D-JEPA encoder (our full model) provides the best performance, showing that while trainable model capacity is important, the pre-trained weights offer valuable initialization that leads to better results when allowed to adapt to the target task.\n\n**Table 1 clarifications (metrics, setting, VLM)**\n\nThe VLM baselines (LLaMA and GPT-4o) use the exact same metrics and settings as other methods (such as Locate 3D) in Table 1. Specifically, the VLM baselines return 3D bounding boxes as detailed in Appendix F.", "claims_and_evidence": "none", "methods_and_evaluation_criteria": "M1. The idea of deriving btter 3D features from 2D pre-trained features seems somewhat unconvincing. How do you demonstrate that the 3D capability is obtained based on 2D pre-trained features. Although PTV3 is used as the encoder, the approach is still primarily based on 2D features and their spatial contextual proximity for supervision. Therefore, I have some concerns about the name \"3D-JEPA\".\n\nM2. What is the strategy for aggregating multi-view features for the same point? Does this strategy affect the accuracy of the final referring expression task? How do you handle multiple features from different perspectives for the same point?\n\nM3. During the 3D-JPEA training, to what extent can the mask be achieved? Have you considered mask the points with non-random yet reasonable sampling methods such as farthest point sampling ?", "theoretical_claims": "none", "experimental_designs_or_analyses": "In Table 1:\nE1. The comparison with GPT-related methods seems unfair. How is the VLM+baseline method implemented is not discussed in the paper. Did it utilize depth information?\nE2. How does it compare with other 3D-LLM methods, such as 3D-LLM[1], Leo[2], or related works, which have also been implemented on datasets like ScanRefer?\nE3. There is a lack of comparison with other 3D semantic alignment and scene-level features.\n\nIn Table 2:\nE4. Is PTV3 trained from scratch or using pretrained weights?\nE5. Why does random initialization perform better than the 3D-JEPA fixed scenario? What is the context of the 3D-JEPA Frozen example, and please provide an explanation.\n\n[1] 3D-LLM: Injecting the 3D World into Large Language Models\n[2] An Embodied Generalist Agent in 3D World", "relation_to_broader_scientific_literature": "None.", "essential_references_not_discussed": "1. 3D-JEPA: A Joint Embedding Predictive Architecture for 3D Self-Supervised Representation Learning", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "Please see above.", "questions_for_authors": "Q1. I am very interested in understanding what additional information, at the feature level, is provided by 3D-JEPA training compared to the original 2D pre-trained features. Additionally, how can this be demonstrated experimentally?\nQ2. I am a little confused the metrics and settings in table1 when it compares to other GP4-4o methods. For those MLLM  methods, how to locate the 3d bbox?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0NZxqhtLAn", "reviewer": "Reviewer_gKhM", "summary": "This paper proposes a method called Locate3D, which addresses the task of referring expression-based localization in 3D. The method consists of three key stages: (1) pre-processing stage in which an input RGB-D stream is processed using 2D foundation models SAM, CLIP and  DINOv2 in order to obtain 2D features which are then lifted to 3D following the ConceptFusion framework. The resulting point cloud representation associates each point with RGB, CLIP and DINOv2 features. (2) This representation is then used to first train a transformer-based encoder following a self-supervised learning framework called 3D-JEPA. This encoder is trained by using the self-supervised task of learning to predict per-point features in regions that are masked-out by a region sampling process. (3) Next, a decoder that takes as input the embedding representation obtained from the 3D-JEPA encoder, as well as a referring-expression is trained using the tasks of dense-3D segmentation as well as 3D bounding box detection. The model is trained using existing referring expression-based localization datasets as well as the LX3D dataset, which is a dataset proposed in this submission with the aim of extending existing 3D training datasets (ScanNet, ScanNet++, ARKitScenes) with referring-expression annotations in an automatized manner to obtain a referring expression dataset with a larger number and variety of scenes.\n\n### Update after rebuttal\nThe clarifications and discussions provided in the rebuttal have sufficiently addressed my concerns. I am still leaning towards acceptance, and will be keeping my original score (4: Accept).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for the thorough reading of our manuscript and valuable comments.  Below we address each point raised in detail\n\n**Clarification regarding our LX3D**\n\nWe noted the reviewer mentions our dataset being automatically generated, we'd like to clarify that our dataset is human-collected. Annotators curated and validated every sample. We invite the reviewer to read more details about our annotation setup in Appendix D.\n\n**Runtime analysis and assumptions [L029-L033]**\n\nIndeed, extracting 2D features and lifting them to 3D is computationally expensive. We address this by caching the environment’s 2D features for each view, as well as the featurized point cloud. For ScanNet experiments, we compute this cache offline; and for robot experiments we compute it while doing the initial environment exploration phase. With this feature cache, a forward pass of our model takes ~1 second for a scene with ~100k feature points and utilizes ~8 GB of VRAM on an A100 GPU.\n\nWe can utilize such caching because our benchmarks operate under static (ScanNet) or quasi-static (robot) environments. Extending our approach to dynamic scenes would require real-time 2D feature computation and continuous updates to the featurized environment. We believe the former is a matter of engineering, while the latter is an active research area, explored by methods like Lifelong LERF (A. Rashid, 2024). We will include these assumptions in our limitations section.\n\n**[L086-089] rewording**\n\nWe propose rewording this claim to more accurately state: \"Locate 3D does not require ground-truth region proposals, high-quality mesh sampled pointclouds, or surface normals at inference time, making it suitable for real-world deployment.\"\n\n**Clarification on positioning**\n\nWe propose revising [L011-L014, right col] to clarify: \"Many existing 3D RefExp models are evaluated on benchmark datasets with pointclouds sampled from high-quality post-processed 3D meshes or object instance segmentations already available. In contrast, Locate 3D operates directly on raw sensor observation streams without requiring intermediate processing steps like mesh reconstruction, instance segmentation, or pre-trained object detectors at inference time.\"\n\nWe want to highlight that most previous work has been trained and evaluated utilizing mesh pointclouds, while our approach leverages pointclouds obtained directly from RGB-D sensors (please see our response to reviewer C37t on this topic).\n\n**Discussion of ConcreteNet**\n\nWe will add ConcreteNet to the related works section and baselines in Table 1. We note that this method operates on ScanNet's mesh-sampled pointclouds rather than sensor pointclouds. While ConcreteNet achieves 56.12% Acc@25 on ScanRefer, our approach demonstrates superior performance (59.9%) under the more challenging setting.\n\n**3D-JEPA exposition**\n\nWe appreciate the observation. In our revision, we will provide a clearer introduction to this concept in section 2.2 explaining (1) The core principles of the JEPA framework (2) Why adapting this approach to 3D pointclouds (3) Main deltas to the original framework.\n\n**Discussion on Limitations**\n\nWe will add a limitations section discussing (1) our current approach's constraints with dynamic environments (2) our method being limited to few-room environments (potentially addressed by exploring RoPE positional embeddings or similar techniques).\n\n**Typos**\n\nWe appreciate the careful proofreading, we will correct all of these in the revised manuscript.\n\n**Release of LX3D dataset**\n\nWe will publicly release the LX3D dataset, trained Locate 3D model, and 3D-JEPA backbone upon publication.\n\n**Generalizability (novel-object localization)**\n\nWe believe both questions can be boiled down to how well CLIP features are preserved through 3D-JEPA pre-training. Our linear probing experiments in Section 4.2 address this, particularly on the \"noun correctness\" task 3D-JEPA achieves 73% accuracy compared to ConceptFusion's 66%. These results demonstrate that 3D-JEPA not only preserves but enhances the semantic understanding from foundation models. This pattern mirrors developments in language modeling, where contextualized embeddings (like BERT) consistently outperform non-contextualized word embeddings for both common and rare words.\n\n**Mask and patch feature aggregation**\n\nFor 3D points observed in multiple views, we aggregate features by weighted averaging. We voxelize the pointcloud (5 cm voxel size) and compute a single feature per voxel by weighted averaging all contained features. Weights are calculated using trilinear interpolation based on distance to voxel boundaries. This weighted averaging approach is common practice in both our baselines (e.g., ConceptFusion) and traditional RGB-D mapping literature (Real-time 3D Reconstruction in Dynamic Scenes using Point-based Fusion - ICCV 2013). We explored alternative aggregation methods (max pooling, simple average, summation) before selecting weighted averaging.", "claims_and_evidence": "1. _[L011-L014, right column]_ The claim \"They (existing 3D RefExp models trained on small benchmark datasets) often require human annotation at inference time in the form of detailed 3D meshes or object instance segmentation, making them difficult to deploy on real-world devices\" is not fully accurate in my opinion. Such methods can use human annotated object masks for instance, but currently there are many strong 3D object instance mask predictors, 3D object detection methods and improved mesh reconstruction methods. The fact these methods were evaluated using GT boxes etc. does not necessarily make them unsuitable to real-world device deployment. I think this is important to clarify as this is critical for positioning the paper. \n2. _[Title]_ \"Locate3D: Real-World Object Localization via Self-Supervised Learning in 3D\": This is not too critical and a relatively minor point: At the first glance, the title leads the reader to start with the assumption that the whole training process takes place within a self-supervised learning framework, whereas the proposed model actually indeed uses GT annotation data consisting of target object localization data paired with the referring expression queries. \n3. _[L029-L033]_  It appears to me that the claim \"Locate3D operates directly on sensor observation streams without requiring manual post-processing (e.g., 3D mesh refinement or ground-truth instance segmentations), making it readily deployable on robots and AR devices\" would mean that given an RGB-D sequence, all processing and inference can take place in close-to real-time. However, I was unable to see an analysis or data on the runtime of the method. If my understanding is correct, the 2D feature computation using SAM/CLIP/DINO also takes place during inference time, and I would expect this part to take a few minutes. This could mean that the claims about real-world deployment readiness might need to be revised.\n4. _[L086-089]_  Similarly, I find the claim \"Locate3D does not require ground-truth region proposals, meshes, or surface normals, making it suitable for real-world deployment.\" partially incorrect if I have a correct understanding of the method. It appears to me that the method Locate3D indeed requires ground-truth region proposals, but only during training. But the claim I quoted sounds like the method never needs such paired annotations of segmentation and referring expressions. Please correct me if I misinterpret this statement or perhaps misunderstand this aspect of the proposed method.", "methods_and_evaluation_criteria": "Proposed method is reasonable, successfully targeting the 3D referring expression-based localization task. Proposed method and its components are thoroughly ablated. Evaluation criteria, metrics and chosen benchmark datasets follow the commonly employed evaluation methodology for the task. There are also additional results on the proposed LX3D dataset.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "I found that the experimental design is well-founded in general. The paper follows the established benchmarks and evaluation methodologies for 3D-RefExp task. There are plenty of ablation studies analysing different aspects of the method: input features, encoder configurations and the use of different foundation models. Additionally, I also appreciated experiments on cross-dataset evaluation as well as real-world deployment with a mobile robot setup.", "relation_to_broader_scientific_literature": "3D-JEPA: This framework very closely follows the original JEPA framework in ideation. I also have identified a few other extensions of the JEPA framework to 3D. Generally I find that the idea is not necessarily novel, but designing 3D-JEPA in the context of improving per-point features lifted from VLM models is a novel idea as far as I can judge. LX3D dataset is generally similar to existing benchmarks for 3D-RefExp task, but its key benefit appears to be the size of the dataset and the automatized generation of referring expression captions. However, such automatic caption generation methods were indeed proposed in prior work. The claims about the proposed contribution about achieving strong 3D-RefExp results \"with fewer assumptions compared to prior models\" need to be made a bit more sound for us to be able to evaluate the novelty of Locate3D a bit better.", "essential_references_not_discussed": "Related works are generally well discussed, but there are a few works that could be added. For instance, ConcreteNet (Unal et al, ECCV 2024, _\"Four Ways to Improve Verbo-visual Fusion for Dense 3D Visual Grounding\"_) only takes a point cloud as input, which appear to be under very similar assumptions as this work. In L394-395 of the submission, it is mentioned that \"few prior works operate under this setting\", but according to my understanding ConcreteNet follows this setting as well.", "other_strengths_and_weaknesses": "*Strengths*\n- The paper addresses a relevant and important task (3D referring-expressiong based localization). I also appreciated that the proposed method aims to target more realistic conditions with sensor PC instead of refined meshes.\n- The paper is generally written well with clarity.\n- I found that the proposed method is reasonable, and it is not an overly-complicated framework.\n- A very systematic and detailed evaluation of the method components is presented. \n- There are additional data contributions (if they will be publicly released)\n\n*Weaknesses*\n- I found that the level of detail regarding the 3D-JEPA framework could be improved as currently it reads a bit high-level. However, as this is one of the main proposed contributions of the paper, I would have expected to see a much detailed discussion on 3D-JEPA, as well as how it relates to the original JEPA framework, especially considering that many readers in the 3D scene understanding domain might not necessarily be familiar with the JEPA framework. \n- I did not find a thorough discussion on the limitations of the proposed method.", "other_comments_or_suggestions": "*Minor Comments*\n- 3D-JEPA term and what it stands for is never explicitly explained, at least not in the main paper. While there is a small reference to the JEPA framework (Assran et al.) in L135, this is still very unclear, as the broader audience/3D scene-understanding community might not be familiar with this particular framework. \n\n*Typos and grammatical issues:*\n- _L094 (left column) \"Finally, it also successfully localised objects ...\":_ \"localises\" might be better suited to the flow. \n- _L117 (left column) \"The point cloud_ $\\mathrm{PtC_{lift}}$ _can be directly use for object localization ...\":_ should be \"used\".\n- L676 (appendix) \"... figure 5 ...\":_ should be \"Figure 5\".\n- _L1038 (appendix) \"... while the PCA projections reveals that the encoder ...\":_ should be \"reveal\".\n- _L1039 (appendix) \"... while the decoder learns more sharped localized features ...\":_ the term \"sharper\" might be better instead of \"more sharped\".", "questions_for_authors": "- Is the LX3D dataset proposed in this work planned to be publicly released?\n- I am a bit unclear about the inference setting. At inference time, does the method still require the RGB-D stream and does it still need to use SAM/CLIP/DINO to obtain initial per-point features? If so, I would expect the feature extraction to take a few minutes for a given small/medium sized indoor scene, which might affect the claims regarding how much the proposed method is suitable for real-world deployment. As one of the main claims is \"Locate3D operates directly on sensor observation streams without requireing manual post-processing (e.g., 3D mesh refinement or ground-truth instance segmentations), making it readily deployable on robots and AR devices\" ([L029-L033]), how much time it takes to get the predictions is critical.\n- How is the generalizability of the method in terms of identifying novel classes affected by the contextualization of the features via the 3D-JEPA encoder? I understand that this process would enable the method to have less noisy features that are also more informative about the spatial relationships between scene objects. However, at the same time would this not result in a decline in novel-object localization performance? Also in the context of the 3D-RefExp task, I would expect the localization performance w.r.t. object-attribute based queries (such as \"red chair\") might also be negatively affected as the scene features now cross-attend to each other. Is there any component that is designed to addresses the preservation of object-attribute related features?\n- Generally, it would be great if a short discussion on the points listed in the \"Claims and Evidence\" section can be provided, especially number 3 and 4 (which mainly discuss the same aspect)\n- How is the mask and patch feature aggregation performed? I wasn't fully clear about this from L107.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The ratings for this paper are two accepts and two weak accepts. All reviewers think the paper has valuable contributions. The major concerns are about insufficient experiments and more discussions. The rebuttal has satisfactorily addressed the major concerns, and two reviewers raised their ratings. Therefore, the AC recommend acceptance.
Decision: Accept (spotlight poster)