From Thousands to Billions: 3D Visual Language Grounding via Render-Supervised Distillation from 2D VLMs
Authors: Ang Cao
Sergio Arnaud
Oleksandr Maksymets
Jianing Yang
Ayush Jain
Ada Martin
Vincent-Pierre Berges
Paul McVay
Ruslan Partsey
Aravind Rajeswaran
Franziska Meier
Justin Johnson
Jeong Joon Park
Alexander Sax
Venue: ICML 2025 poster (2025)
Keywords: 3D language grounding
Pretraining
Training 3D model using 2D
Primary area: applications->computer_vision
TL;DR: We use differentiable rendering to train 3D language grounding models

Abstract: 3D vision-language grounding faces a fundamental data bottleneck: while 2D models train on billions of images, 3D models have access to only thousands of labeled scenes--a six-order-of-magnitude gap that severely limits performance. We introduce \textbf{\emph{LIFT-GS}}, a practical distillation technique that overcomes this limitation by using differentiable rendering to bridge 3D and 2D supervision. LIFT-GS predicts 3D Gaussian representations from point clouds and uses them to render predicted language-conditioned 3D masks into 2D views, enabling supervision from 2D foundation models (SAM, CLIP, LLaMA) without requiring any 3D annotations. This render-supervised formulation enables end-to-end training of complete encoder-decoder architectures and is inherently model-agnostic.  LIFT-GS achieves state-of-the-art results with 25.7\% mAP on open-vocabulary instance segmentation (vs. 20.2\% prior SOTA) and consistent 10-30\% improvements on referential grounding tasks. Remarkably, pretraining effectively multiplies fine-tuning datasets by 2×, demonstrating strong scaling properties that suggest 3D VLG currently operates in a severely data-scarce regime. Project page: \url{https://liftgs.github.io}.

From Thousands to Billions: 3D Visual Language Grounding via Render-Supervised Distillation from 2D VLMs
Ang Cao 12 Sergio Armaud? 2 Oleksandr Maksymets 2 Jianing Yang" 12 Ayush Jain" 23 Ada Martin
Vincent-Pierre Berges² Paul McVay? Ruslan Partsey² Aravind Rajeswaran* Franziska Meier
Justin Johnson'
Jeong Joon Park'
Alexander Sax
Abstract
3D vision-language grounding faces a fundamen- tal data bottleneck: while 2D models train on billions of images, 3D models have access to only thousands of labeled scenes-a six-order-of- magnitude gap that severely limits performance. We introduce LIFT-GS, a practical distillation technique that overcomes this limitation by us- ing differentiable rendering to bridge and 2D supervision. LIFT-GS predicts 3D Gaussian rep- resentations from point clouds and uses them render predicted language-condittooee 3D masks into 2D views, enabling supervision from 2D foundation models (SAM, CLIP, LLaMA) with- out requiring any 3D annotations. This render- supervised formulation enables end-to-end train- ing of complete encoder-decoder architectures and is inherently model-agnostic. LIFT-GS achieves
Grounding Model
[Taa]] (larat [the]
(the)
EHII
FHII
2D VLM
2D Grounding Loss
Model
Figure 1: LIFT-GS Overview. We train powerful 3D vision lan- guage grounding model (i.e., 3D mask decoder) with point clouds and language as inputs by learning from VLM foundation mod- els without any 3D supervision.
of objects and 3D locations in space. Such 3D language grounding provides particularly natural interface for peo- ple to communicate about their surroundings. For Al sys- tems operating in physical spaces, identifying the 3D masks or bounding boxes indexed by language queries represents core functionality, with applications across autonomous navigation, robotic manipulation, and AR/VR.
results with 25.7% mAP spen-vocabulary instance segmentation (vs. 20.2% prior SOTA) and consistent 10-30% im- provements on referential grounding tasks. Re- markably, pretraining effectively multiplies fine- tuning datasets by 2x, demonstrating strong scal- ing properties that suggest 3D VLG currently op- erates in severely data-scarce regime. Project page: htttps://iiffti github.io.
Yet despite its importance, 3D vision-language grounding (3D VLG) faces fundamental bottleneck: data scarcity. While 2D vision-language models are trained on billions of labeled images and masks (Achiam et al., 2023; Touvron et al., 2023; Radford et al., 2021; Labs, 2023), existing 3D VLG models have access to only thousands of labeled 3D scenes and masks. This six-order-of -faagnniude gap in data availability severely limits the capabilities of current 3D grounding systems, creating one of the most significant challenges in embodied AI.
1. Introduction
When mentions the keys by the door the blue mug on the table, they use language to indicate specific set
A common workaround to this scarcity constructs 3D fea- ture fields from 2D features (e.g., CLIP embeddings) and performs text queries via dot products between the text and 3D embeddings. Although this provides good generaliza- tion, performance degrades with more detailed descriptions typical of real-world queries, as illustrated in Figure 3. From this perspective, the dual-encoder approach falls short of 3D grounding as it contradicts a core grounding requirement.
This work was partially done during the internship of Ang Cao at Meta. 'University of Michigan, Ann Arbor FFundamental AI Research (FAIR), Meta 'Carnegie Mellon University. Corre- spondence to: Ang Cao <ancao @ umu, Alexander Sax <ssax @ met..memtac......
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
In this paper, we ask: can we combine the best part of both

---

From Billions to Thousands: 3D Language Grounding via Render-Supervised Distillation
pipelines, L.e,, training powerful grounding model while still overcoming the data scarcity by learning from powerful 2D models? The key insight is that differentiable rendering provides natural bridge between 3D and 2D. If we can predict 3D masks and render them into 2D views, we can supervise them using 2D foundation models that have been trained on internet-scale data. This approach could enable training 3D models without any 3D mask annotations.
[The] [telephone] [that] [is] [besides] [the] [chair]
We introduce Sagguage-Inexxed Field Transfer with Gaus- sian Splatting (LIFT-GS), which implements this idea as a practical training pipeline. Given a point cloud and lan- guage query. LIFT-GS predicts 3D Gaussian representations that can be rendered into multiple 2D views. These ren- dered masks are then supervised using pseudo-labels from 2D foundation models. We show results where SAM pro- vides mask supervision (Kirillov et al., 2023), and CLIP or LLLMM provide language understanding (Radford et al., 2021; Meta AI, 2024). The approach effectively distills internet-scale 2D knowledge into 3D understanding.
Figure 2: 3D Referential Grounding, For each mentioned in- stance in text description, predict a 3D mask and map it to corresponding text tokens.
pervision is powerful, but we demonstrate that it can serve as a bridge for large-scale knowledge transfer from 2D foundation models to 3D models. Any 3D/4D task with renderable outputs can potentially leverage 2D supervision. As 2D foundation models continue to improve and scale, 3D models trained using render-superviser distillation are positioned to benefit. This opens the possibility of train- ing 3D understanding models at the scale of 2D datasets- which would represent a fundamental shift from the current paradigm of limited 3D annotations.
This render-supervised formulation offers several key ad- speci- fying only the outputs leaves flexibility in underlying model overcome fundamental
vantages. First, it inherently
design. Second, this allows scaling limitations by training a large transformer decoder instead of previous dual-encoder approaches (as shown in Fig 3) (Zhu et al., 2023b; Gu et al., 2024). Third, the ap- proach is highly practical: LIFT-GS operates directly on raw point clouds from sensors, such as the outputs from SLAM or SfM systems, eliminating the preprocessing and feature fusion required by methods like ConceptFusion (Jatavallab- hula et al., 2023b; Arnaud et al., 2025), reducing inference time from 60 seconds to just second.
To summarize, our contributions are:
A render-supervised training pipeline for 3D vision- language grounding that requires only 2D supervision. We show how diffcrentiable rendering enables training 3D models with 2D losses, eliminating dependence on scarce 3D annotations.
Demonstrating pseudo-labeling strategy for distill- ing 2D foundation models into 3D. LIFT-GS shows using SAM, CLIP, and LLMs generate 2D supervision. State-of-the-art performance in realistic evaluations. LIFT-GS achieves SOTA results using sensor point clouds common in embodied settings, with detailed ablations revealing scaling properties.
Our experiments validate the effectiveness of this approach. LIFT-GS achieves state-of-the-ar performance on standard 3D VLG benchmarks, with 25.7% mAP on open- open-vocabulary instance segmentation (vs. 20.2% previous SOTA) and con- sistent 10-30% relative improvements referential ground- ing tasks. More importantly, we observe that across data scales for SFT. pretraining effectively "multiplies" the fine- tuning dataset by approximately a constant factor (2x). That is, a pretrained model with 50% of fine-tuning data matches the performance of training from scratch with 100% data. This somewhat counterintuitive observation indeed matches empirical data scaling laws for pretraining in other modali- ties (Hernandez et al., 2021), and the fact that this scaling coefficient remains constant without diminishing returns across data scales adds an empirical data point that 3D VLG currently operates in the very low-data regime'!
2. Related Work
2.1. The Data Scarcity Challenge in 3D VI.
3D Vision-Language Grounding (3D VLG) maps language descriptions to corresponding 3D masks or bounding boxes in observed scenes (Yuan et al., 2021; ; Roh et al., 2021; Yang et al., 2021). Despite its fundamental importance for embodied AI, existing 3D VLG datasets contain only thousands of annotated scenes (Dai et al., 2017: Yeshwanth et al., 2023) compared to billions of images used for training large multimodal models (Meta AI, 2024).
The implications extend beyond 3D grounding. Render su-
This scarcity stems from prohibitive annotation costs. Cre- ating 3D instance masks requires minutes per example even with assisted tools (Dai et al., 2017), and human verification remains necessary (Arnaud et al., 2025; Majumdar et al.,
1 (Hernandez et al.. 2021) define a "low-data regime as having 10% or less of the amount of data it would take to get to 99% of the performance that infinite data would yield."
2

---

From Billions to Thousands: 3D Language Grounding via Render- Distillation
2024). All existing 3D VLG methods require ground-truth
The cnar which clss to tse IV"
"Mev chai under riee"
3D masks or bounding boxes during training (Yuan al.,
2021; Zhu et al., 2023c; 2024; Zhang et al., 2024), with many also requiring them during inference (Fang et al.,
2024; Zhang et al., 2023b). This creates a fundamental scal- ing barrier-these approaches cannot leverage the vast 2D data that has driven progress in 2D understanding.
Render View
Th MIS nox the wathe"
2.2. Bridging 2D and 3D: From Lifting to Learning
2D-to-3D Liftting via Optimization. Recent work addresses data scarcity by lifting 2D models to 3D through per-scene optimization. Methods use depth unprojection with heuris- tic merging (e.g., voxel voting) (Jatavallabhula et al., 2023a; Zhou et al., 2024; Xu et al., 2023b) differentiable ren- dering to optimize 3D representations matching 2D fea- tures (Kerr et al., 2023; Kim et al., 2024; Gu et al., 2024) or masks (Cen et al., 2023; Xu et al., 2023a). While these leverage 2D foundation models, they suffer from: (1) slow optimization (minutes per scene), (2) accumulated errors from reconstruction and merging, and (3) inability to im- prove with more data. The fixed lifting pipelines may be bottleneck that explains why LIFT-GS outperforms models trained on these lifted pscudolabels (Genova et al., 2021; Peng et al., 2023).
Figure 3: 3D grounding with CLIP-style (dual-decoder) method. Grounding heatmaps from a representative approach (Guo et al., 2024). Heatmaps are computed using dot product similarity be- tween visual tokens and text tokens (as in the CLIP objective), encoded independently. This performs effectively with very short prompts, but fails with more detailed queries, as shown the image. LIFT-GS addresses this by jointly predicting tokens using a transformer decoder with expressive attention masks (see Figure 6 and experiments).
causes 3D models to inherit these fundamental limitations; as shown in the limited ability of 3D dual-encoder models to handle relational language crucial for referential grounding (e.g.. "the chair next to the table") (Fig. 3).
Multimodal Decoders. Transformer decoders address the bag-of-words behavior by jointly processing the modalities together through learned attention mechanisms that enable proper handling of spatial relationships and multi-object references. Following large multimodal language models (LLaMA 3, GPT-4o, and Qwen 2.5), as well as recent 3D) VLG SotA (Kamath et al., 2021; Jain et al., 2025; Arnaud et al., 2025), LIFT-GS employs decoder-based architecture. LIFT-GS introduces the grounding loss described in Sec. 3.2 to train the decoder.
Render- Supervied Learning. Differentiable rendering en- ables training 3D models directly by rendering their predic- tions into 2D and supervising via 2D losses. While initially used for reconstruction (Hong et al., 2024; Tang et al., 2024; Cao et al., 2024; Szymanowicz et al., 2025). (Irshad et al., 2024; Zhu et al., 2023b) use it to representation learning, like P2 adds CLIP losses on rendered pixels (Zhu et al., 2023a). However, existing 3D VLG methods have significant limitations: PonderV2 only trains encoders and relies on ground-truth category labels, while other meth- ods apply simple photometric losses without leveraging 2D VLMs. LIFT-GS overcomes these challenges by extending render-supervision in two key ways: (1) it enables training unified encoder-ddeooer architectures for 3D VLG tasks, and (2) it performs knowledge distillation from 2D founda- tion models (VLMs, SAM, CLIP) through pseudo-labeling removing the need for 3D annotations during pretraining.
2.4. Foundation Model Distillation at Scale
Recent work on scaling up pseudolabeling pipelines shows that although 2D foundation models are increasingly capa- ble (Hong et al., 2023; Arnaud et al., 2025), they currently exhibit significant limitations in spatial understanding over multiple frames, frequently hallucinating 3D spatial rela- tionships (Majumdar et al., 2024; Yang et al., 2024). This is why LIFT-GS uses pseudolabels for pretraining: just as LLMs require supervised fine-tuning (SFT) to align noisy in- ternet text with desired behaviors, LIFT-GS leverages noisy pseudolabels for large-scale pretraining, then uses 3D VLG SFT for state-of-the-arr performance.
2.3. Architecture: From Dot-Products to Joint Attention
Limitations of Dual-Encoder Methods. Almost all prior 3D grounding approaches that distill from 2D vision- language models eventually compute the mask as the dot- product similarity between 3D features and text embed- dings (Radford et al., 2021: Guo et al., 2024; Qin et al., 2023; Kerr et al., 2023; Peng et al., 2023). When modality embeddings are computed independently, as is the case in CLIP, these "dual-encoder" models behave as bag-of systems (Yuksckgonul et al., 2022). Using these encoders
Our scaling analysis reveals suggests that even imperfect 2D spatial understanding generates meaningful training signal (multiplying fine-tuning data effectiveness by 2x in our experiments). As 2D models advance in spatial reasoning, this transfer benefit should amplify, potentially reducing reliance on 3D annotations and moving toward more zero- shot spatial understanding.
3

---

From Billions to Thousands: 3D Language Grounding via Render-Supervised Distillation
Imput Image
SAM-CLIP Feature
This dual design elegantly handles two key challenges: (1) text tokens can refer multiple instances (e.g., "the chairs"), and (2) instances can be referenced multiple times with different descriptions throughout the text (e.g. both "chair" and "it"). It also extends readily to segmenting other modalities by adding more mappings.
SAM Mask
SAM
2
Seg
Seg
CLIP
Sig 21
Seg
CLIP Embedding
The pointeloud has |PI points and is of shape |P| x (XYZ RGB), and the query token embeddings and are matrix of size: Q E R FQ. The text mapping C and Gaus- sian/point cloud mapping M are shown, with each instances highlighted in a different color in Figure 2.
Figure SAM-CLIP Pseudo-Label Generation. We lever- age powerful 2D foundation models to generate pxeudo language queries, i.c., CLIP embeddings, along with their corresponding ground-truth 2D masks for training. All pixels within the same mask share the same features.
3.1.2. 3D VLG with Gaussian Masky M
This positions our approach at the intersection of three key to learning-
trends: (I) the shift from optimization- based 3D understanding, (2) the move from dual-encoder to multimodal decoder architectures for complex language grounding, and (3) the emergence of foundation model dis- experiments
Existing 3D VLG methods are limited by costly point cloud annotation, where annotation for point cloud masks is by far the costliest and slowest step in 3D VLG data collection. This is because human annotators must carefullly segment the mask using brush or assisted tool, which takes on the order of minutes per example (Dai et al.. 2017). Even with model assistance, human verification is required in practice (Arnaud et al., 2025; Majumdar et al., 2024). This restricts training to only thousands of scenes, making data scarcity the key bottleneck.
tillation as a solution to 3D data scarcity. Our exueriieemeee demonstrate that this approach not only achieves state-of- the-art performance but also exhibits strong scaling proper- ties, suggesting significant potential for using cross-scene render-superviie distillation with improved multimodal foundation models and in other settings in besides 3D VLG.
LIFT-GS addresses this by leveraging differentiable render- ing, distilling knowledge from multimodal vision-language models trained on billions of images. It predicts the 3D Gaussians from the point cloud input ferd-forwardly, which are used for rendering later. 3D Gaussians are represented with xyz locations TR. Nx3 covariance matrices R-Vx6 color matrices R and feature embeddings xx LIFT-GS predicts G R Nx containing m masks plus 12 channels for shape/locationncolor and 512 for feature loss:
3. Method
LIFT-GS is a (pre-)training pipeline for 3D VLG without 3D GT annotation. During training, it renders the predicted 3D masks from the target viewpoints for 2D supervision; during testing, 3D masks are used as outputs. This section covers the 3D VLG formalism in Sec.3.1, 2D loss used for supervision in Sec.3.2, pseudolabel generation in Sec.3.4, and implementation details in Sec.3.3.
LIFT-GS: (P.Q)= - (G,M, C)2
(2)
3.1. Task Formulation
In practice, training the masks using differentiable render- ing adds only small overhead during training. With the shapes of the inputs and outputs specified, the sections be- low describe each component: the losses used, pseudolabel generation, and the model architecture.
3. 3D Vision-Language Grounding
Figure 2 shows an example using a point cloud input (P)
and the text query (Q) "the black chair close to the table
near the wall". Following MDETR (Kamath et al., 2021)
and (Jain et al., 2025), LIFT-GS outputs a set of (m 256)
3D mask candidates M, and the correspondence matrix C.
3.2. Losses
3D VLG: (P,Q) + (M,C)
(1)
With differentiable rendering, LIFG-GS enables training
3D VLG models using the simplest 2D grounding losses. During training. LIFT-GS only requires (sparse) point cloud,
Matrix C E R= indicates the correspondence (i.e.
probability logits) between m 3D mask candidates and |Q]
2D posed images as inputs without any other 3D annotations, which data can be easily obtained from RGB-D videos or SIM (Yang et al., 2025; Wang et al., 2025a; 2023; 2025b; Leroy et al., 2024) as done in (Szymanowicz et al., 2025).
text tokens, enabling the mapping of each text query to its most probable mask candidate based on the logits.
3D Mask M E Rm stores mask logits over N Gaussian primitives for m 3D mask candidates. Each logit M. represents the probability (after applying sigmoid) that the j-
LIFT-GS utilizes two groups of losses to train the model: Eyound for grounding (Jain et al., 2025), and per-pixel losses Lep commonly used to improve results with differentiable
th Gaussian primitive is included in the i-th mask candidate.
4

---

From Billions to Thousands: 3D Language Grounding via Render-Supervised Distillation
Visual
Dot Product
Gaussian Head
Token
Encoder Backbone
Mask Decoder
Transformer
Language Query
Embedding
GS with G Predicted Masks M
Point Cloud P
Language
Token
Mask Proposal Token
Figure Architecture Design. LIFT-GS predicts Gaussian Splatting 3D masks M given a point cloud and language query embeddings Q as inputs. The 3D masks M are generated by a Transformer-baan Mask Decoder.
rendering. For reference, rendered 2D masks, rgh im- ages, and feature maps are denoted as M20 Hx xm, F'D H WxF,, and I E Hxxxxx respectively. Their corresponding ground-trum truth counterparts have analo- gous shapes: MaD, F2D, and I, where Mæ Hx
Optimal matching, m this function measures the pair- wise distance between the 3D grounding results and the ground-truth values, and is used for matching. imple- mented similarly to Eground but with different loss weights.
The model always predicts the maximum number of in- stances (256) but avoids false positive detections by match- ing the unused instance to special no-mate ch text to- ken (Jain al., 2021). The maximum of 256 was not major limitation in practice, but could be increased.
3.2.1. Grounding losses:
LIFT-GS uses the MDETR-Styl mask grounding loss: Las and Lmask with dmaach (matching distance). Since the num- ber of predicted and ground-truft trunt masks may differ, we apply Hungarian matching to pair them based on clmanch, and then optimize the matched predictions using Lground- We use the 2D variant of these losses under 2D rendering super- vision, which can be easily replaced by their 3D counterpart (on 3D Gaussian centers) when 3D labels are available.
3.2.2. Per-Pixel Losses
While grounding loss alone is sufficient for stable pretrain- ing (Table 4), LIFT-GS benefits from joint training with additional photometric and feature losses for faster conver- gence. These losses are only used during pretraining, not for finetuning with 3D annotations.
Cground Σ DoLmua (M MíD) +heee (3)
Reconstruction loss, LRGB: supervises photometric recon- struction using L1 and SSIM losses (Hong et al., 2024: Zhu et al., 2023b; Wang et al., 2004)
(11) = arg min clmaann (M, M. C)
(4)
Cross Entropy Loss, Lex this loss supervises the corre- spondence between 3D mask candidates and input language tokens (i.e., the matrix C) by framing it as a (soft) classifi- cation problem. Recent work suggests that using BCE may produce sharper masks for long VLG text queries (Jain et al., 2025; Arnaud al., 2025; Zhai al., 2023), echoing similar findings in image segmentation (Cheng et al., 2021a).
LRGB = A1 A2CssIN(L,
(6)
Feature loss, Lteat' uses CLIP-style contrastive regular- ization to align rendered features FAD with ground-truth features FED as in (Zhu et al., 2023b):
(
(CE(Coto),) i) log K ΣK (xp((C(())))
(5)
H.W Σ U2P
exp(f ((,)) .ff)
Ltear
log
H W
Σ, exp(f (f f,)
Mask loss, Lmask: this loss supervises the predicted 3D masks by comparing them to paired ground-truth 3D masks. Following SAM (Kirillov et al., 2023), we apply a combina- tion Focal (Lin et al., 2017) and Dice (Sudre et al., 2017) losses to supervise the predicted 3D masks effectively.
where ((,,)) is the rendered feature at pixel (u, v) from EAD. fy is the corresponding ground-truth feature, and f,
represents the batch of unique features.
5

---

From Billions to Thousands: 3D Language Grounding via Render-Supervised Distillation
3.3. Architecture
"White"
LIFT-GS is network-agnostic with minimal architectural constraints. The design can be applied to other architectures, with constraints only arising from the specific losses used. This section describes the network used in our experiments, shown in Figure 5.
"Black oabinet near wall"
3. Grounding Decoder: (see Grounding Losses)
The grounding decoder is a transformer based on Mask- Former (Cheng et al., 2021b) that predicts the correspon- dence matrix C m and 3D Gaussian masks M E R xm LIFT-GS uses the mask decoder from Uni- VLG (Jain et al., 2025) with minimal modifications.
Figure 6: Zero-Shot 3D Segmentation. Trained using only 2D pscudo-labels, LIFT-GS can localize ohjects in 3D from real text inputs in zero-shot manner. From left to right, we visualize the input point clouds, segmented 3D masks(in yellow), rendered images from predicted 3DGS. and rendered segmentation masks. Language queries include both high-level abstract concepts (e.g.. white) and detailed descriptions (e.g., black cubinet near the wall).
The transformer takes Gaussian features G and language query embeddings Q as inputs, using a set of learnable to- kens as 3D mask proposals. Cross-attention is computed between the learnable tokens and both the language and Gaussian tokens. After extensive information exchange within the transformer, M (or C) is computed as dot prod- ucts between mask proposal tokens and Gaussian tokens (or mask proposal tokens and language tokens, respectively). The detailed LIFT-GS architecture and hyperparameter set- tings used in our experiments are provided in Appendix
Table I: Open- Vocabulary 3D Instance Segmentation. We evaluate model on ScanNet200 by using category names text queries and compare against SOTA models.
Model
MAPt
mAP251
mAP501
OpenScene (Peng et al., 2023)
11.7
17.8
15.2
OpenMusk3D (Takmaz al., 2023)
15.4
23.1
19.9
PQ3D (Zhu et al., 2024)
20.2
32.5
28.0
LIFT-GS- Seratcchhhh LIFT-GS
22.5
35.1
30.7
25.7
40.2
35.0
3.3.2. Gaussian Decoder Head: (see Per-Pixel Losses)
+3.2 1
+55.1
+4.3
LIFT-GS predicts G E NxF using a learned pointwise MLP applied to the point cloud encoder outputs. While the number of predicted Gaussians |G] can differ from input points [P), we set |G| PP with a bijective mapping for consistency with point cloud evaluation tasks. Notably, the Gaussian decoder does not require direct supervision-3D Gaussians can be treated as latent variables for 3D VLG, as shown in Table when per-pixel losses are disabled.
erate pseudo-labels that enable reasonable zero-shot perfor- mance and significantly enhance downstream fine-tuning.
As shown in Figure 4, we generate pseudo-labels using SAM (Kirillov et al., 2023) and CLIP (Radford et al., 2021). For each image, SAM provides segmentation masks, and for each segmented region, we extract CLIP image embed- dings as pseudo language query embeddings. Since CLIPP' text and image embeddings share the same feature space, LIFT-GS can use text embeddings during inference. We con- catenate these CLIP embeddings to form Q and construct C s.t. each instance maps to exactly one query token.
3.3.3. Input Encoders:
Pointcloud Encoder: Tokenizes RGB point clouds of shape x6 (xyz + RGB) using a sparse convolutional UNet (Contributors, 2022), following PonderV2 (Zhu et al., 2023b). Weights are randomly initialized and learned.
With 2D pseudo-labels, LIFT-GS performs zero-shot 3D grounding using real text queries without fine-tuning (Fig- ure 6). However, zero-shot performance suffers from low ac- curacy and struggles with complex expressions a common limitation of CLIP-based methods that function as bag-of- words models (Yuksekgonul et al., 2023). Future improve- ments in pseudo-labeling, such as better captioning (Meta AI, 2024) and 2D language grounding models (Liu et al., 2023), could reduce reliance on fine-tuning.
Text Encoder: LIFT-GS uses CLIP text embeddings, which remain frozen during training.
These encoders represent common choices in 3D VLG. Stronger architectural choices like transformer-base point- cloud encoders or different text embeddings would likely improve performance.
LIFT-GS demonstrates that even simple pseudo- labeei-ga strategies can be effectively distilled into 3D models. In the experiments below, pretraining with 2D pseudo-labels substantially improves downstream task performance. Fine- tuning data scaling results are consistent with established transfer learning "scaling laws" (Hernandez et al., 2021).
3.4. SAM-CLIP 2D Pseudo-Label
While LIFT-GS eliminates the need for 3D annotations, ob- taining high-quality 2D supervision remains challenging. We show one way which 2D foundation models can gen-
6

---

From Billions to Thousands: 3D Language Grounding via Render-Supervised Distillation
4. Experiments
4.2.1, GROUNDING SIMPLE NOUNS IN 3D
Although supervised via 2D loss, LIFT-GS is a fully 3D model that explicitly outputs 3D masks. Beyond the zero- shot setting using 2D pseudo-labels (Figure 6), LIFT-GS can be readily fine-tuned with 3D annotation data using 3D losses for significantly improved performance. To enhance practical applicability, we focus on fine-tuning with 3D an- notations and demonstrate how 2D model distillation boosts performance.
We first evaluate simple grounding for simple noun-phrases, using object categories without spatial relationships. Fol- lowing the protocol in (Zhu et al., 2024), we convert the standard 3D instance segmentation benchmark on ScanNet into an open-vocabulary 3D instance segmentation task. The categories of objects are used as language queries, which are input to the model to predict the corresponding 3D masks.
Evaluation setting: We evaluate using the standard metric mAP, measure of mask overlap averaged across categories. We fine-tune LIFT-GS for 500 epochs.
In this section, we first provide training details and show how pretraining significantly improves downstream task performance through series of carefully designed ablations. We also reveal several insights from scaling the amount of pretraining and fine-tuning data, and explore the impact of different 2D foundation models.
baselines PQ3D (Zhu et al., 2024) and OpenMask3ID (Takmaz et al., 2023), our pretrained model (LIFT-GS) achieves substantial performance gains (mAP 25.7% vs 20.2%), as shown in Table 1. It significantly outperforms its counterpart trained from scratch (LIFT-GSS-Srrtch mAP +3.2%%
Results: Compared against the
4.1. Training Details
We provide details below with more details in the Appendix.
4.2.2. GROUNDING COMPLEX PHRASES IN 31)
Datasets We use ScanNet (Dai et al., 2017) as the primary dataset for downstream task fine-tuning and evaluation, as its annotations form the basis for established benchmarks. We primarily pretrain using ScanNet (Dai et al., 2017) for comparison to other methods. LIFT-GS enables training diverse unlabeled 3D datasets, and we show additional pre- training scaling experiments using ScanNet++ ScanNet-<(eshwanth (Yeshwanth et al., 2023). Taskonomy (Zamir et al., 2018) and Aria Syn- thetic (Somasundaram et al., 2023).
Next, we examine grounding multiple objects using more complex phrases that contain spatial references, referred to as 3D Referential Grounding (3D RG).
Evaluation Setting. We evaluate LIFT-GS on the most common 31) Reserential Grounding benchmarks: SeanRe- fer (Chen et al., 2019), SR3D, and NR3D (Achlioptas et al., 2020; Abdelreheem et al., 2022). We use standard top-1 I accuracy as the evaluation metric, considering a predicted bounding box correct if its IoU with the ground truth ex- ceeds 0.25 or 0.5. Since LIFT-GS outputs masks instead of axis-aligned bounding boxes, we derive bounding boxes by extracting the extreme corner points from the point cloud within the predicted masks.
Architecture LIFT-GS method imposes minimal architec- tural constraints. In the following experiments, backbones consist of text encoder (frozen CLIP-L), point cloud en- coder (Sparse 3D UNet (Çişek et al., 2016)), and grounding decoder (8-layer Transformer, hidden size 512 following (Jain et al., 2025)), and an MLP for the Gaussian decoder.
LIFT-GS is designed to be practical and we evaluate it using the "real-world" settings used in more recent 3D VLG work where (1) we predict 3D masks without assuming known ground-truth 3D bounding boxes, and (2) we utilize sensor point clouds (Sensor PC) from RGB-D scans instead of using mesh-derived point clouds that leak label informa- tion (Mesh PC). This realistic setting is more challenging, as reflected in the significant performance drop of BUTD)- DETR (Jain et al., 2021) when transitioning from Mesh PC to Sensor PC (Table 2), consistent with findings in (Jain et al., 2024). A more complete comparison of these settings is provided in (Jain al., 2021; 2025; Arnaud et al., 2025).
Training Models are trained end-to-end for 76k steps with batch size 32 on 32 A100s using AdamW (Loshchilov & Hutter, 2017) (Ir=1e-4, weight decay=le-4). Point clouds are voxel voxel-downsampled to 5cm for an average of 50k points, and we render masks at resolution 512x512 to ensure small masks are captured. Complete implementation details are provided in the appendix.
4.2. Evaluation on 3D Vision-Language Grounding
We fine-tune and evaluate our pretrained model on two rep- resentative 3D VLG tasks: 3D open-vocabulary instance segmentation and 3D referential grounding. Our results, shown in Tables 1 and 2, demonstrate significant improve- ments over models trained from scratch and achieve state- of-the-art performance with pertaining.
Baselines We compare LIFT-GS against the state-of-the oftte- art two-stage methods, 3D- VisTA (7huu et al., 2023c) and PQ3D (Zhu et al., 2024), as well as the SOTA single-stage method, BUTD-DETR (Jain et al., 2021). All two-stage basclines assume access to ground-truth 3D masks or boxes during inference, so we re-evaluate them using predicted
7

---

From Billions to Thousands: 3D Language Grounding via Render-Supervised Distillation
Table 2: 3D Referential Grounding. We report top-1 accuracy with various IoU thresholds (0.25, 0.5).
SR3D
NR3D
ScanRefer
Method
Acc
Acc@50 50
2cc@
Acc@
Acc@:2
55
Mesh PC
LanguageRefer (Roh et al., 2021)
39.5
28.6
SAT-2D (Yang et al., 2021)
35.4
31.7
44.5
30.1
BUTD-DETR (Jain et al.. 2021)
52.1
43.3
52.2
39.8
3D-VisTA (Zhu al., 2023c)
56.5
51.5
47.7
42.2
51.0
46.2
PQ3D (Zhu et al., 2024)
62.0
55.9
52.2
45.0
56.7
51.8
Sensor PC + Bounding Box Proposals using Mesh PC
3D-VisTA (Zhu et al., 2023c)
47.2
43.2
42.1
37.4
46.4
42.5
Sensor PC
BUTD-DETR (Jain al., 2021)
43.3
28.9
32.2
19.4
42.2
27.9
LIFT-GS-Scratch
44.0
28.8
37.2
23.
45.0
29.5
LIFT-GS
50.9
36.5
43.7
29.7
49.7
36.4
+6.9(16%)
+7.7(27%)
+6.5(17%)
+6.6(29%)
+4.7(10%)
+6.9(23%)
Table Comparison with other Pretraining Baseline. LIFT-GS clearly outperforms Ponder-v2 and its variant Ponder-v2f, which is trained on the same SAM-CLIP CLIP features as ours,
Table 4: Loss Ablation. show impact of different pretrain- ing losses on 3D referential grounding task. Lgound significantly improves results, particularly at high IoU thresholds.
Model
Acc00.25
Acc
Acc@0.75
Model
Egmind
ERGB
Eior
Acc100.25
Acc
Acc
Scratch
42.19
27.23
9.66
Scratch
42.19
27.23
9.66
46.34
31.54
12.50
Ponder-v2 (official)
40.92
25.97
8.84
46.67
31.81
12.45
Ponder-v21
45.40
29.36
9.29
47.69
31.35
11.36
LIFT-GS
47.53
33.75
13.49
47.53
33.75
13.49
against PonderV2 (Zhu et al., 2023b), a state-of-the-art point cloud pretraining method that also uses render- supervision. Since the official PonderV2 relies on limited human-annotated text labels, retrain it using our SAM- CLIP pseudo-labels for fair comparison (PonderV2f f in Ta- ble 3). This demonstrates the value of distillation, improv- ing the performance over using GT labels (45.4% vs 40.9% Acc@0.25). Moreover, LIFT-GS substantially outperforms Vo (47.5% vs 45.4% Acc@0.25) underscoring the impact of multimodal decoder architectures enabled by the LIFT-GS render-supervised formulation.
boxes from the SOTA object detector Mask3D (Schult et al., 2022). For fairness, re-train 3D-VisTTA and BUTD- DETR on sensor point clouds. Because PQ3D uses multiple backbones and a multi-stage training pipeline, we were not able to reproduce PQ3D on the sensor point cloud setting,
Results Our model without pretraining LLIFT-GS-Serateel achieves slightly better performance than the state-of-the-art single-stage method BUTD-DETR (Jain al., 2021), likely due to architectural similarities with extra modifications.
Loss Ablation Existing pretraining pipelines primarily fo- cus the encoder (Zhu et al., 2023b; Banani et al., 2021), whereas the render-supervised formulation can pretrain the entire architecture in a unified manner using the ground- ing loss. We find that grounding loss alone can be used to pretrain the model end-to-end in Table 4. model trained with Egrand alone (row 2) substantially improves down- stream task performance, performing only slightly worse than the model trained with all losses (row 5). Further- more, comparing models with and without Lground (row 5 vs. row 4) clearly shows that Legonnd significantly enhances downstream performance, particularly in more challenging scenarios (loU thresholds of 0.5 and 0.75).
With pretraining. LIFT-GS achieves significant improve- ments across all three datasets, with relative gains of 10% 30%, demonstrating the effectiveness of our pretrain- ing approach. Notably, LIFT-GS outperforms 3D- Ti in Acc@25, despite 3D- VisTA being two-stage method with bounding box proposals from Mask3D using Mesh PC.
4.3. Pretraining Ablations
We conduct an in-depth analysis of the proposed method through series of ablation and scaling experiments. For these evaluations, we model pretrained only Scan- Net the baseline. To simplify the presentation for the ablations, we report results on the combined evaluation set of ScanRefer, SR3D, and NR3D. Additionally, we report the higher accuracy threshold Aco@0.75.
4.4. Data Scaling
LIFT-GS exhibits strong scaling properties that reveal 3D VL. operates in severely data-scarce regime.
Compare to SOTA pretraining methods We compare
8

---

From Billions to Thousands: 3D Language Grounding via Render-Supervised Distillation
Table Fine-tune Data Scaling, We show Acc@0.5 results with different ratio of tine-tuning data on referential grounding task.
Table 6: Pretraining on OOD data. Adding more pretraining data from ScanNet++ improves performance. Taskonomy and Arial helped less than ScanNet++, likely due to distribution difference.
Finetuning Data Ratio
10%
20%
50%
100%
Pretraining Data
Acc00.25
Acc@0.5
Acc80.75
Scratch
6.93
15.04
23.00
27.23
LIFT-GS
14.70
23.03
28.89
33.75
Scannet
47.53
33.75
13.49
+Scannet++
48.29
34.35
14.06
++ Taskonomy and Arial
48.49
34,41
14.35
Accge0.25
Acci0.5
Accg00.75
12-
Table 7: 2D Foundation Model Exploration.
40
35
25
1--
30
20
2D Models
A.c@0.25
Acc@0.5
Acc@0.75
Scratch
Scratch
SAM-B + CLIP-B
46.31
31.50
12.41
15
2.2
0
1.0
0.5
1.5
SAM-H + CLIP-L
47.53
33.75
13.49
SAM-HI + LLAMML++LLAACppppii
47,50
32.78
13.25
Figure 7: Fine-tune Data Scaling, We show how Grounding Accuracy changes with increasing Data Ratio from 0.1 to 1.0.
4.5. 2D Foundation Models Scaling and Exploration
Finetuning Data Scaling We observe that pretraining effec- tively "multiplies" the fine-tuning dataset by approximately 2x. As shown in Figure 7 and Table 5, a pretrained model using 50% of fine-tuning data matches the performance of training from seratch with 100% data. This scaling co- efficient remains constant across different data amounts (10%, 20%, and 50%) without diminishing-mate mattinnng em- pirical scaling laws from other modalities (Hermandez et al., 2021). The benefits are most pronounced at higher IoU thresholds, where a pretrained model achieves seratch-level performance using only 30-40% of the fine-tuning data. Additional results on Instance Segmentation are provided in Appendix A.3.
Our pipeline leverages powerful 2D foundation models to generate pseudo-labels, Here, we investigate their impact by analyzing performance variations with different 2D foun- dation models, with results presented in Table 7.
Weaker CLIP and SAM The main experiments use SAM- H and CLIP-L for pseudo-labeling. Replacing them with smaller models, MobileSAM (Zhang et al., 2ViT tiny) and CLIP-B, leads to a noticeable performance drop, especially at higher accuracy thresholds. This suggests that render-supervised distillation directly benefit from advance- ments in 2D foundation models.
Pretraining Data Scaling Expanding pretraining data consistently improves downstream performance (Table 6). Adding ScanNet++ yields notable gains (+0.8%), while incorporating Taskonomy and Aria Synthetic provides addi- tional improvements despite distribution differences (likely due to the mesh reconstruction quality in Taskonomy).
Captions from LMMs Table 7 shows results using large LMMs instead of CLIP to generate queries (LLAMA-3V + SAM grounding, details in Appendix). After segmenting objects in 2D images using SAM, we prompt LLAMA-V to describe the segmented regions. Pretraining with these captions achieves performance comparable to our original pipeline with SAM-H and CLIP-L. As LMMs continue to improve, we believe text-based captions hold significant potential for future research, and the approach highlighted in the paper is positioned to benefit from LMM improvements.
Comparing the scaling results from pretraining and fine- tuning demonstrates the strong data efficiency of LIFT-GS. As shown in Table 6, adding ScanNet++ ( (30% of the data used for pretraining) yields a performance gain equivalent to adding 15% more ScanNet fine-tuning data with 3D an- notations, based on the curve in Figure 7. This indicates an effective transfer ratio of roughly 1:2-ie., collecting twice as many raw videos provided improvements comparable to building a fully annotated 3D SFT dataset.
5. Conclusion
LIFT-GS tackles data scarcity that limits 3D VLG by intro- ducing render-supervised distillation from 2D VLM models. By training 3D models using only 2D supervision from mod- els like SAM and CLIP, LIFT-GS achieves state-of-the-art performance for 3D VLG. Our findings, including consistent 2x data multiplication effects, reveal that 3D grounding cur- rently operates with substantial data limitations. LIFT-GS circumvents key 3D annotation bottleneck by introduc- ing scalable training approach that benefits directly from advancements in frontier multimodal language models. It offers a practical technique to leverage progress in 2D to accelerate the development of other data-scarce capabilities essential for robotics, AR/VR, and embodied AI.
Therefore, pretraining on ScanNet++ is not only highly ef- fective but also cost-efficient, especially considering that annotating 3D referential grounding data requires signifi- cantly more effort than collecting raw videos alone.
Data Scarcity Implications The consistent 2x multiplier without saturation, combined with continued gains from more pretraining data, strongly suggests that current 3D VLG models are severely limited by data availability- opening a path for future 3D VLG improvements through scaling alternate sources (such as 2D foundation models).
9

---

From Billions to Thousands: 3D Language Grounding via Render-Supervised Distillation
Acknowledgment
R., Jin, H., Jin, D., Jomoto, S., Jonn, B., Jun H., Kaf- tan, T., Kaiser, L.,, Kamali, A., Kanitscheider, I., Keskar, N. S., Khan, T., Kilpatrick, L.,, Kim, J. W., Kim, C., Kim, Y., Kirchner, H.. Kiros, J. R., Knight, M., Kokotajlo, D., Kondraciuk, I.., Kondrich, A., Konstantinidis, A., Kosic, K., Krueger, G. Kuo, V., Lampe, M., Lan, I., Lee, T., Leike, J., Leung, J., Levy, D., Li, C. M., Lim, R., Lin, M., Lin, S., teusz Litwin, M., Lopez, T., Lowe, R., Lue, P., Makanju, A., Malfacini, K., Manning. S., Markov, T., Markovski, Y., Martin, B., Mayer, K., Mayne, A., Mc- Grew, B., McKinney, S. M., Mcl.avey, C., McMillan, P., McNeil, J., Medina, D., Mchta, A., Menick, J., Metz, L., Mishchenko, A., Mishkin, P., Monaco, V., Morikawa, E., Mossing, D. P., Mu, T., Murati, M., Murk, O., M'ely, D., Nair, A., Nakano, R., Nayak, R., Neelakantan, A.. Ngo, R., Noh, H., Long, O., Keefe, C.,, Pachocki, J. W.. Paino, A., Palermo, J., Pantuliano, A., Parascandolo, G., Parish, J., Parparita, E., Passos, A., Pavlov, M., Peng, A., Perelman, A., de Avila Belbute Peres, F., Petrov, M., de Oliveira Pinto, H. P., Pokomn, M., Pokrass, M., Pong, V. H., Powell, T., Power, A., Power, B., Prochl, E., Puri, R., Radford, A., Rae, J. W., Ramesh, A., Raymond, C., Real, F,, Rimbach, K., Ross, C., Rotsted, B., Roussez, H., Ryder, N., Saltarelli, M. D., Sanders, T., Santurkar, S., Sastry, G., Schmidt, H., Schmurr, D., Schulman, Sel- sam, D., Sheppard, K. Sherbakov, T., Shieh, J., Shoker, S., Shyam, P., Sidor, S., Sigler, E., Simens, M., Sitkin, J., Slama, K., Sohl, I., Sokolowsky, B., Song. Y., Stau- dacher, N., Such, F. P., Summers, N., Sutskever, I., Tang, J., Tezak, N. A., Thompson, M., Tillet, P., Tootoonchian, A., Tseng, E., Tuggle, P., Turley, N., Tworek, J., Uribe, J. E. C., Vallone, A., Vijayvergiya, A., Voss, C., Wainwright, C. L., Wang, J. J., Wang, A., Wang, B., Ward, J., Wei, J., Weinmann, C., Welihinda, A., Welinder, P., Weng. J., Weng, L., Wiethoff, M., Willner, D., Winter, C., Wolrich, S., Wong, H., Workman, L., Wu, S., Wu, J., Wu, M., Xiao, K., Xu, T., Yoo, S., Yu, K., ing Yuan, Q., Zaremba, W.. Zellers, R., Zhang, C., Zhang, M., Zhao, S., Zheng. T., Zhuang, J., Zhuk, and Zoph. B. Gpt-4 technical report. 2023. URL https: /tt::///ppppp semant innmanticscnn cscholar. org/ Stsg/ooopuIID:22533000
This work is done during Ang Cao's internship at Meta. We thank Andrew Owens, Andrea Vedaldi, Stella Yu, Ziyang Chen, Yiming Dou, Xuanchen Lu for their helpful discus- sion and feedback.
Impact Statement
We propose a method for training 3D models without 3D supervision, advancing 3D vision-language research. Our approach significantly improves 3D referential grounding, key task for robotics, embodied AI, and AR/VR applications.
The resulting model enables agents to precisely localize objects from language inputs, bridging high-level reasoning with real-world actions.
As our model is distilled from 2D foundation models, it may inherit their biases. However, since our primary task grounding, it is unlikely to introduce significant aesthetic biases.
References
Abdelreheem, A., Olszewski, K., Lee, H.-Y., Wonka, P. and Achlioptas, P. Scanents3d: Exploiting parase-to-3d-objed correspondences for improved visio-linguistic models in 3d scenes. 2024 IEEE/CVF Winter Conference on Applications of Computer Vi- sion (WACV), pp. 3512-3522, 2022. URL https: //api. semant 254591182.
CorpusID:
Achiam, O. Adler, S., Agarwal, S., Ahmad, L., Akkaya, I, Aleman, F. I.., Almeida, D.,, Altenschmidt, J., Alt- man, S., Anadkat, S., Avila, R., Babuschkin, I., Balaji, S., Balcom, V. Baltescu, P., ing Bao, H., Bavarian, M., Belgum, J., Bello, I., Berdine, J., Bernadett-Shapirg G., Berner, C., Bogdonoff, L., Boiko, O., laine Boyd, M., Brakman, A., Brockman, G., Brooks, T., Brundage, M.. Button, K., Cai, T., Campbell, R., Cann, A., Carey, B., Carlson, C., Carmichael, R., Chan, B., Chang, C., Chantzis, F. Chen, D., Chen, S., Chen, R., Chen, J., Chen, M., Chess, B., Cho, C., Chu, C., Chung, H. W., Cummings, D., Currier, J., Dai, Y., Decareaux, C., Degry, T., Deutsch, N., Deville, D., Dhar, A., Dohan, D., Dowl- ing, S., Dunning, S., Ecoffet, A., Eleti, A., Eloundou, T., Farhi, D... Fedus, L., Felix, N., Fishman, S. P., Forte, J., abella Fulford, I.. Gao, L., Georges, E., Gibson, C., Goel, V., Gogineni, T., Goh, G., Lopee R., Gordon, J.,
Achlioptas, P., Abdelrcheem. A., Xia, F. Elhoseiny, M., and Guibas, L. J. Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. In European Conference on Computer Vision, 2020. URL https: api. www.ntiismco...o..... icscholar. org/ /ttp:/CooDDD:2212388B000
Arnaud, S., Mevay, P., Martin, A., Majumdar, A., Jataval- labhula, K. M., Thomas, P., Partsey, R., Dugas, D., Gejji, A., Sax, A., Berges, V.PP., Henaff, M., Jain, A., Cao, A., Prasad, I., Kalakrishnan, M., Rabbat, M., Ballas, N., Assran, M., Maksymets, O.. Rajeswaran, A., and Meier, F. Locate 3d: Real-world object localization via
Grafstein, M., Gray, S., Greene, R., Gross, J., Gu, S. S., Guo, Y., Hallacy, C., Han, J.. Harris, J., He, Y., Heaton, M., hannes Heidecke, J., Hesse, C., Hickey, A., Hickey, W., Hoeschele, P., Houghton, B., Hsu, K., Hu, S., Hu, X., Huizinga, J., Jain, S., Jain, S., Jang, J., Jiang, A., Jiang,
10

---

From Billions to Thousands: 3D Language Grounding via Render-Supervised Distillation
self-supervised learning in 3d. ArXiv, abs/2504.14151,
Dai, A., Chang, A. X., Savva, M., Halber, M., Funkhouser, T., and Nieβner, M. Scannet: Richly-annotated 3d recon- structions of indoor scenes. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 2017.
2025. URL httpps:/app...emmniiccchhooo semanticscholar. org/CorpusII: :27
Banani, M. E., Gao, L., and Johnson, J. Unsupervisedr&r: Unsupervised point cloud registration via differentiable rendering. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7125-7135, 2021. URL https///ap..eemntiiicc.o semanticscholar: org/Corpus ID: 232014069.
Fang, J., Tan, X., Lin, S., Vasiljevie, I., Guizilini, V. C., Mei, H., Ambrus, R., Shakhnarovich, G., and Walter, M. R. Transcrib3d: 3d referring ex- pression resolution through large language models. 2024 JEEE/RSI International Conference on Intel- ligent Robots and Systems (IROS), pp. 9737-9744, 2024. URL httpps//api.smmmattt wet.sssssttttsool..... org/CorpusTD:200 :694557177
/
Cao, A., Johnson, J., Vedaldi, A., and Novotný, D. Lightplane: Highly-scalable components for neural 3d fields. ArXiv, 1b7/2404.19760, 2024. URL https: //api. www.sttsseannnnooeeeee.e wtt.:/prrr..r/oooussssis. 269456986.
Genova, K., Yin, X., Kundu, A., Pantofaru, C., Cole, F., Sud, A., Brewington, B., Shucker, B., and Funkhouser, T. Learning 3d semantic segmentation with only 2d image supervision. In 2021 International Conference on 3D Vi- sion (3DV), pp. 361-372, 2021. doi: 10.1109/3DV5399.
Çişek, O., Abdulkadir, A., Lienkamp, S. S., Brox, T., and Ronneherger, O. 3d u-net: Learning dense volu- metric segmentation from sparse annotation. In Med- ical Image Computing and Computer-Asiiste Inter- vention MICCAl 2016: 19th International Confer- ence, Athens, Greece, October 17-21, 2016, Proceed- ings, Part II, pp. 424-432, Berlin, Heidelberg, 2016. Springer- Verlag. ISBN 978-3-319-46722-11
Gu, Q. Lv, Z., Frost, D., Green, S., Straub, J., and Sweeney, C. Egolifter: Open-world 3d segmentation for egocentric perception. arXiv preprint arXiv:2403. / 18, 2024.
doi: 10. URL https:://do 420-99-66778-88
Guo, J., Ma, X.. Fan, Y.. Liu, H., and Li, Q. Seman- tic gaussians: Open-vocabulary scene understanding with 3d gaussian splatting, ArXiv, abs/2403.15624, 2024. URL https: /tpi. www.nticscoo.oo.o.. org/ htt//CrrpsIDD:: 2688844
1007/978-3-319-467222 org/10. org/10.1007/978-
Cen, J., Zhou, Z., Fang, J. Shen, W., Xie, L., Jiang, D., Zhang, X., and Tian, Q. Segment any- thing in 3d with nerfs. ArXiv, abs/2304.12308,, 2023. URL https://pp..emaaticchhee semanticscholar. ort//oorpusDD:2712011777
Hernandez, D., Brown, T., Greenwald, E., Kaplan, J., Abbeel, P., and McCandlish, Scaling laws for transfer. arXiv preprint urXiv:2102.012953 2021.
Chen, D. 7., Chang, A. X., and Nießner, M. Scan- refer: 3d object localization in rgb-d scans us- ing natural language. ArXiv, abs/1912.088800, 2019. URL https:/app..emmaniiccchoh
Hong, Y., Zhen, H., Chen, P., Zheng, S., Du, Y., Chen, Z., and Gan, C. 3d-llm: Injecting the 3d world into large language models. ArXiv, abs/2307.12996
2023. URL https://aat //ppi. semanticscholar. org/ CorpusID:
Cheng, B., Misra, I. Schwing, A. G., Kirillov, A., and Girdhar, R. Masked-attention mask transformer for universal image segmentation. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR), pp. 1280-1289, 2021a. URL https: //api. www.nteesccooeeeee.... org/CorpusID: 244799297.
Hong, Y., Zhang, K., Gu, Bi, S., Zhou, Y., Liu, D., Liu, F., Sunkavalli, K., Bui, T., and Tan, H. Lrm: Large recon- struction model for single image 3d. In International Conference on Learning Representations (ICLR), 2024.
Irshad, M. Z., Zakahrov, S., Guizilini, V. C., Gaidon, A., Kira, Z., and Ambrus, R. Nerf-mae: Masked au- toencoders for self-supervised 3d representation learn- ing for neural radiance fields. ArXiv, abs/2404.01300. 2024. URL https: /pi. semanticscholas org/ C/tp//orrpp:b//// 268856940.
Cheng, B., Schwing, A. G., and Kirillov, A. Per-pixel classification is not all you need for semantic segmen- tation, b021b. URL attps:://arxt org org/abs/ abs/ 2107.06278
Jain, A., Gkanatsios, N., Mediratta, I., and Fragki- adaki, K. Bottom up top down detection transform- ers for language grounding in images and point clouds. ArXiv, abx/2112.08879, 2021. URL https:
Contributors, S. Spconv: Spatially sparse convolution library. https:/ htap::///gthbb..../ http://gthhub..ccrrraalleeeeeeee wwtp://aw.e/eeaaees/... spconv, 2022.

---

From Billions to Thousands: 3D Language Grounding via Render-Supervised Distillation
//api semmatttsschoolarr.rr///roossspe 250921818.
Labs, B. F. Flux. https://ge https:://gittuubcom. flux, 2023.
Jain, A., Katara, P., Gkanatsios, N., Harley, A. W., Sarch, G. H., Aggarwal, K., Chaudhary, V., and Fragkiadaki, K. Odin: A single model for 2d and 3d segmenta- tion. 2024 IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pp. 3564-3574, 2024. URL https: httpps//ap...eematiicieiiit semantccscholar. http://oppSS::26755500
Leroy, V., Cabon, Y., and Revaud, J. Grounding image matching in 3d with mast3r. In European Con- ference on Computer Vision, 2024. URL https: seressiematiccceee org/CorpusID: 270521424.
Lin. T.Y.. Goyal, P., Girshick, R. B., He, K., and Dollár, P. Focal loss for dense object detection. 2017 IEEE International Conference on Computer Vixion (ICCV), pp. 2999-3007, 2017. URL https: semanticscholats org/ CorpusID:47222288
Jain, A., Swerdlow, A., Wang. Y., Arnau,, S., Martin, A., Sax, A., Meier, F,, and Fragkiadaki, K. Unifying 2d and 3d vision-language understanding. arXiv preprint arXiv:2503.10745,1 2025.
Liu, S., Zeng. Z., Ren, T., Li, F., Zhang, H., Yang, J., Li, C., Yang, J., Su, H., Zhu, J., et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection, arXiv preprint arXiv:2303.054 2023.
Jatavallabhula, K., Kuwajerwala, A., Gu, Q. Omama, M., Chen, T., Li, S., Iyer, G., Saryazdi, S., Keetha, N., Tewari, A., Tenenbaum, J., de Melo, C., Krishna, M., Paull, I.., Shkurti, F., and Torralba, A. Conceptfusion: Open-set multimodal 3d mapping. Roboties: Science and Systems (RSS), 2023a.
Loshchilov, I. and Hutter, F. Decoupled weight de- cay regularization. In International Conference on Learning Representations, 2017. URI. https: /api. s semanticscholase org/ htt.aaar....ooooou02:5552522
Jatavallabhula, K. M., Kuwajerwala, A., Gu, Q., Omama, M., Chen, T., Li, S., Iyer, G., Saryazdi, S., Keetha, N. V., Tewari, A. K., Tenenbaum, J. B., de Melo, C. M., Krishna, M.. Paull, L., Shkurti, F. and Torralba, A. Conceptfusion: Open-set multimodal 3d mapping. ArXiv, abs/2302.07241, 2023b. URL https: //tp://appp wwwppissematttschh... otr//GrrpusD::26686699
Majumdar, A., Ajay, A., Zhang, X., Putta, P., Yenamandra, S. Henaff, M., Silwal, S., Mevay, P., Maksymets, O., Amaud, S., Yadav, K., Li, Q., Newman, B., Sharma, M., Berges, V.-P., Zhang, S., Agrawal, P., Bisk, Y., Batra, D., Kalakrishnan, M., Meier, F., Paxton, C., Sax, A., and Rajeswaran, A. Openeqa: Embod- ied question answering in the era of foundation mod- els. 2024 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16488-16498, 2024. URL https::/p/// //api///pppp// semant ias.aattccchol..... azg/CorpusID:22 :6:20666:268
Kamath, A., Singh, M. LeCun, Y., Misra, I. Syn- naeve, G., and Carion, N. Mdetr - modu- lated detection for end-to-end multi-modal under- standing. 2021 IEEE/CVF International Confer- ence on Computer Vision (ICCV), pp. 1760-1770, 2021. URL. pestps://api.ssee semanticcchoool....... ott//CorpusID::23333...
Meta AI. Llama 3: The llama 3 herd of models. https:
//ai.metaa com/ 1 alama/, 2024. [Large language model].
Kerr, J.. Kim, C. M., Goldberg, K., Kanazawa, A., and Tancik. M. Lerf: Language embedded radiance fields. In International Conference on Computer Vision (ICCV), 2023.
Peng. S., Genova, K., Jiang, C. M., Tagliasacchi, A., Polle- feys, M., and Funkhouser, T. Openscene: 3d scene under- standing with open vocabularies. In CVPR, 2023.
Qin, M. Li, W., Zhou, J., Wang. H., and Plis- ter, H. Langsplat: 3d language gaussian splat- ting. 2024 IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pp. 20051-20060, 2023. URL atttps://app semant iistnoaaaccc.oar...... org/ /ccC//oopIID::655500000
Kim, C. M., Wu, M., Kerr, J., Tancik, M., Goldberg, K., and Kanazawa, A. Garfield: Group anything with radiance fields. In Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., Dollar, P., and Girshick, R. Segment anything. In Proceedings of the IEFF/CVF International Conference on Computer Vision (ICCV), pp. 4015-4026, October 2023.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning trans- ferable visual models from natural language supervi- sion. In International Conference on Machine Learning,
12

---

From Billions to Thousands: 3D Language Grounding via Render-Supervised Distillation
2021. URL htt.:///pp.ssmmaniccchhll.l.a
foundation language models. ArXiv, abs/2302.13971, 2023. URL https://api semanticscholar org/ /ttCCCopsI:::::21110101
arg/CorpuSSD:2215990
Roh, J., Desingh, K., Farhadi, A., and Fox, D. Lan- guagerefer: Spatial-language model for 3d visual grounding, ArXiv, abss21007003333. 2021. URL https: //api wwweemttiscchoo.o.....m www.cholaa.ooo/ooopopiioooo 235765540.
Wang. J., Chen, M., Karaev, N., Vedaldi, A., Rup- precht, C., and Novomý, D. Vggt: Visual geom- etry grounded transformer. ArXiv, abs/2503.11651 2025a. URL https://app httt::::/app.ssmaattsccse..e wwwant org/ :
Schult, J., Engelmann, F., Hermans, A., Litany, O., Tang, S., and Leibe, B. Mask3d: Mask transformer for 3d semantic instance segmentation. 2023 IEEE International Confer- ence Robotics and Automation (ICRA), pp. 8216-8223, 2022. URL
Wang. Q. Zhang. Y., Holynski, A., Efros, A. A., and Kanazawa, A. Continuous 3d perception
model with persistent state. 2025b. URL https:
htt. org/CozpusII: 2580079940
/ //api.semanticn. wwwpscholar.rg//oorreeeeeee 275789153.
Somasundaram, K. K., Dong, J., Tang, H., Straub, J., Yan, M.. Goesele, M., Engel, J. J., Nardi, R. D., and New- combe, R. A. Project aria: A new tool for egocen- tric multi-modal ai research. ArXiv, 1ls/3388.15611 2023. URL https: tttpppp...aataa.caaaeaeeooooeeee org/ hogp/orppsID::200 251224332222236.
Wang. S., Leroy, V., Cabon, Y., Chidlovskii, B., and Revaud, J. Dust3r: Geometric 3d vision made easy. 2024 IEEE/CVF Conference Computer Vi- sion and Pattern Recognition (CVPR), pp. 20697-20709, 2023. URL https:ttt//// seman org/ org/CorpusIDi /ctC//ooppII:2644300000000
Sudre, C. H., Li, W., Vercauteren, T. K. M., Ourselin, S., and Cardoso, M. J. Generalised dice overlap as deep learning loss function for highly unbalanced segmenta- tions. Deep learning in medical image analysis and mul- timodal learning for clinical decision support Third International Workshop, DLMIA 2017, and 7th Inter- national Workshop, ML-CDS 2017, held in conjunction with MICCAI 2017 Quebec City, QC...., 2017:240 248, 2017. URL https: http::///aapp.aaaii.co..aao.
Wang. Z., Bovik, A. C., Sheikh, H. R., and Simoncelli, E. P. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Im- age Processing, 13:600-612. 612, 2004. URL https: / paapi.semanticaccin wtw.ollarr..rg/orouutt... 207761262.
Xu, C.. Wu, B., Hou, J., Tsai, S. S., Li, R., Wang, J., Zhan, W., He, Z., Vajda, P., Keutzer, K., and Tomizuka, M. Nerf-det: Learning geometry-aware volumetric representation for multi-view 3d object detection. 2023 IEEE/CVF International Confer- ence Computer Vision (ICCV), pp. 23263-23277 2023a. URI. https: / /api. semant semanttcschooas cscholar. org Stss/CoooPIII::2202233333
Szymanowicz. S., Zhang, J. Y., Srinivasan, P. P., Gao, R., Brussee, A., Holynski, A., Martin-Brualla, R., Barron, J. T., and Henzler, P. Bolt3d: Generat- ing 3d scenes in seconds. ArXiv, abs/2503.14445, 2025. URL https::/api..smmmiicchhheh semanticsscooll.. oxt//oozpusID::27..41555
Xu, M., Yin, X., Qiu, L., Liu, Y. Tong, X., and Han, X. Sampro3d: Locating sam prompts in 3d for zero-shot scene segmentation. ArXiv, abs/2311.17707,, 2023b. URL attps://appt http::////apissmniiisssseeeee www.stiismcmnooo........ htg/CorpusI::2654/4
Takmaz, A., Fedele, E., Sumner, R. W., Pollefeys, M., Tombari, F., and Engelmann, F. OpenMask3D: Open- Vocabulary 3D Instance Segmentation. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Yang. J., Chen, X., Madaan, N., Iyengar, M., Qian, S. Fouhey, D. F., and Chai, J. 3d-grand: A million-scale dataset for 3d-lims with better ground- ing and less hallucination. ArXiv, abs/2406.05132. 2024. URL https://apii semanticscholar. org/ htgg/CooppsTD::703574400
Tang, J., Chen, Z., Chen, X., Wang, T., Zeng, G., and Liu, 7. Igm: Large multi-view gaussian model for high-resolution 3d content creation. In European Conference on Computer Vision, 2024. URL https: //api. semant cscholar.. Corg/CorrssTD 267523413.
Yang, J., Sax, A., Liang, J., Henaff, M., Tang, H., Cao, A., Chai, J., Meier, F., and Feiszli, M. Fast3r: Towards 3d reconstruction of 1000+ images in one forward pass.
Touvron, H., Lavril T., Izacard, G., Martinet, X., Lachaux, M.-A,, Lacroix, T., Rozière, B., Goyal, N., Ham- bro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient
2025. URL https: www.aissmmttiichhh.eeeeoe reg/CorrpsTD::255800
13

---

From Billions to Thousands: 3D Language Grounding via Render-Supervised Distillation
Yang, Z., Zhang, S., Wang, L.,, and Luo, J. Sat: 2d semanties assisted training for 3d visual ground- ing. 2021 IEEE/CVF International Conference Computer Vision (ICCV), pp. 1836-1846, 2021. URL. https:/ /aps://ap... http://aap.ssaaaiiii.... semanticccholar CorpusID:23516679
2024. URL https://aagp org/ org/CorpussD::27365555
semanticscholl
Zhang. Y., Gong, Z., and Chang, A. X. Multi3drefer:
Grounding text description to multiple 3d objects. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 15179-15179, 2023b. URL https:
Yeshwanth, C., Liu, Y.C., Nießner, M.. and Dai, A. +cannet++: A high-fidelity dataset of 3d in- door scenes. 2023 IEEE/CVF International Con- ference Computer Vision (ICCV), pp. 12-22, 2023. URL pttts::/api.semaatiioo ott/CooppuSD::21166777
/ ser.demmmatticccchooooooo oawaaatticccooaa..ooooooooooooop 261681990.
Zhou, Y., Gu, J., Chiang, T. Y., Xiang, F,, and Su, H. Point-sam: Promptable 3d segmentation model for point clouds. ArXiv, abs/2406.17741, 2024. URL https: wwt.pi..eattisccsoo...ooo org/ /ts//CoopSI::20011110
Yuan, Z., Yan, X., Liao, Y., Zhang, R., Li, Z., and Cui, S. Instancerefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi- level contextual referring. 2021 IEEF/CVE International Conference on Computer Vision (ICCV), pp. 1771-1780, 2021. URL https:/ app. semapmicccaainccchll.a.. org/Corpus 2332099523.
Zhu, H., Yang, H., Wu, X., Huang, D., Zhang, S., He, X.. He, T., Zhao, H.. Shen, C., Qiao, Y., and Ouyang, W. Ponderv2: Pave the way for 3d foundation model with universal pre-training paradigm. ArXiv, abs/2310.08586. 2023a. URL https: http://pppp.mmainssssesceee semanticschola.. org/ http/CorpusD:::200 :
Yuksekgonul, M. Bianchi, F, Kalluri, P., Jurafsky, D., and Zou, J. Y. When and why vision- language models behave like bags-of-words, and what to do about it? ArXiv, abs/2210.00136,, 2022. URL orgg/ooppuID::2273355
Zhu, H., Yang, H., Wu, X., Huang, D., Zhang, S., He, X., He. T., Zhao, H., Shen, C., Qiao, Y., and Ouyang, W. Ponderv2: Pave the way for 3d foundation model with a universal pre-training paradigm. arXiv preprint arXiv:2310.0855 2023b.
Yuksekgonul, M., Bianchi, F., Kalluri, P. Jurafsky, D., and Zou, J. When and why vision-language models be- have like bags-of- words, and what to do about it? In International Conference on Learning Representations, 2023. URL https:://///....... http://pweeeeeeeeeeeeeeeeeeeeeee forum? id KRLUvxh8uaX.
Zhu, Z., Ma. X., Chen, Y., Deng, Z., Huang, S., and Li, Q. 3d-vista: Pre-trained transformer for 3d vision and text alignment. 2023 IEEE/CVF International Con- ference on Computer Vision (ICCV), pp. 2899-2909, 2023c. URL https: /api. semanticscholar. org /CorppusI::20004495333
Zamir, A., Sax, A., Shen, B. W., Guibas, L. J., Malik, J., and Savarese, S. Taskonomy: Disentangling task transfer learning. 2018 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 3712-3722, 2018. URL https: /api....... semanticschonn org/CorpusID: 5046249.
Zhu, Z., Zhang, Z., Ma, X., Niu, X., Chen, Y., Jia, B., Deng, Z. Huang, S., and Li, Q. Unify- ing 3d vision-language understanding via promptable queries. In European Conference Computer Vision, 2024. URL https:://///. //ppi.app./////// www.nticscooolie...o org org/Cord prg//oorpusTD:26921131
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sigmoid loss for language image pre-training. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 11941-11952, 2023. URL https: //api. www.sssseennnccchseeeeeeeeeceeee wrw.oorpssr::/ 257767223.
Zhang, C., Han, D., Qiao, Y., Kim, J. U., Bae, S.-H., Lee, S., and Hong, C. S. Faster segment anything: Towards lightweight sam for mobile applications. arXiv preprint arXiv:2306. 14289, 2023a.
Zhang, H., Yang, C.-.,, and Yeh, R. A. Multi-object 3d grounding with dynamic modules and language- informed spatial attention. ArXiv, abs/2410.22306,
14

---

From Billions to Thousands: 3D Language Grounding via Render-Supervised Distillation
A. More Details
A.1. Training Details
LIFT-GStakes point clouds and posed RGB images for training. For efficiency, we preprocess point clouds and posed RGB images, caching the processed features.
Point clouds originate from multi-frame RGB-D seans. We unproject them using depth information and fuse the unprojections into the final point clouds. Each dataset sample is preprocessed into Scm-resolution point cloud chunks with corresponding posed RGB images.
For 2D pseudo-labels, we precompute SAM-CLIP features and cache them. Given the large size of the feature map, we decompose into two components: Semantics and Index2Semantics.
Semantics: A tensor of shape H W, where each pixel stores the index of the segment belongs to. Index2Semantics: tensor of shape N F, where N is the number of unique segments, and F is the CLIP feature dimension. This decomposition significantly reduces storage costs. When computing the feature rendering loss Cieas directly use features from Index2Semuntics for contrastive loss.
Each training sample consists of sparse point cloud and posed image with corresponding SAM-CLIP features. We randomly sample up to unique instances, using their CLIP features as pseudo language queries and their masks as target 2D masks. To ensure mask quality, we filter out masks smaller than 1024 pixels.
Randomly sampling instances is important for training, especially for zero-shot segmentation, it prevents the model reconstruct the whole images given all the input embeddings.
For grounding loss, we assign weights of 15.0, 2.0, and 6.0 to the mask cross-entropy loss, soft token loss, and Dice loss, respectively, We also use a photometric loss (L1 and SSIM) with weight of 1.0 and a feature loss with weight of 0.1.
layers, maximum channel dimension of 256, output feature dimension of 96. MaskDecoder: 8-layer Transformer decoder with a hidden state size of 512. uses 256 learnable mask proposal tokens, generating 256 masks. Each Transformer block has attention heads, a feedforward MLP of dimension 2048, and a dropout ratio of 0.15, Language Encoder: We use cl -lppp---- iap--vii-lle ceatt-laage-aee wwagee-ppcchhee 4, with feature dimension of 768.
UNet Encoder:
A.2. Comparison to 3D pseudolabels
Table 8: Comparison to 3D pseudolabels. A mask decoder trained top of frozen LIFT-GS features matches and even outperforms decoder trained on top of lifted 3D pscudolabels (voxel-poolee ConceptFusion (Jatavallabhula et al., 2023a)). LIFT-GS learns pool features in 3D in order to optimally reproduce the pscudolabels after rendcring, which outperforms using a hand-crafted aggregation. Note: in this experiment we used a more expressive mask decoder in this experiment with a larger MLP ratio, which improves the results for all methods, including LIFT-GS.
Features
Acc@0.25
Acc00.5
Scratch (RGB)
44.1
30.6
3D pscudolabels
50.1
34.7
2D pseudolabels (LIFF--G features)
51.8
38.3
L.FT--GS (finetuned)
54.7
40.5
A.3. Data Scaling Results
A similar trend is observed for 3D open-vocabulary instance segmentation, though the benefits of pretraining are slightly less pronounced due to the task's lower complexity. This aligns with our findings that pretraining is more beneficial for challenging tasks, such as those with higher IoU thresholds or greater complexity.
A.4. VLM Captions
We explore using vanion-language models (VLMs) to generate captions for each SAM-segmented object and encode these captions into CLIP embeddings as pseudo language queries.
Specifically, given a Seg-segee region, we draw red bounding box on the 2D image and highlight the masked region
15

---

From Billions to Thousands: 3D Language Grounding via Render-Supervised Distillation
mAP
mAP@25%
mAP@50%
25
40
35
35
30
20
30
25
15
25
20
20
10
Scratch Pretrain
Scratch Pretrain
15
Scratch
15
10
Pretrain
0.1
0.2
0.5
1.0
0.1
0.2
0.5
1.0
0.1
0.2
0.5
1.0
Figure 8: Finetunning Data Scaling on Open Vocabulary 3D Instance Segmentation. We show how mAP changes along with increasing Data Ratio from 0.1 to 1.0
using alpha blending, as shown in Figure 9. We then prompt VLM, such as -.ama-3.2x, with the following instruction:
You helpful assistant for image captioning. You given image with red bounding box specifying the object of interest. Caption that object in few words, keeping it precise and concise. The object is also slightly highlighted. Examples output: "a red traffic light," "the box near the wall." Just output the caption; no other text is needed.
This approach leverages VLM-generated textual descriptions to improve pseudo-language queries for training.
B. Discussion and Limitations
The contribution of LIFT-GS is training 3D model without 3D supervision by leveraging differentiable rendering and distilling knowledge from 2D foundation models. This approach is novel and motivated by the fact that 2D foundation models, trained on vast amounts of 2D data, currently outperform any existing 3D model. Distilling knowledge from these powerful 2D models presents promising and scalable direction for 3D learning.
Our proposed pipeline is general and unified. Beyond masks, any renderable 3D attributes can, principle, be trained using 2D supervision. This idea could extend to dynamic scenes and other properties, opening new opportunities for model training.
However, LIFT-GS is inherently constrained by how well leverage 2D foundation models for pseudo-labeling. Currently, we use CLIP image embeddings as text queries, but CLIP's claim of shared embedding space for images and text is imperfect. In practice, these embeddings can differ significantly, leading to challenges in zero-shot 3D segmentation.
Our CLIP-SAM features may not be optimal pseudo-labels for pretraining, and we anticipate that improved pseudo-labeling strategies will lead better scaling properties, stronger performance, and even robust zero-shot 3D segmentation without fine-tuning. Addressing our current limitations presents key opportunity for future work.
Although LIFT-GS significantly improves performance and surpasses the stage SOTA method BUTD-DETR (Jain et al., 2021), it still falls short of two-stage SOTA methods like 3D- VisTA (Zhu et al., 2024) on 3D referential grounding at an IoU threshold of 0.5. A robust single-stage 3D VLG model would have major impact across various applications. We hope that our architecture-agnostic pretraining pipeline can further enhance future models.
16

---

From Billions to Thousands: 3D Language Grounding via Render Distillation
Figure 9: Input Image to VL. for Captions. Given the segments from SAM, we draw red bounding box around the segments and ask VLM models to descrihe the segments inside the red bounding boxes.
17


Reviews:
[{"review_id": "CMbbzge9dK", "reviewer": "Reviewer_zvKK", "summary": "The paper presents LIFT-GS, a feedforward 3D vision–language grounding model that accepts a point cloud and a language query as inputs. It converts the point cloud into 3D Gaussians and uses differentiable rendering to supervise training with only 2D losses. The system is distilled from 2D foundation models to achieve 3D mask predictions for language-described objects.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First, we would like to thank the reviewer for taking the time and effort to engage with the paper, and we look forward to a productive discussion. Before addressing the individual questions, we would like to clarify the core focus of our work. \n\nWhile our method involves 3D scene reconstruction as an intermediate step, the primary goal of this paper is **3D vision-language grounding**—specifically, localizing 3D instances (i.e., point cloud masks) based on complex language queries. 3D spatial understanding is a known weakness of existing VLMs ([1; Fig. 5]), but is a critical capability for applications across robotics and embodied AI. For example, it can identify pick-and-place locations in long-horizon manipulation and rearrangement tasks.\n\nThe key challenges and contributions of our work lie in developing an effective and scalable pipeline for (pre-)training large 3D visual-language grounding models (the **Decoder**), and the contribution is to do this using differentiable rendering using 2D supervision (Line 97).\n\n---\n> similar ideas have been explored via … feedforward transformers (Large Spatial Model, SAB3R)\n\nCould the reviewer kindly provide a citation or reference for SAB3R? Despite our best efforts, we have been unable to find a corresponding paper or preprint. On the project website, we found [3], the paper link on the project website returns a 404. The only reference we found is on a personal website [2], which does not link to any publication or arXiv submission.\nMoreover, none of the other referenced papers evaluate on any of the standard 3D visual-language grounding benchmarks (ScanRefer, SR3D, NR3D).\n\n---\n> Why point cloud inputs?... why focusing on point clouds is beneficial\n\nAs mentioned in the paper, the losses used in LIFT-GS are largely agnostic to inputs and could work with multi-view image inputs. The fact that the introduced problem formulation is independent of the input type highlights the primary focus of our paper, which is **a new method for training 3D visual-language grounding models (especially Decoder)**, and we believe this is a strength of the method. \n\nWe chose to focus primarily on point cloud inputs, since they are currently one of the most widely adopted 3D representations in robotics, and the return type of SLAM and SfM. Point clouds offer flexibility and modularity, as they decouple the 3D reconstruction process from the grounding task, allowing us to leverage a wide range of input sources—including single-view, sparse-view, or long RGB(-D) videos, as well as LiDAR scans or other sensors. This makes point cloud-based methods especially well-suited to diverse real-world conditions.\n\nSince pointcloud reconstructions can be incrementally updated and are largely independent of the specific sensor package (camera, depth sensor, IMU, etc). These properties make point clouds a practical choice for real-world robotic systems. For better or worse, they have become a de facto representation for 3D language grounding.\nWe will make this point much clearer in the final version.\n\n---\n\n> Technical Novelty and compared to prior works. The method sections (3.1–3.4) describe a task formulation, standard 2D-supervised training for 3D reconstruction, and a conventional network for lifting point clouds into 3D Gaussians. The contributions in terms of problem and model novelty are not sufficiently distinguished from prior work. A stronger claim on what specific challenges are uniquely solved by focusing on point clouds and 3D Gaussian representations is needed.\n\nReconstructing 3D Gaussian splats from point clouds is neither the focus nor a claimed contribution of our work. Our primary contribution lies in **(pre-)training a large Transformer-based mask decoder for 3D visual-language grounding without requiring 3D labels**, made possible through the use of differentiable rendering.\n\nIn contrast, the mentioned prior methods—whether based on per-scene optimization or feedforward models—focus on reconstructing the **RGB and feature fields** of the 3D scene and perform language grounding by computing **dot products between language CLIP features and 3D feature fields**. These approaches primarily consist of encoder-only architectures, whereas our work emphasizes the use of a powerful Transformer-based decoder for mask prediction. We discuss it in L96-109, L153-164.\n\nDue to the limitations of CLIP embeddings, dot-product-based grounding methods struggle with handling **slightly complex language expressions** and cannot reliably **reason about relative spatial relationships**. This restricts their effectiveness in realistic grounding tasks. Because of character limits, we provide quantitative comparisons and discussions in the reply for Reviewer eYEH.\n\n[1] https://open-eqa.github.io/\n\n[2] https://tianx-ia.github.io/\n\n[3] https://web.archive.org/web/20250501000000*/https://uva-computer-vision-lab.github.io/sab3r/static/pdf/Semantic_Augmented_3D_Foundation_Models__Writing.pdf", "claims_and_evidence": "- Supervision Without 3D Labels\n\nThis is not a novel concept nor a new problem; similar ideas have been explored via NeRF-based per-scene optimization (LERF, NeRF-DFF), Gaussian splatting (LangSplat), feedforward transformers (Large Spatial Model, SAB3R), and even 3D LLMs (3D-LLM). Moreover, the choice to use point clouds rather than multi-view images appears as a mere setting difference. It remains unclear why the authors focus exclusively on point clouds.\n\n- Technical Novelty\n\nThe method sections (3.1–3.4) describe a task formulation, standard 2D-supervised training for 3D reconstruction, and a conventional network for lifting point clouds into 3D Gaussians.\nThe contributions in terms of problem and model novelty are not sufficiently distinguished from prior work. A stronger claim on what specific challenges are uniquely solved by focusing on point clouds and 3D Gaussian representations is needed.", "methods_and_evaluation_criteria": "- Input Modality and Framework Choice\n\nThe paper focuses solely on point cloud input. Since differentiable rendering with known camera poses is well established for RGB images, the choice to restrict the input to point clouds raises questions.  It would strengthen the work to clarify why point clouds are preferred in this context and to discuss how the framework might extend (or why it might not extend) to settings with multiple images.", "theoretical_claims": "The manuscript does not propose any novel theoretical claims; its contributions are primarily empirical and architectural.", "experimental_designs_or_analyses": "Clarify how the model’s performance scales with data volume (how many scenes are used) and whether the absence of RGB images (as inputs) impacts the supervision via differentiable rendering.", "relation_to_broader_scientific_literature": "The paper’s positioning relative to prior 3D reconstruction and vision–language grounding work should be discussed in greater depth. The literature shows many similar approaches; hence, a clear delineation of the unique aspects of this work is essential.", "essential_references_not_discussed": "While the references are adequate, the paper does not make a clear case for a distinct problem being solved or a novel method proposed. A more critical comparison to similar recent methods would help clarify its contributions.", "other_strengths_and_weaknesses": "### Strengths:\n\n- The paper leverages a scalable pipeline using differentiable rendering and distillation from 2D foundation models.\n- It demonstrates state-of-the-art performance on downstream 3D vision–language grounding tasks.\n\n### Weaknesses:\n\n- The choice of point clouds over multi-view images is not sufficiently justified.\n- The overall novelty in both problem setting and technical method is not clearly established beyond what prior works have already addressed.", "other_comments_or_suggestions": "The authors should clarify why focusing on point clouds is beneficial and what new challenges are addressed by their specific combination of 3D Gaussian representations and 2D supervision. Strengthening the discussion on how their approach differs from existing feedforward or per-scene optimization methods would improve the paper’s impact.", "questions_for_authors": "See the questions above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NfdOAuXnNg", "reviewer": "Reviewer_eYEH", "summary": "This paper presents LIFT-GS, a scalable pretraining approach for 3D vision language grounding. Specifically, the model takes in a point cloud of the scene along with the language query embeddings to produce 3D Gaussians with features together with the predicted masks for grounding. For training LIFT-GS, the reconstruction loss on RGB, feature loss on image features from 2D foundation models, and the grounding loss on the predicted masks are applied. The method does not require ground truth 3D annotations or even 2D annotations. Instead, it leverage 2D foundation models to generate pseudo-labels for training. Experimental results on 3D language grounding demonstrate the decent the performance of the proposed method.\n\n## Update after rebuttal\n\nMy concerns about comparing with more recent state-of-the-arts and the training cost comparison are mostly solved. However, since the high-level idea is somehow similar to LangSplat which makes the Gaussians to predict other features and properties beyond the original properties, I would regard the method as a good direction to try on 3D language grounding, but the concept itself is not super novel. Therefore, I would like to maintain my score of weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback and appreciate the recognition that using 2D supervision is “very useful for scaling up the training data,”  “the claim… is supported by the experiments.”,  “cross-scene render-supervision is innovative…”. We address the questions below and will incorporate improvements in the final version.\n\n---\n> Compared to baselines, which output language embedding in 3D Gaussians with additional features, like Langsplat. \n\nThanks for the suggestion — we will clarify it in the final version. Both LIFT-GS and LangSplat use render-supervised distillation. However, LangSplat's per-scene optimization method is primarily designed for object segmentation (nouns only), while LIFT-GS is a cross-scene method supporting 3D referential grounding, involving more complex language.\n\nLangSplat and its variants rely on **language-vision feature dot products for open-vocabulary segmentation**: reconstructing 3D RGB and CLIP feature fields and computing dot products with text embedding for segmentation (**Encoder only**). However, contrastive language-vision models (e.g., CLIP) tend to behave like bag-of-words models [2], making them struggle with handling slightly long language expressions with relational structure, such as spatial relationships, key in referential grounding.\n\nIn contrast, LIFT-GS employs structured supervision from MDETR, training a **transformer decoder** over visual and language tokens to directly predict 3D masks and groundings through learned attention and referential loss. \n**We illustrate the failure modes of dot-product-based methods on both  3D and 2D with language inputs in Fig** https://postimg.cc/bd6QKTYk, https://postimg.cc/47Y9ZCxc.\n\nWe compare LIFT-GS and LangSplat variants on 3D referential grounding benchmark ScanRefer. As LangSplat operates on rendered 2D images instead of 3D, we use a variant Semantic Gaussians [1] that reports much higher performance and directly segments 3D; as well as LERF (which both papers compare to). \n\n| Method | Acc@0.1 | Acc@0.25  | Acc@0.5| \n|-----------|---------------|-----------------|-------------|\n|  Semantic Gaussians |  18.2% |  8.2% | 3.0%|\n| LERF | - | 4.4% | 0.3% |\n| LIFT-GS | - | 49.7% | 36.4%| \n\n\n---\n> Comparison and discussion about SceneVerse [5], D-LISA [6]. \n\nThanks for your suggestions, and we will include comparisons and discussions in the final version. \nThe submission incudes comparisons to PQ3D (ECCV 2024), which reports stronger performance than both SceneVerse and D-LISA. For open-vocabulary segmentation, we compare against the numbers reported in PQ3D and show that LIFT-GS outperforms it.\n\nFor the 3D referential grounding task, we tried our best efforts to retrain baselines under the Sensor Point Cloud setting for a fair comparison; adopting PQ3D, 3D VISTA, and BUTD-DeTr as our main baselines. Despite our best efforts, we weren't able to train PQ3D. However, we believe that 3D-VISTA is the appropriate comparison here, since the requested references perform comparably to that method on mesh point clouds. We compare their reported results as published below. \n\nFinally, we emphasize that LIFT-GS is both **model- and data-agnostic**, making it orthogonal to advances in data (e.g., SceneVerse) or architecture (e.g., D-LISA). Stronger models or larger 3D datasets can be seamlessly incorporated into our framework for better performance, as indicated in Figure 6. \n\n| Method       | SR 0.5@Acc-Multiple | SR 0.5@Acc-Unique|  Contribution|\n|-----------------|-------------------------------|----------------------------| ----------------|\n| PQ3D         |46.2| 78.2 | - |\n| 3D-VisTA  | 39.1| 75.1 | -|\n| D-LISA      | 40 | 75.5 | Architecture|\n| SceneVerse| 42.7| 77.9| New Data| \n\n---\n> Computation Resource Comparison.\n\nThe difference in computational cost primarily arises from the **type of point cloud data** used during training. We follow the setting of BUTD-DeTR, using sensor point clouds—unprojected from posed RGB-D frames (frame skip of 30, voxel size 0.05 cm)—resulting in ~30k points per scene. This setup reflects real-world use cases more closely (L305). \n\nIn contrast, most other methods,  including SceneVerse, use ScanNet mesh segments, which are pre-annotated via face clustering, reducing scenes to ~300–1500 segments (∼100× fewer elements). This significantly lowers computational cost.\n\nHowever, meshing is not viable for real-world or real-time applications—it is computationally expensive and also contains human-annotated information. That said, for fairness, LIFT-GS can also operate on mesh segments, and under this setting, training requires comparable compute (8×A100 GPUs for 2–3 days).\n\n---\n> Typos:\n\nWe greatly appreciate these catches and will fix them in the final version.\n\n[1] Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splatting\n\n[2] [When and Why Vision-Language Models Behave like Bags-Of-Words, and What to Do About It?](https://openreview.net/forum?id=KRLUvxh8uaX)", "claims_and_evidence": "Yes, the claim that the proposed LIFT-GS acts as a pretraining solution for 3D language grounding without 3D supervision is supported by the experiments.", "methods_and_evaluation_criteria": "Yes, the proposed methods and the evaluation criteria make sense.", "theoretical_claims": "The paper does not have any proofs or theoretical claims.", "experimental_designs_or_analyses": "Yes, the experimental designs or analyses are sound and valid, except that I have some concerns about whether there are stronger and more recent state-of-the-arts for comparison. I will elaborate on this point in the \"Other Strengths And Weaknesses\" section.", "relation_to_broader_scientific_literature": "The paper is focusing on 3D vision language grounding, which is broadly related to 3D multi-modal language models. It can be a primitive step towards artificial general intelligence.", "essential_references_not_discussed": "There might be some more recent works that can be used for comparison. For example, SceneVerse [1] and D-LISA [2]. Also, LangSplat [3] should be discussed in the literature as it is the first to enable open-vocabulary language grounding in 3D scenes with Gaussian Splatting, although LangSplat is per-scene optimized instead of training across scenes in a feed-forward way as the proposed method. Other than that, there are no essential references that are not discussed.\n\n\n[1] Jia et al. SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understanding. ECCV 2024.\n\n[2] Zhang et al. Multi-Object 3D Grounding with Dynamic Modules and Language-Informed Spatial Attention. NeurIPS 2024.\n\n[3] Qin et al. LangSplat: 3D Language Gaussian Splatting. CVPR 2024.", "other_strengths_and_weaknesses": "**Strengths:**\n- The proposed method achieves training a 3D vision language grounding model with no ground truth 3D supervision. Moreover, the 2D labels can also be pseudo-labels generated by 2D foundation models. It can be very useful for scaling up the training data as the training process does not require 3D annotations. \n- The proposed training pipeline of cross-scene render-supervision is innovative to me in the context of 3D vision language grounding. Nevertheless, I have to admit that both (1) cross-scene 3D Gaussians / neural rendering and (2) language embedding in 3D Gaussians with additional features are not completely new ideas. For (1), the related works include pixelSplat [1], MVSplat [2], etc. For (2), the related works include LangSplat [3].\n\n\n**Weaknesses:**\n- For the experimental comparisons, the most recent state-of-the-art for fair comparison seems to be 3D-VisTA [4], which seems to be a relatively out-of-date work for me. I think there are more recent advancements for 3D vision language grounding, including SceneVerse [5], D-LISA [6], etc. The authors seem to miss on discussing these related works, and therefore missing the experimental comparisons with them.\n- The training cost seems to be huge, which requires 32 A100 GPUs. Compared to other methods like SceneVerse [5] (which is already a method for scaling up data) which only requires 8 A100 GPUs for 2 days, the training cost of the proposed method seems to be very intimidating.\n\n[1] Charatan et al. pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction. CVPR 2024.\n\n[2] Chen et al. MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images. ECCV 2024.\n\n[3] Qin et al. LangSplat: 3D Language Gaussian Splatting. CVPR 2024.\n\n[4] Zhu et al. 3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment. ICCV 2023.\n\n[5] Jia et al. SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understanding. ECCV 2024.\n\n[6] Zhang et al. Multi-Object 3D Grounding with Dynamic Modules and Language-Informed Spatial Attention. NeurIPS 2024.", "other_comments_or_suggestions": "There are some typos and strange grammars in some parts of the paper. I list some of them here:\n- Line 22: \"For training, only need images and camera pose, and 2D labels.\" The sentence is weird.\n\n- Line 25-26: \"We demonstrate this to pretrain a network ...\" The expression of this sentence is strange.\n\n- Line 163: \"ganulariry\" -> \"granularity\"\n\n- Line 182: \"Lmask\" -> \"$L_{\\rm mask}$\"\n\n- Line 205: \"$C\\sigma(i)$\" -> \"$C_{\\sigma(i)}$\"\n\nThe correspondence matrix $C$ in Equation 1 has not be explained until Lines 194-195, which makes me feel very confused when I first read the paper.", "questions_for_authors": "- How is the proposed method compared with the more recent state-of-the-arts like SceneVerse [1] and D-LISA [2]? Are there any specific reasons for the authors not discussing these works and not comparing with them in the experiment section? If not, I think comparing with \n- What is the training cost comparison between the proposed method and the compared baselines? The paper only states that the method needs to run 76K steps on 32 A100 GPUs. However, there are no specific training time / cost comparison between LIFT-GS and the other baselines.\n\n\n[1] Jia et al. SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understanding. ECCV 2024.\n\n[2] Zhang et al. Multi-Object 3D Grounding with Dynamic Modules and Language-Informed Spatial Attention. NeurIPS 2024.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fCW8GqEyf5", "reviewer": "Reviewer_aEzm", "summary": "The work addresses the problem of open vocabulary based 3d segmentation, that is predicting 3d masks for an RGB point cloud that adhere to the language based query.  \nIn order to do so the authors propose a feedforward architecture that predicts 3D Gaussians which carry information about their belonging to segmentation masks.  These masks are predicted via a transformer decoder from the language query and the encoded point cloud (with learnt mask tokens similar to the decoder of MaskFormer).  \nA key point of the paper is the supervision via 2D pseudo ground truth (with is plenty) rather than 3d labels.  This is achieved by rendering the Gaussians and their masks and comparing them against 2d segmentations built from SAM masks and corresponding CLIP language embeddings.  The model is optimized on the (i) RGB reconstruction, (ii) mask and language consistency, and (iii) a feature loss.  \nExperimental results demonstrate the effectiveness of the approach for open vocab instance segmentation (ScanNet200) and 3D referential grounding (ScanRefer, SR3D, NR3D).  Further ablation studies provide insights into the losses, the impact of pretraining, and the importance of good 2D foundational models for pseudo ground truth generation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s recognition of LIFG-GS, including “the method is well presented and builds upon existing building blocks”,  and believing  “As such the proposed 3d supervised pre-training task is valuable to the wider community.”. We answer the question below and will make them more clear in the final version.  \n\n---\n>How are the different scales of maks that are generated by SAM handled in the loss? Segmentations in different images that see the same object will contradict such that a consistent 3d segmentation is not easily possible. He. et al. \"View-Consistent Hierarchical 3D Segmentation Using Ultrametric Feature Fields\", ECCV'24 addresses this problem via a hierarchical segmentation.\n\nThat’s an excellent question, and we appreciate the opportunity to elaborate on this point. A key advantage of our approach is that it does **not** require 3D view-consistent masks, as our method is fully learning-based and relies solely on 2D supervision.\n\nIn contrast, existing methods that leverage SAM often rely on hand-crafted hierarchical structures or empirically designed mask-merging heuristics to improve consistency across views. These procedures are typically complex and heuristic-driven, introducing additional sources of noise and making the pipeline less robust.\n\nOur framework takes a different route: the 3D model is queried using 2D CLIP embeddings, and training is guided only by corresponding 2D mask and grounding losses. As a result, there is no requirement for the 2D masks to be consistent across views. Instead, we encourage the Transformer decoder to **learn how to produce coherent 3D masks** that align with these independently supervised 2D views, in a fully data-driven manner—without relying on manually designed post-processing steps.\n\nTo extract masks from SAM, we sample a grid of points across each image and generate masks accordingly. We apply Non-Maximum Suppression (NMS) to reduce overlapping masks and discard extremely small ones. Aside from these minimal filtering steps, we do not apply any further merging or heuristic post-processing.\n\n---\n >Which model is used to compute the ground truth feature map F_2D?\n\nWe use the same pipeline as in Figure 3 to compute the ground-truth 2D feature map F_2D​. Specifically, we apply the CLIP image encoder to extract features for each segmented region, assigning the resulting feature vector to all pixels within that region. For segmentation, we adopt the SAM-H model, and we use the CLIP-L model to extract features. This pipeline is flexible and can also accommodate alternative backbones, such as DINO-v2.\n\n---\n>A comparison between PQ3D and the proposed approach is missing on the very same input data…. However, the novelty of the approach lies within the generic (pre)training from 2d ground truth rather than beating the top performer on any benchmark.\n\n\nWe sincerely appreciate the reviewer’s recognition that the novelty of our work lies in proposing a generic (pre)training framework using 2D supervision, rather than merely achieving state-of-the-art results on specific benchmarks.\n\n\nRegarding PQ3D, we note that the reason why our method does not use exactly the same input data—i.e., mesh point clouds—as PQ3D,  is that mesh point clouds are impractical in real-world applications due to the need for time-consuming meshing and human annotations (L298-303). To ensure a fair comparison, we made every effort to retrain all baselines under our unified setting for an apples-to-apples comparison. However, we were unable to successfully reproduce PQ3D due to its complex, multi-stage training pipeline that spans across multiple datasets.\n\nWe greatly appreciate the reviewer’s suggestions and agree that more rigorous, apples-to-apples evaluations—using identical input settings as prior works—would further improve the clarity and fairness of our comparisons. While we are actively working on this, the time and computational constraints during the rebuttal period prevent us from including those results here. We plan to incorporate them in a future revision of the paper.\n\n---\n> Additional Reference\n\nThanks for pointing out, and we will include them in the final version.", "claims_and_evidence": "The main claim of the paper is to distilling knowledge from 2D foundation models for supervision rather than relying on 3d labels.  This argument is well accepted in the community and underlined by the scarcity of 3d labels.", "methods_and_evaluation_criteria": "The method is well presented and builds upon existing building blocks (differentiable rendering via Gaussian splatting, mask prediction via learnt proposal tokens, SAM based segmentation, CLIP for language embeddings) but also the usage of 2d ground truth has been well explored in the community (which is appropriately cited in the related work section).", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "A comparison between PQ3D and the proposed approach is missing on the very same input data.  The authors argue that they didn't succeed in retraining PQ3D on \"Sensor PC\" input data.  Though, LIFT-GS could have been trained and evaluated on \"Mesh PC\" data.  Even though this input data might not be available in real-world applications, it would allow for an apples-to-apples comparison to the SoTA method.  Given the present analysis it appears that PQ3D might outperform the proposed method.  \nA comparison would provide clarity. However, the novelty of the approach lies within the generic (pre)training from 2d ground truth rather than beating the top performer on any benchmark.", "relation_to_broader_scientific_literature": "Open vocabulary 3d instance segmentation is a very active research field.  As such the proposed 3d supervised pre-training task is valuable to the wider community.", "essential_references_not_discussed": "Liu et al., \"Weakly Supervised 3D Open-vocabulary Segmentation\", NeurIPS'23\n\nQin et al., \"LangSplat: 3D Language Gaussian Splatting\", CVPR'24", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "- How are the different scales of maks that are generated by SAM handled in the loss?  Segmentations in different images that see the same object will contradict such that a consistent 3d segmentation is not easily possible. He. et al. \"View-Consistent Hierarchical 3D Segmentation Using Ultrametric Feature Fields\", ECCV'24 addresses this problem via a hierarchical segmentation.\n\n- Which model is used to compute the ground truth feature map F_2D?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yD1mjmQBVd", "reviewer": "Reviewer_35FE", "summary": "This paper proposes an approach for 3D vision-language understanding by leveraging rendered RGB images, grounding masks, and 2D feature loss for model training, rather than incorporating explicit 3D supervision. The model follows a pretrain-finetune paradigm, with evaluations conducted on open-vocab 3D instance segmentation and 3D referential grounding. Ablation studies demonstrate that the pretraining stage enhances fine-tuning efficiency by reducing the amount of required data.\n\n## Post rebuttal\nI appreciate the authors’ clarifications regarding the model architecture, pretraining strategy, and the paper’s claims. Although this direction is promising, I cannot currently recommend acceptance due to several issues:\n(1) the omission of critical technical details in the methods section;\n(2) numerous typos and grammatical errors;\n(3) overstatements of the model’s capabilities; and\n(4) insufficient explanation as to why pretraining on ScanNet is beneficial compared to ScanNet++. Instead, the authors only compare pretraining on ScanNet++ followed by fine-tuning on ScanNet.\n\nConcerns (1) and (2) were also raised by Reviewer eYEH, and (3) was flagged by Reviewer zvKK. The authors acknowledge these points in the rebuttal. In the paper’s current form, there is more confusion than clarity: for instance, the model details remain unclear, it’s not included which data the model is pretrained on, how exactly that pretraining is conducted, and why ablations and data scaling are evaluated on a combined set of ScanRefer, SR3D, and NR3D. Furthermore, the remaining grammatical errors and overstated claims may mislead the readers, and I encourage the authors to revise and polish the paper to improve its coherence.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate that the reviewer likes our results and data scaling performance. We answer the questions below and will improve the writing given the valuable suggestions. \n\n---\n\n> Model architecture and its operational flow… \n\n**Encoder Backbone**:  is a SparseConv UNet [1] (Sec. 3.4), following PonderV2 implementation.\n\n**Gaussian Head** is a lightweight MLP (2 layer, 128 dim)  mapping each point from the Encoder Backbone to Gaussian parameters (L182-L190), including scaling, color, etc.\n\nThe **Mask Decoder Transformer** follows the MaskFormer pipeline [2], using m learnable tokens (mask proposal tokens in Figure 4) to predict potential 3D masks. Each token corresponds to a binary 3D instance mask over the Gaussians. \n\nThe **correspondence matrix** C (m x |Q|) grounds the m instance masks into the Q language tokens. Each element indicates the probability that the mask corresponds to that particular language token.\n\nThe **mask decoder** is a Transformer decoder, where the visual and language tokens serve as the Key and Value, while the proposed mask tokens act as the Query inputs to the decoder.\n\nTo further aid clarity and reproducibility, we will release the code and improve the final version. Lastly, we note that LIFT-GS is model-agnostic, imposing minimal architectural constraints and being readily adaptable to other architectures (Lines 224–230). \n\n\n> Pretraining Data\n\nWe used the training scenes from ScanNet and ScanNet++ for pretraining, except for the ablation study in Table 6. For images, we sampled frames from the original video trajectories with a frame skip of 30 (i.e., at 1 Hz). Each selected RGB-D frame was unprojected using the provided camera intrinsics and poses, then voxel-pooled at a 5 cm resolution, so the total point cloud number is controlled.\n\n\n### Claims\n\n> Using rendered features/images as supervision is not new in 3D VL understanding... This paradigm is the first in the 3D referential domain, to the best of my knowledge.\n\nWe agree and will make the claim more precise by specifying that we mean grounding with complex language (3D referential grounding).\n \n\n\n> The claim \"render supervised framework…\" is quite bold to make. For example, extending... or to 4D dynamic scenes is inherently non-trivial.\n\nWe appreciate the reviewer’s suggestion and will revise it to make it more accurate.  We appreciate any recommendations, and suggest the following: “The render-supervised framework provides a general and extensible design. LIFT-GS shows how to use it for highly structured tasks, such as 3D referential grounding and object detection”.\n\n We genuinely believe that our pipeline presents a general and extensible design. While adapting it to new tasks may involve non-trivial effort, we argue that such extensions are both feasible and conceptually straightforward.\n\nFor image inputs, one could use methods like Dust3R [3] to regress point maps from images as point clouds. For dynamic scenes, it is possible to regress motion basis coefficients for each point, as Shape-of-Motion[4]. These examples illustrate our belief that the proposed pipeline can be extended to support a wide range of applications beyond the current scope.\n\n\n### Experimental analysis\n\n> Role of L_{RGB} loss \n\nWe keep L_{RGB} loss because it is necessary to supervise the reconstruction of the 3D Gaussian fields. We agree that this loss may not significantly benefit downstream tasks, as it provides limited semantic information.\n\n> Improvement of adding Scannet++ in Table6?\n\nExcellent point. In Tab. 5, increasing the finetuning data by 100% yielded about a 5%  Acc@0.5 improvement. Increasing the pretraining data by 30% yielded a 1% improvement. \n\nBased on the curve in Figure 6 (we show the real and estimated values based on the curve below), the performance gain from adding ScanNet++ data (30% more) for pretraining is roughly equivalent to adding 15% more ScanNet finetuning data with 3D annotations.  It shows that the effective transfer ratio is roughly 1 / 2; i.e., collecting twice the number of raw videos alone can yield improvements comparable to building a fully annotated 3D dataset.  \n\nTherefore, **pretraining on ScanNet++ is highly efficient and cost-effective**, considering that annotating 3D referential grounding data requires orders of magnitude more effort than collecting raw video alone.\n\n|Finetune Data Ratio| 10%   | 20%    | 50%  | 100% | 115%| 130%  | Pretraining on Scannet++(130%)|\n|-|-|-|-|-|-|-|-|\n| Acc@0.25             | 24.96  | 36.11  | 43.80 | 47.53 | 48.28 |  48.94| 48.29|\n| Acc@0.5              |  14.70  | 23.03 | 28.89  | 33.75 | 34.72 | 35.59 | 34.35|\n| Acc@0.75            |  4.89    | 8.26   | 11.42  | 13.49 | 13.91 | 14.27 | 14.06|\n\n[1] https://github.com/facebookresearch/SparseConvNet/blob/main/examples/ScanNet/unet.py\n\n[2] MaskFormer: Per-Pixel Classification is Not All You Need for Semantic Segmentation\n\n[3] DUSt3R: Geometric 3D Vision Made Easy\n\n[4] Shape of Motion: 4D Reconstruction from a Single Video", "claims_and_evidence": "- The paper proposes a framework for pretraining and fine-tuning on 3D VL tasks, and its performance is verified by the experiments and ablations.\n\n- However, \"The approach is new for vision-language understanding.\" (Line 16) is not accurate as using rendered features/images as supervision is not new in 3D VL understanding, as also pointed out by the paper's related work section, e.g., Ponderv2, Point Cloud Unsupervised Pre-training via 3D Gaussian Splatting (Liu et all.). This paradigm is the first in the 3D referential domain, to the best of my knowledge.\n\n- The claims in the introduction \"render supervised framework can be used with essentially any 3D/4D task or model, provided the results are renderable\" is quite bold to make. For example, extending the proposed method to tasks requiring image inputs (e.g., fine-grained caption) instead of sparse point clouds or to 4D dynamic scenes is inherently non-trivial.", "methods_and_evaluation_criteria": "The paper lacks sufficient details of the proposed methodology, including both the model and data perspectives.\n\n- The paper lacks a detailed explanation of the model architecture and its operational flow. While it specifies the inputs and outputs of each module, it does not clarify how inputs are processed within the model or how outputs, such as the correspondence matrix C, are subsequently utilized. Furthermore, Figure 4 and the accompanying text are not well-aligned, leading to significant confusion for readers trying to understand the overall framework.\n\n- The paper does not mention how (much) pretraining data is generated or used in the pretraining. Are all the images used for each scene? Is the sensor PC organized by the RGBD point clouds for all the frames? Then how to handle the increasingly large point clouds as the model input.", "theoretical_claims": "No theoretical claims in the paper.", "experimental_designs_or_analyses": "The experiment results generally validate the effectiveness of the proposed model. I have two questions:\n\n1) In Tab.4 Loss Ablation, what is the performance when only L_{RGB} is excluded? It seems the L_{RGB} loss plays a negligible role in the performance.\n\n2) Could the author provide insights into why adding Scannet++ only marginally improves the performance in Tab. 6? ScanNet++ dataset, while smaller than ScanNet, is roughly 1/3 of the ScanNet scenes. Is using half of the ScanNet scenes realizing similar performance, similar to the data scaling of the fine-tuning data?", "relation_to_broader_scientific_literature": "I believe the proposed method aims to address the scarcity of data in 3D vision language understanding domain, by leveraging the benefits of neural rendering, e.g., Gaussian Splatting, and its strong relationship with point clouds. The paper presents evidence of improving 3D referential grounding through this paradigm, which I believe can bring potential insights for the community.", "essential_references_not_discussed": "Not that I'm aware of.", "other_strengths_and_weaknesses": "The writing of the paper could be further refined. Currently, it contains typographical errors, unjustified hypotheses in the introduction, and a method section that lacks sufficient details.", "other_comments_or_suggestions": "Typos, for example, on Line 22-23 \"For training, only need images and camera pose, and 2D labels.\", Line 25-26 \"We demonstrate this to pretrain a network\", Line 90 \"Specifically, render supervised framework can be used\".", "questions_for_authors": "While I appreciate the results demonstrated in the experiment section, as well as the data scaling performance in the ablation, I cannot give a positive rating at the moment primarily due to my concerns over the method and pretraining data as outlined above. Especially, how do the readers comprehend the effectiveness of pretraining when the details of pretraining data are not presented? The authors are encouraged to address my concerns above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: After the review and rebuttal, two reviewers recommend acceptance, two rejection. The authors resolved most issues. The paper seems to be of general interest, and the AC has sufficient confidence that the authors can (and will) address the few remaining concerns in a revision to the paper:

As summarized by Reviewer 35FE

> (1) the omission of critical technical details in the methods section;
>
> (2) numerous typos and grammatical errors;
>
> (3) overstatements of the model’s capabilities; and
>
> (4) insufficient explanation as to why pretraining on ScanNet is beneficial compared to ScanNet++. Instead, the authors only compare pretraining on ScanNet++ followed by fine-tuning on ScanNet.
Decision: Accept (poster)