ReferSplat: Referring Segmentation in 3D Gaussian Splatting
Authors: Shuting He
Guangquan Jie
Changshuo Wang
Yun Zhou
Shuming Hu
Guanbin Li
Henghui Ding
Venue: ICML 2025 oral (2025)
Keywords: 3D Gaussian Splatting
Referring Segmentation
ReferSplat
Primary area: applications->computer_vision

Abstract: We introduce Referring 3D Gaussian Splatting Segmentation (R3DGS), a new task that aims to segment target objects in a 3D Gaussian scene based on natural language descriptions, which often contain spatial relationships or object attributes. This task requires the model to identify newly described objects that may be occluded or not directly visible in a novel view, posing a significant challenge for 3D multi-modal understanding. Developing this capability is crucial for advancing embodied AI. To support research in this area, we construct the first R3DGS dataset, Ref-LERF. Our analysis reveals that 3D multi-modal understanding and spatial relationship modeling are key challenges for R3DGS. To address these challenges, we propose ReferSplat, a framework that explicitly models 3D Gaussian points with natural language expressions in a spatially aware paradigm. ReferSplat achieves state-of-the-art performance on both the newly proposed R3DGS task and 3D open-vocabulary segmentation benchmarks. Dataset and code are available at https://github.com/heshuting555/ReferSplat.

ReferSplat: Referring Segmentation in 3D Gaussian Splatting
Shuting He'
Guangquan Jie
Changshuo Wang? Yun Zhou
Shuming Hu
Guanbin Li
Henghui Ding ?
Abstract
Training
Testing
We introduce Referring 3D Gaussian Splatting Segmentation (R3DGS), new task that aims segment target objects in 3D Gaussian scene based on natural language descriptions, which often contain spatial relationships or object at- tributes. This task requires the model to identify newly described objects that may be occluded or not directly visible in novel view, posing a significant challenge for 3D multi-modal under- standing. Developing this capability is crucial for advancing embodied AI. To support research in this area, we construct the first R3DGS dataset, Ref-LERF. Our analysis reveals that 3D multi- modal understanding and spatial relationship mod- eling are key challenges for R3DGS. To address these challenges, we propose ReferSplat, frame- work that explicitly models 3D Gaussian points with natural language expressions in a spatially aware paradigm. ReferSplat achieves state-of- the-art performance on both the newly proposed R3DGS task and 3D open- vece-vocabylary segmenta- tion benchmarks. Dataset and code are available at https//ggttbbbcooo/hhhtgggg/...
Multi Images
Elephant not directly visible
Novel-view Image
Output Mask
Text
Sreve olgon polncd rechonee poopphiy roal sefher
The one standing on the table long nose"
Figure I. Referring 3D Gaussian Splatting Segmentation (R3DGS) aims at segmenting the target objects described by a given natural language descriptions within 3D Gaussian scene, requiring the model to identify newly descrihed ohjects that may he occluded or not directly visible in novel view.
more attention, particularly in open-vocabulary 3DGS seg- mentation (3DOVS) (Qin et al., 2024; Ye et al., 2025; Zhou et al., 2024b), which relies on fixed-pattern linguistic class names input for segmentation.
However, despite these advancements, free-form natural language interactions with 3D scenes remain underexplored. The ability to interpret and localize objects based on arbi- trary language descriptions is crucial for various real-world applications, such embodied AI (Shorinwa et al., 2024), autonomous driving (Gu et al., 2024), and VR/AR sys- tems (Jiang et al., 2024). To bridge this gap, we introduce a new task: Referring 3D Gaussian Splatting Segmenta- tion (R3DGS), aims at segmenting objects in a 3D Gaussian scene based on natural language expressions that typically encode spatial relationships and descriptive attributes. As shown in Fig. 1, R3DGS requires the model to identify newly described objects, even when occluded or not directly visible in the novel view, posing a significant challenge for 3D multi-modal understanding. To support future research in R3DGS, we construct Ref-LERF, a new dataset rich in complex and spatially grounded language expressions.
1. Introduction
3D Gaussian Splatting (3DGS) (Kerbl et al., 2023), re- cently proposed neural rendering technique, has attracted significant attention because of its fast training, real-time rendering capabilities, and explicit point-based represen- tation (Keetha al., 2024; Yang et al., 2024; Tang et al., 2024; Zhou al., 2024a). As 3DGS continues to advance, text-driven 3D scene understanding has attracted more and
'MoE Key Laboratory of Interdisciplinary Research of Com- putation and Economics, Shanghai University of Finance and Eeonomics, Shanghai, China 'Institute of Big Data, College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China "Nayaan Technological University, Singapore "Sun Yat-sen University, Guangzhou, China. Correspondence to: Henghui Ding <henghui.ding gmail.com>.
A straightforward baseline for R3DGS to adapt exist- ing open- -ecaaulayy 3D scene understanding methods by replacing short and simple class names with complex natu- ral language expressions. Existing open-vocabulary meth- ods primarily project semantic features onto images for pixel-level understanding, leveraging semantic knowledge from pre-trained 2D languae models (Radford et al.,
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

ReferSplat: Referring Segmentation in 3D Gaussian Splatting
features are refined through position-guided attention, ef- fectively aligning 3D Gaussian representations with text descriptions. Despite spatial awareness, sentences with similar semantics but different target objects may cause confusion thus degrade performance. To address this issue, we employ Gaussian- Text Contrastive Learning between positive Gaussian embeddings and text embeddings, where Gaussian embeddings are computed from selectively chosen positive Gaussian referring features. This helps the model differentiate fine-grained referring expressions, improving cross-modal understanding. Extensive experiments demon- strate that ReferSplat achieves state-of-the-un performance on both vocabulcyyy 3DGS segmentation and the newly proposed referring 3DGS segmentation tasks.
Rasterize
Semantic
Gamssians
Rendered Feature
Output Mask
Pseudo GT Feature
Text Query
The one sitting on a stool close to the apple"
Rasterize
Referring
Gaussians
Similarity Map
Rendered Mask
Psenda GT Mask
Text Query
The one sifting on stool close to the apple"
Training and Testing -osuee Testing Only
Loss Supervision
Figure Comparison of (a) existing open- vocabulary 3DGS seg- mentation pipeline and (h) the proposed ReferSplat for R3DGS.
In summary, our contributions are follows:
We introduce a new task termed Referring 3D Gaussian Splatting (R3DGS) and construct a new dataset Ref- LERF to support future research in R3DGS.
2021; Kirillov et al., 2023) as ground truth feature to guide 3D scene representation learning. During inference, output masks are obtained by matching the input open- open-vocabulary class names with the rendered feature, as shown in Fig. 2 (a). However, these methods face significant limitations when applied to R3DGS. One major draw bwwbbckk/n is the lack of inter- action between the text query and Gaussian representations during training. Existing methods rely on matching the text query with 2D rendered features instead of performing local- ization directly in 3D space, which limits their performance in complex scenarios. Moreover, this training paradigm overlooks position information, as rendered feature cannot inherently understand spatial relationships between objects described in sentence. Instead, it primarily focuses on semantic understanding, making it ineffective for tasks re- quiring spatial reasoning. This raises a question: Can we design network that models 3D Gaussians with language expressions in spatially aware paradigm?
To tackle the challenges of R3DGS, we propose Refer- Splat, a spatially aware framework that models 3D Gaussians based on natural language expressions, achieving sfate-of-the-art performance on Ref-LERF.
We construct 3D Gaussian Referring Fields, incor- porating high-quality pseudo masks generated via confidence-weighteed IoU strategy as ströng pipeline.
We design Position-aware Cross-Modis Interaction module, which integrates position information into the cross-modal interaction, enhancing spatial reasoning and feature alignment between 3D Gaussians and text.
We employ Gaussian- Text Contrastive Learning to im- prove the model's ability to generate discriminative multi-modal representations, effectively distinguishing semantically similar expressions.
2. Related Work
In this work, we propose ReferSplat, an end-to-end frame- work that models 3D Gaussian points with natural language expressions a spatially aware paradigm for Referring 3D Gaussian Splatting Segmentation (R3DGS). To enable lan- guage perception, assign each 3D Gaussian referring feature vector, forming referring fields that interact with text queries during training. Similarity is computed between textual features and 3D Gaussian referring features, and the rendered 2D segmentation mask is obtained accordingly, as shown in Fig. 2 (b). For segmentation supervision, we generate pseudo ground truth masks using confidence- weighted loU strategy. The constructed 3D Gaussian Re- ferring Fields enable the model to identify objects that are occluded or not directly visible by leveraging 3D scene knowledge learned from multi-view training images. To enhance spatial reasoning, we introduce a Position-aware Cross-Modal Interaction module that extracts position fea- tures for both Gaussians and language descriptions. These
3D Neural Representations. Recent developments in 3D representation have achieved notable progress, with Neural Radiance Fields (NeRF) (Mildenhall al., 2021) standing out for the ability generate high-quality novel view synthe- sis. Despite their effectiveness, NeRF's reliance on implicit neural networks can lead to extended training and rendering durations. To address these limitations, Instant-NGP (Müller et al., 2022) accelerates the process using hash encoding. Re- cently, 3D Gaussian Splatting (3DGS) (Kerbl et al., 2023) proposes an explicit way represent 3D scenes using a collection of 3D Gaussian distributions. Through the opti- mization of their spatial positions and visual attributes, this method achieves real-time, high-quality rendering. Since the introduction of 3DGS, its superior performance has attracted increasing attention, leading to numerous studies focusing on enhancing rendering quality (Li et al., 2025b; Yu et al., 2024a; Huang et al., 2024), improving scene reconstruc-
2

---

ReferSplat: Referring Segmentation in 3D Gaussian Splatting
tion accuracy (Yu et al., 2024b; Dai et al., 2024; Guédon
g = - ) where A is the number of Gaussians. Each Gaussian di is parameterized by its mean position b4 TR3, covariance matrix Σ; E g opacity o; E IR,, and color C E R where de = 3 for RGB color parameters.
& Lepetit, 2024), and feed-forward optimization (Xu al., 2025; Chen et al., 2025; Charatan et al., 2024). In this work, we leverage the strengths of 3D Gaussian Splatting as a foundation for 3D neural representations, using its explicit paradigm to achieve efficient and high-quality rendering.
To render an image, each 3D Gaussian is projected onto the 2D image plane, where it contributes to the pixel color based on its opacity. The color a given pixel v, denoted as C(v), is computed by blending the contributions of all Gaussians according to their opacity. This process is formulated as:
3D Segmentation in Gaussian Splatting, Inspired by the success of Gaussian Splatting (Kerbl et al., 2023; Xu et al., 2025; Chen et al., 2025), numerous studies have explored 3D segmentation within this framework. SAGA (Cen et al., 2025) introduces a promptable segmentation method lever- aging 3D Gaussian Splatting to highlight its potential in semantic understanding. Following this, various works (Hu et al., 2024; Choi et al., 2025; Jain et al., 2024) emerged, further advancing promptable segmentation techniques. In- tegrating foundation models such CLIP (Radford al., 2021) and SAM (Kirillov et al., 2023; Ravi et al., 2025), several approaches (Li et al., 2025a; Zhou et al., 2024b; Zuo et al., 2024; Ye et al., 2025; Wu et al., 2024b; Liang et al., 2024; Shi et al., 2024; Qu et al., 2024; Ji et al., 2024; Peng et al., 2025) extend Gaussian Splatting to semantic and open-vocabulary 3D segmentation. While these methods in- corporate some level of language perception, they primarily focus on category-level segmentation and struggle to com- prehend complex natural language queries. This gap under- scores the necessity for methods capable of understanding and segmenting objects based on nuanced linguistic cues.
C(o) = Σ qas [[(--a)), 1=1
where a is the color vector of the i-th Gaussian, and aq o G,GPP(v) (v), G? (v) is the 2D projection function.
3.2. Problem Statement and Method Overview
During training for Referring 3D Gaussian Splatting Seg- mentation (R3DGS), we are given set of RGB images I (1,)221 from S different training views, along with set of natural language expressions T = {Ti) where each Ti refers to an object visible in a single training view. At test time, the model is given a referring expression and a novel view, and is required to predict the corresponding object mask. The key challenge lies in segmenting the tar- get object in this unseen view, where it may be partially occluded or even entirely invisible. Unlike 2D referring segmentation, which operates on a single image and cannot resolve occlusions, R3DGS leverages the 3D scene learned from multi- supervision to reason about occluded ob- jects based on their visibility in other training views. Com- pared to 3D referring segmentation, which relies on explicit point clouds and annotated 3D masks, R3DGS aims to learn from multi- 2D training images without requiring an- notated mask supervision. This setting makes the proposed R3DGS task more practical for real-world applications.
Referring Segmentation. 2D Referring Expression Seg- mentation (RES) (Ding et al., 2023e; 2021; 2023a;b; Liu et al., 2023; Yang et al., 2022; Ding et al., 2025a;b) and 3D posnt-based Referring Expression Segmentation (3DRES) (Huang et al., 2021; He et al., 2024; 2023; He & Ding, 2024; Wang et al., 2024; 2025) aim to segment a tar- get object within a 2D image or 3D point cloud scene based on a natural language expression. In recent years, RES and 3DRES have seen significant progress, with Transformer- based architectures becoming the dominant choice. Among them, Grounded SAM (Ren et al., 2024) stands out as robust RES method in the 2D domain, integrating two state- DINO (Liu et al., 2025) and SAM (Kirillov et al., 2023). Besides, the emergence of point-based benchmarks such as ScanRefer (Chen et al., 2020) and Multi3DRefer (Zhang et al., 2023) has signifi- cantly accelerated the progress of 3DRES. However, despite these advances, existing methods and datasets remain con- strained to 2D or 3D basnt- representations and cannot be directly applied to R3DGS.
An overview of our proposed approach ReferSplat is shown in Fig. 3. The approach begins with the construction of 3D Gaussian Referring Fields. In Sec. 3.3, we extract word features fw and a sentence embedding fe from the referring expression Ti using a text encoder. To infuse language- awareness into the 3D Gaussians, we introduce new prop- erty called referring features. The segmentation mask is then obtained by modulating these referring features with word features fw. followed by rendering and supervision with the pseudo-mask generated. To enhance the interaction between referring features and word features fw, we intro- duce Position- aware Cross-Modal Interaction in Sec. 3.4, which strengthens the alignment between spatial and linguis- tic cues. Finally, in Sec. 3.5, we employ contrastive learning to distinguish between different text queries by enforcing semantic consistency between positive Gaussian embed- dings with fe, leading to a more robust language-guider
of-the-art Gondels-Groundee
3. Method
3.1. Preliminaries
In 3D Gaussian Splatting (3DGS) for RGB image rendering, a scene represented by a set of 3D Gaussian distributions
3

---

ReferSplat: Referring Segmentation in 3D Gaussian Splatting
3D Gaussian Referring Fields
Rasterize
fri
bce
Pseudo Mask
Generation
fw
M
M
Position
Similarity Map
Rendered Mask
Pscudo Mask
Multi-view Images
Colvariance
Opucity
Position-aware Cross-Modal Interaction
Gaussian- Text Contrastive Learning
Color
fp
fr.
fe
Reffrring
Top-t
fra
sofimar
f
con
3D Gaussians
p.w.
fw
volimas
Text Imput
The one sitting on a stool close to the apple
Text
fw Word Feature
Push Close
Multiplication
Selection
Encoder
000 Sentence Feature
Away
Figure 3. Overview of the proposed approach ReferSplat. Firstly. to infuse unguage-awareneee into the 3D Gaussians, we introduce new property called referring features, constructing 3D Gaussian Referring Fields. The segmentation mask obtained by modulating these referring features with word features fw. followed by rendering and supervision with the generated pseudo-mask. To enhance the interaction between referring features and word features fe. we introduce Position-aware Cross-Modal Interaction. Finally, we employ contrastive lcarning to distinguish semantically similar language expressions.
segmentation in 3D Gaussian scene.
and the rendered segmentation mask, computed
3.3. 3D Gaussian Referring Fields
M(O)
m,a II a1-aa)
(3)
Inspired by methods that incorporate semantic feature vec- tors construct semantic-aware fields (Qin al., 2024; Zhou al., 2024b; Qu et al., 2024), we introduce refer- ring feature vector fr, E dd for each 3D Gaussian, where d, denotes the feature dimension. This forms referring fields that equip 3D Gaussians with language perception capabilities. In our framework, the referring feature en- codes semantic and referring information, allowing us to compute the text response for each Gaussian by measuring similarity between the 3D Gaussian referring feature and the input text. Unlike LangSplat (Qin et al., 2024), which performs retrieval- based matching between rendered seman- tie features and text embeddings, we directly model the relationship between 3D Gaussians and text embeddings. Specifically, we compute the similarity between each Gaus- sian referring feature and the input sentence representation by aggregating responses across all words:
Finally, we employ binary cross-entropy (BCE) loss to supervise the output mask, enforcing consistency with the pseudo ground truth mask, which we introduce later:
Lice Σ L log + (1 9) log(1 -y)]
(4)
where ý represents the pseudo ground truth mask M and y denotes the predicted mask M(r) at pixel v.
Unlike traditional 3DGS, which primarily renders color val- ues predefined semantic features, our approach directly renders Gaussian-language similarity responses, enabling explicit interaction between textual descriptions and 3D scene representations. As shown in Tab. our method sur- passes existing approaches, establishing superior referring segmentation framework in 3D Gaussian scenes. Addition- ally, our constructed 3D Gaussian Referring Fields empower the model to recognize occluded or non-visible objects by leveraging 3D scene knowledge learned from multi-view training images.
m; Efss
fes x fwij
where m, E R. represents the overall Gaussian- language similarity score for the i-th Gaussian, and fu.j denotes the feature representation of the j-th word in the sentence. To ensure computational efficiency, both features are mapped to the same feature dimension, denoted as D.
Pseudo Mask Generation. To generate high-quality 2D pseudo masks, we input the image and referring expression into Grounded SAM (Ren et al., 2024). Since single object can be described by multiple language expressions, this pro- cess produces K candidate masks {M& |K= each assigned a confidence score that exceeds predefined threshold e. naive approach selects the nighest-confidenc mask, but this often fails confidence does not always correlate
Next, similar to traditional Gaussian Splatting, we apply rasterization process. Instead of rendering RGB values, we rasterize my. The response on the 2D image plane at pixel v, denoted M(r), serves as both the projected text response

---

ReferSplat: Referring Segmentation in 3D Gaussian Splatting
with correctness. Our analysis reveals that while the cor- rect mask is usually present, it does not always receive the highest confidence score.
mation, we first extract position features from 3D Gaussian representations. Specifically, the center coordinates (14 J of Gaussians are projected through an MLP layer to obtain position embeddings fp.i E R2, serving as position-ane aware representations of Gaussian referring features. Meanwhile, understanding spatial relationships from language descrip- tions important. However, unlike 3D Gaussians, text features lack explicit positional information. Since lan- guage expressions often describe objects in relation to one another, we infer text position information from the position embeddings of Gaussians by leveraging their cross-modal correspondence. To achieve this, we first compute the rela- tionship between word features fw and Gaussian referring features fr.i. Using this relationship. we extract the cor- responding position information from Gaussian position features fp.i, formulated as:
To improve selection, we first compute the Intersection over Union (loU) consistency across all K" candidate masks and select the most frequently occurring or overlapping mask. However, this purely IoU-based approach disregards con- fidence scores, potentially favoring low-quality masks. To address this, propose confidence weeiihence-weighte IoU strat- egy, defining an overlapping score ps as:
K Me n mrg M7, Pk =
Σ j=1 Ma U U,
(5)
where Me and M, are candidate masks with confidence scores "ys and 7y, respectively. This formulation prioritizes masks with both high IoU consistency and confidence. Fi- nally, we select the mask with the highest Ps: as the final pscudo-mask M. Our pseudo Mask Generation strategy significantly improves mask quality, enhancing accuracy and robustness in referring segmentation in 3D Gaussians.
frefe's VD
fp.u.s = softmax
Sp.
(6)
where D is the feature dimension, and fp.ent represents the position feature aligned with the word features. In this way, the model dynamically extracts text position information based on textual descriptions and Gaussian referring cues.
3.4. Position-aware Cross-Modal Interaction
The accuracy of the output mask is determined by the sim- ilarity score m; in Eq.2 that measures the relationship be- tween Gaussian referring features and text embeddings. To improve the accuracy of similarity score, it is essential to establish explicit mrosl interactions, facilitating in- formation exchange and effectively leveraging cross-modal dependencies for enhanced semantic understanding. Further- more, Gaussian referring features primarily capture seman- tic information, which helps recognize object categories and attributes. However, comprehending sentence like "the red object on the left" requires not only semantic recogni- tion ("red object") ")) but also spatial reasoning ("on the left") Therefore, position information, which represents spatial locations and object relationships, is crucial for accurately understanding language expressions and achieving precise object segmentation within 3D Gaussian representations.
Position-aware Attention for Feature Refinement. After obtaining position information fp, for Gaussian referring features and fp.w.i for text embeddings, we further refine the referring feature representation using a cosition-guided attention mechanism. We integrate structural geometry con- straints to guide attention computation:
fr, fr. + softmax (Ur++ Sp. + +spmms) Sw-
VD
This formulation ensures that the updated referring feature fra is enriched with both position and semantic cues, im- proving its ability to localize the target object. Besides, we retain the original fr, to preserve fine fine-grained details that may be lost in the attention process. Then, f", replace fra in Eq.2 to obtain more accurate text response. The proposed Position-aware Cross-Modal Interaction module establishes stronger relationship between Gaussian referring features and text descriptions by explicitly integrating position in- formation. This integration enhances spatial understanding, and strengthens moosl feature alignment, enabling more precise R3DGS.
To address these issues, we propose Position-aware Cross- Modal Interaction module that injects position information into the cross-modal attention mechanism to facilitate inter- actions between textual entities and 3D Gaussians beyond mere semantic aligmment. This module enables a richer and mutually informed fusion of spatial and semantic informa- tion, strengthening the alignment between 3D Gaussians and text embeddings. In this framework, position, semantics, and language are interdependent: position provides spatial context for semantics, while semantic information aids in understanding spatial relationships, ensuring a coherent and context-aware cross-modal representation.
3.5. Gaussian-Text Contrastive Learning
While the proposed Position-aware Cross-Modal Interac- tion module effectively captures the relationship between Gaussian representations and text descriptions, distinguish- ing between languages with similar meanings but referring to different objects remains a significant challenge. Such
Position Feature Extraction. To integrate position infor-
5

---

ReferSplat: Referring Segmentation in 3D Gaussian Splatting
ambiguities can lead confusion in referring segmentation, particularly in complex 3D Gaussian scenes. To address this issue, we introduce Gaussian-Text Contrastive Learn- ing in the Gaussian feature space. This learning paradigm enhances the model's ability to disambiguate similar lan- guage expressions that refer to different objects, enforcing a stronger Gaussian-language understanding and alignment for more robust referring segmentation.
JoJec
round
green
sur surracc race
redd
tabll
e shape
Sma
acee
smooth Fop neat
Word Cloud Ref-EER
Distribation tolcea length expresions
Figure 4. Dataset analysis of our constructed Ref-LERR.
A key challenge of Gaussian Text Contrastive Learning lies in extracting positive Gaussian referring features that accu- rately match the text description. In a typical 3D Gaussian scene, numerous Gaussians exist, and 3D mask annotations for precisely outlining the Gaussians corresponding to the text are unavailable. However, from 2q.2, we can obtain the response m, of the text on the 3D Gaussians and select Gaussian points with high similarity. Therefore, we con- sider Gaussian referring features whose m, are within the top-7 percentile as positive examples. The positive Gaus- sian embedding, denoted as fg. corresponding to the text is then obtained by averaging the features of these chosen Gaussian referring features:
4. Experiments
4.1. Ref-LERF Dataset and Evaluation Metries
The LERF dataset (Kerr et al., 2023) is collected using the Polycam iPhone app and consists of four diverse, com- plex, real-world scenes. It was originally developed for 3D object localization tasks. LangSplat (Qin et al., 2024) introduces ground truth (GT) mask annotations to enhance its complexity. forming the LERF-OVS dataset for 3D open- vocabulary segmentation. We introduce Ref-LERF, incor- porating language expression annotations to further expand its capabilities to enable R3DGS evaluation. Each scene contains approximately five expressions per object, with 236 language descriptions used for training and 59 for testing, totaling 295 descriptions for 59 objects. Besides, annota- tions emphasize positional relationships, providing richer contextual grounding for precise object segmentation.
fo E.E Sr. SEN,
Nr
where N, denotes the set of indices corresponding to the top-T percentile of mi, and N, is its cardinality.
After obtaining paired Gaussian embedding and text embed- ding, we employ object-wise contrastive learning:
Dataset Analysis. The word cloud of our newly proposed Ref-LERF dataset, visualized in Fig. 4 (a), highlights its richness in spatial and detailed descriptions. A significant portion of the dataset consists of relative position words such as "placed", "placed",""near"" "'nar"", and "next", as well as fine-grained object attributes like "round" and "surface". This demon- strates that Ref-LERF places a stronger emphasis on spatial reasoning and detailed object understanding compared to previous datasets. Furthermore, illustrated in Fig. 4 (b), Ref-LERF presents significantly greater challenge than LERF-OVS, featuring more complex sentences and richer object descriptions. The average sentence length exceeds 13.6 words, making it approximately eight times longer than those in LERF-OVS. This increased complexity ensures that models trained on Ref-LERF must develop a deeper under- standing of language and spatial relationships, making it more realistic and comprehensive benchmark for R3DGS.
1 P
explfy ff) JJEP EREPNN exp(S= -2)
Loon
log
(9)
where P is the set for positive textual samples with positive Gaussian embedding and N represents the set of negative textual samples, which are drawn from different text queries in the scene. This formulation encourages the model to max- imize similarity between Gaussians and their correspond- ing textual descriptions while ensuring sufficient separation from unrelated textual descriptions, ultimately enhancing accurate referring segmentation in 3D Gaussian Splatting. The total training objective is:
Lioss Lhoe + XLcons
(10)
where À is used for balancing the contrastive loss Loo
Evaluation Metrics. The average IoU (mloU) is calculated between the masks rendered from the text response on the 3D Gaussians and the GT object masks.
Following (Mirzaci et al., 2023), we adopt a two-stage op- timization approach to further refine segmentation masks. Instead of using the pseudo masks from Sec. 3.3, we lever- age the rendered masks generated by our trained ReferSplat model in the first stage to supervise a secondary ReferSplat model. This progressive refinement provides more accurate initial masks, improving segmentation performance.
4.2. Implementation Details
We extract text embeddings for each sentence using BERT (Devlin et al., 2019). Following the default configura- tion of LangSplat (Qin et al., 2024), we first train the RGB
6

---

ReferSplat: Referring Segmentation in 3D Gaussian Splatting
Table 1. Ablation study our method. PCMI, and GTCI, de-
Table 3. Ablation study Pseudo Mask Generation.
note components of Position- aware Cross-Modal Interaction, and Gaussian- Text Contrastive Learning, respectively.
Results
Mask Quality
Method
ramen
kitchen
ramen
kitchen
Components
Results
Top-1
23.1
15.7
47.1
42.8
Index
PCMI
GTCL
ramen
kitchen
SAM2
22.6
15.5
41.2
37.8
Baseline
28.4
18.5
IoU w/o
25.3
17.1
48.2
45.3
33.5
22.8
IoU w/ y (Ours)
28.4
18.5
52.9
49.7
X
32.8
21.9
Ours
35.2
24.4
Two-stage
36.9
25.2
Table 4. Ablation study on Cross-Modal Interaction Design.
Results
Method
ramen
kitchen
Table 2. Ablation study on Baseline Configuration.
w/o fp and ppwws
27.9
18.1
Results
w/o fpows
29.2
18.8
Method
ramen
kitchen
Sp.s + frys
30.3
20.5
LpngSplat
12.0
17.9
w/ fpss and fp.wd (Ours)
33,5
22.8
SPIn-NeRF
7.3
10.3
Cosine Similarity
25.4
16.8
Multiplication (Ours)
28.4
18.5
our two-stage optimization in Sec. 3.5 further refines the generated masks through ReferSplat trained in the previ- ous stage, improving 3D scene understanding and ensuring more consistent and reliable segmentation results.
representation of the 3D scene and then freeze its parame- ters before training the components of our proposed model. We optimize the Gaussian referring features for 45,000 it- erations, using a learning rate of 0.0025, while other pa- rameters, such as the MLP, are trained with a learning rate of 0.0001. The Adam optimizer (Kingma & Ba, 2015) is used for optimization. To enable gradient pack-propagation through extended referring feature attributes, we modify the CUDA kernel to render referring features on 3D Gaussians. Training is conducted on an NVIDIA RTX A6000 GPU. For hyper-parameter optimization, we set dr. D, e, and À to 16, 128, 0.3, and 0.02, respectively. While J varies with training, following the schedule: т = 0.1 x 0.6 (iterationn /1000).
Baseline Evaluation. We set several baselines for compari- son: 1) LangSplat Adaptation: we modify LangSplat (Qin et al., 2024) from open-vocabulary segmentation to refer- ring segmentation by replacing phrase inputs with referring expressions during testing. 2) SPIn-NeRF Adaptation: adapt SPIn-NeRF (Mirzaei et al., 2023) from prompt-based segmentation to referring segmentation by incorporating text input and summing textual features with original se- mantic features to generate masks. 3) Cosine Similarity: we compute similarity m, in Eq.2 using cosine similarity for text response. The results in Tab. 2 show that our method significantly outperforms all baselines, demonstrating that the proposed 3D Referring Feature Fields effectively models the relationship between 3D Gaussians and text.
4.3. Ablation Study
The following ablation studies are conducted on the ramen and kitchen scenes on the Ref-LEERF dataset.
Pseudo Mask Generation. To assess the quality of the gen- crated pseudo labels and support further research, we manu- ally annotate ground truth masks for comparison. As shown in Tab. 3, our method achieves about 50% mloU against ground truth, validating the effectiveness of our pseudo mask generation. In contrast, alternative approaches-such as using the top-1 prediction, propagating the first-frame mask with SAM2 (Ravi et al., 2025), or selecting masks solely based on IoU without confidence weighting- yield inferior results. These findings underscore our superiority.
Module Effectiveness. We conduct ablation experiments to evaluate the effectiveness of different components. As shown in Tab. 1, incorporating PCMI (index 1) improves mIoU by 5.1% and 4.3%, respectively compared to the baseline, which is our constructed Referring Feature Fields. This demonstrates that PCMI enhances spatial reasoning and strengthens feature alignment between 3D Gaussians and text. Next, we introduce Gaussian- Text Contrastive Learning (GTCL, index 2) to construct discriminative multi- modal representations, improving the model's 's ability to distinguish semantically similar expressions. Incorporating GTCL leads to 4.4% and 3.4% mloU improvement, respec- tively. When integrating all components (index 3), referred to as ReferSplat, we achieve a substantial performance gain, reaching a new state-of tfte--of-hh--ar. art- This result demonstrates the effectiveness of our proposed approach. Furthermore,
Position-aware Cross-Modal Interaction Design. We con- duct experiments to evaluate the impact of different cross- modal interaction designs based on baseline. As shown in Tab, 4, removing components fp.s and fp.w, from Eq.7 results in performance dropping below the baseline, indicat- ing that vanilla cross-attention alone is ineffective for our task. Adding fp. without fp.w, still yields unsatisfactory
7

---

ReferSplat: Referring Segmentation in 3D Gaussian Splatting
Table 5. Analysis of Computation Costs.
Table 7. Ablation study number of feature dims.
Method
Training
FPS
Storage
mloU
Results
LangSplat
176min
12.4
46MB
13.9
Number
ramen
kitchen
GS-Grouping
66min
54.2
2.3MB
14.4
21.7
13.2
ReferSplat (Ours)
58min
26.8
3.3MB
29.2
4
23.1
15.4
16
28.4
18.5
32
27.2
16.9
Table 6. Choice of Language Encoder.
Method
ram.
fig.
tea.
kit.
avg.
Table R3DGS result on the Ref-LERF dataset.
CLIP
23.5
23.2
26.2
21.0
23.5
Method
ram.
fig.
tea.
kit.
avg.
BERT
35.2
25.7
31.3
24.4
29.2
Grounded SAM
14.1
16.0
16.9
16.2
15.8
LangSplat
12.0
17.9
7.6
17.9
13.9
SPIn-NeRF
7.3
9.7
11.7
10.3
9.8
results, suggesting that both components contribute to ef- fective interaction. Additionally, analyze the impact of directly incorporating position information fp. into the fr Although this improves performance, it remains less effec- tive than our attention-based position modeling. Therefore, our proposed method is crucial for accurate R3DGS.
GS-Grouping
27.9
14.8
6.3
14.4
GOI
27.1
16.5
22.9
15.7
20.5
ReferSplat (Ours)
35.2
25.7
31.3
24.4
29.2
Grounded SAM (Ren et al., 2024) and 3D-based approaches such as LangSplat (Qin et al., 2024) and Gaussian Group- ing (Ye al., 2025), demonstrating its effectiveness. Our 3D Gaussian Referring Fields enable the model to recognize occluded or non- visible objects by leveraging multi-vies 3D scene knowledge- an inherent limitation of 2D-based meth- ods. While Grounded-SAN generates high-quality masks during training, it is restricted to visible objects within a single view (Sec. 3.2). Additionally, our position-awars aware cross-modal interaction and contrastive learning enhance spatial reasoning and feature alignment, improving language comprehension in complex 3I) environments.
Analysis of Computation Costs. We conduct experiments on the ramen scene from the Ref-LERF dataset using the same NVIDIA A6000 GPU to compare the computational cost of our ReferSplat against SOTA methods in Tab. 5. Results show that ReferSplat achieves significantly lower computational complexity and faster inference speed than SpngSplat (Qin et al., 2024). While GS- SSSGruupi (Ye et al., 2025) excels in storage and FPS, ReferSplat outperforms all methods in segmentation performance. ReferSplat also has the shortest training time, thanks to a lightweight pre- processing pipeline that avoids costly operations like lan- guage feature compression (LangSplat) or mask association with video tracking methods (GS- (GS-Grouping). These results demonstrate that ReferSplat's 's compact, efficient design is well-suited for real-world applications.
Qualitative Visualization. Fig. 5 qualitatively presents that ReferSplat effectively captures the spatial relationship between Gaussian points and text, enabling superior segmen- tation even in challenging scenarios with heavy occlusion or non-visible objects, as illustrated in (a) and (b).
Choice of Language Encoder. we conduct experiments comparing BERT and CLIP embeddings for language fea- tures in R3DGS in Tab. 6. Results show that BERT con- sistently outperforms CLIP. This is likely because CLIP focuses more on noun categories, while referring expres- sions often involve spatial and attribute- based descriptions.
4.5. 3D Open- Vocabulary Segmentation Result
To further validate our method, we evaluate it on 3D open- vocabulary segmentation benchmarks, as shown in Tab. and 10. Despite not being specifically designed for 3DOVS tasks, our approach achieves state-of-the-ard performance. This can be attributed to the integration of 3D Gaussian Re- ferring Fields, position-aware cross- modal interaction, and gaussian-text contrastive learning, which enhance spatial reasoning and feature alignment, significantly improving language comprehension in complex 3D environments.
Impact of Referring Feature Dimension. We study the effect of the referring feature dimension d, in Tab. 7 for each 3D Gaussian. In our experiments, we set d, to 1, 4, 16, and 32, and find that 16 achieves the best results. Smaller dimensions (e.g., 1 or 4) lack the capacity to store discriminative features, while larger dimensions (e.g.. introduce redundancy and noise, degrading performance.
5. Conclusion
We introduce Referring 3D Gaussian Splatting Segmenta- tion (R3DGS), new task for segmenting target objects in 3D Gaussian scenes using natural language descriptions involving spatial relations and object properties. To sup- port research in this area, we construct the first R3DGS
4.4. Results on the Ref-LERE Dataset
Quantitative Results. We evaluate ReferSplat against state- the- 2D and 3D methods on the Ref-LERF dataset in Tab. ReferSplat outperforms 2D-based methods like
8

---

ReferSplat: Referring Segmentation in 3D Gaussian Splatting
RGB
Grounded-SAN
LangSplat
GaussianGrouping
Ours
Ground Truth
figurines
(a)
"A brightly colored toy placed next to the box and next to the green chair."
ramen
bb)
A container with water resting wooden tabletop and near wall." "
teatime
(c)
"Round object behind a sheep with a soft surface and near a concrete pillar." "
kitchen
(d)
"appliance used refrigerate food with various decorations plastered all over the surface.
Figure Qualitative R3DGS comparisons on the Ref-LERF dataset. Blue masks represent spatial descriptions.
Table Open- vocabbocyyy segmentation result the LERF-OVS.
Table 10. Open-vocabulary segmentation result the 3D-OVS.
Method
ram.
fig.
tea.
kit.
avg.
Method
bed bench room
scfa
lawn
avg.
Feature-3DGS
43.7
58.8
40.5
39.6
45.7
Feature-3DGS 83.5
90.7
84.7
86.9
93.4
87.8
LEGaussians
46.0
60.3
40.8
39.4
46.6
LEGaussians
84.9
91.1
86.0
87.8
92.5
88.5
LangSplat
51.2
65.1
44.7
44.5
51.4
LangSplat
92.5
94.2
94.1
90.0
96.1
93.4
GS- Ss-Grruping
45.5
60.9
40.0
38.7
46.3
GS-Grout (S--Gruping
83.0
91.5
85.9
87.3
90.6
87.7
GOI
52.6
63.7
44.5
41.4
50.6
GOI
89.4
92.8
91.3
85.6
94.1
90.6
ReferSplat (Ours)
55.1
67.5
50.1
48.9
55.4
ReferSplat (Ours)
94.8
94.6
91.8
dataset Ref-LERF. To address R3DGS challenges, we pro- pose ReferSplat which spatially models 3D Gaussian points based on natural language expression. Experiments show that the proposed ReferSplat achieves state-of-the-an per- formance on both R3DGS and 3DOVS tasks.
limited number of scenes, which restricts the model's abil- ity to achieve the same robust generalization as 2D-based approaches. In the future, we aim to construct a large-scale dataset, enabling better scene diversity and representation learning, thereby advancing research in this field.
Impact Statement
6. Limitation and Future Work
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
1) Our current method does not account for dynamic factors, which are crucial for real-world applications. Integrating our approach with 4D Gaussian Splatting (4DGS) (Wu et al., 2024a) could enhance its capability handle temporal vari- ations and dynamic environments. 2) While we focus 3D referring segmentation in Gaussian Splatting, our method does not incorporate 3D visual grounding. Extending our framework to support precise object size estimation could further improve its applicability in spatial reasoning and real- world localization tasks. 3) The current dataset includes
A Akknowledgements
This project was supported by the National Natural Science Foundation of China (NSFC) under Grant No. 62472104, Shanghai Pujiang Programme 24PJD030, Natural Science Foundation of Shanghai 25ZR1402138 138, and partially sup- ported by NSFC Grant No. 62322608.

---

ReferSplat: Referring Segmentation in 3D Gaussian Splatting
References
Ding, H., Liu, C., He, S., Ying, K., Jiang, X., Loy, C. C., and Jiang, Y.GG. MeViS: A multi- dataset for refer- ring motion expression video segmentation. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 2025a.
Cen, J., Fang, J., Yang. C., Xie, L., Zhang, X., Shen, W., and Tian, Q. Segment any 3d gaussians. In Proceedings of the AAAI conference on artificial intelligence, 2025.
Charatan, D., Li, S. L., Tagliasacchi, A., and Sitzmann, V. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19457-19467, 2024.
Ding, H., Tang, S., He, S., Liu, C., Wu, Z. and Jiang, Y.- G. Multimodal referring segmentation: A survey. arXiv, 2025b.
Gu, Q. Kuwajerwala, A., Morin, S., Jatavalaabbuula, K. M., Sen, B., Agarwal, A., Rivera, C., Paul, W. Ellis, K., Chellappa, R., et al. Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 5021-5028. IEEE, 2024.
Chen, D. Z., Chang, A. X., and Nießner, M. Scanrefer: 3d object localization in rgb-d scans using natural language. In European conference on computer vision, pp. 202-221. Springer, 2020.
Chen, Y., Xu, H., Zheng, C.,, Zhuang, B., Pollefeys, M., Geiger, A., Cham, T.-J., and Cai, J. Mysplat: Efficient 3d gaussian splatting from sparse multi-view images. In European Conference on Computer Vision, pp. 370-386. Springer, 2025.
Guédon, A. and Lepetit, V. Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high- quality mesh rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5354-5363, 2024.
Choi, S., Song, H., Kim, J., Kim, T., and Do, H. Click- gaussian: Interactive segmentation to any 3d gaussians. In European Conference on Computer Vision, pp. 289- 305. Springer, 2025.
He, S. and Ding, H. Refmask3d: Language-guider trans- former for 3d referring segmentation. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 8316 8325, 2024.
Dai, P., Xu, J., Xie, W., Liu, X., Wang, H., and Xu, High- quality surface reconstruction using gaussian surfels. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-11, 2024.
He, S., Jiang, X., Jiang, W., and Ding, H. Prototype adap- tion and projection for few-and zero-shot 3d point cloud semantic segmentation. IEEE Transactions Image Processing, 32:3199-322 3211, 2023.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: pre-training of deep bidirectional transformers for lan- guage understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pp. 4171-4188 2019.
He, S., Ding, H., Jiang, X., and Wen, B. Segpoint: Segment any point cloud via large language model. In European Conference on Computer Vision, pp. 349-367. Springer, 2024.
Ding, H., Liu, C., Wang. S., and Jiang. X. Vision-language transformer and query generation for referring segmen- tation. In Proceedings of the IEEF/CVF international conference on computer vision, pp. 16321-16330, 2021.
Hu, X., Wang. Y., Fan, L.,, Fan, J., Peng, J., Lei, Z., Li, Q. and Zhang, Z. Semantic anything in 3d gaussians. arXiv preprint 178877 2024.
Huang, B., Yu, Z.,, Chen, A., Geiger, A., and Gao, S. 2d gaussian splatting for geometrically accurate radiance fields. In ACM SIGGRAPH 2024 conference papers, pp. 1-11, 2024.
Ding, H., Liu, C., He, S., Jiang, X., and Loy, C. C. MeViS: A large-scale benchmark for video segmentation with motion expressions. In Proceedings of the IEEE/CVF International Conference Computer Vision, 2023a.
Ding, H., Liu, C., He, S., Jiang, X., Torr, P. H., and Bai, S. MOSE: A new dataset for video object segmentation in complex scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023b.
Huang, P.-H., Lee, H.-H., Chen, H.-T., and Liu, T.-... Text- guided graph neural networks for referring 3d instance segmentation. In Proceedings of the AAAI conference on artificial intelligence, pp. 1610-1618, 2021.
Ding, H., Liu, C., Wang, S., and Jiang, X. VLT: Vision- language transformer and query generation for referring segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(6))7900-7916, 2023c
Jain, U., Mirzaci, A., and Gilitschenski, I. Gaussiancut: Interactive segmentation via graph cut for 3d gaussian splatting. In Advances in Neural Information Processing Systems, 2024.
10

---

ReferSplat: Referring Segmentation in 3D Gaussian Splatting
Ji, Y., Zhu, H., Tang, J., Liu, w., Zhang, Z., Xie, Y., and Tan, X. Fastigs: Speeding up language embedded gaussians with feature grid mapping. In Proceedings of the AAAI conference on artificial intelligence, 2024.
Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Jiang, Q., Li, C., Yang, J., Su, H., et al. Grounding dino: Marry- ing dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision, pp. 38-55. Springer, 2025.
Jiang, Y., Yu, C., Xie, T., Li, X., Feng, Y., Wang, H., Li, M., Lau, H., Gao, F., Yang, Y., et al. Vr-gs: A physical dynamics-aware interactive gaussian splatting system in virtual reality. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-1, 2024.
Mildenhall, B.., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., and Ng, R. Nerf: Representing scenes as neural radiance fields for view synthesis. Communica- tions of the ACM, 65(1):99-106, 2021.
Keetha, N., Karhade, J., Jatavallabhula, K. M., Yang, G., Scherer, S., Ramanan, D., and Luiten, Splatam: Splat track & map 3d gaussians for dense rgb-d slam. In Pro- ceedings of the IEEE/CVF Conference Computer Vi- sion and Pattern Recognition, pp. 21357-21366, 2024.
Mirzaei, A., Aumentado-Armstee
T., Derpanis, K. G., Kelly, J., Brubaker, M. A., Gilitschenski, I., and Levin- shtein, A. Spin-nerf: Multiview segmentation and percep- tual inpainting with neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20669-20679, 2023.
Kerbl, B., Kopanas, G., Leimktihler, T., and Drettakis, G. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139-1, 2023.
Müller, T., Evans, A., Schied, C., and Keller, A. Instant neural graphics primitives with multiresolution hash encoding. ACM transactions on graphics (TOG), 41(4): 1-15, 2022.
Kerr, J.. Kim, C. M., Goldberg, K., Kanazawa, A., and Tancik, M. Lerf: Language embedded radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19729-19739, 2023.
Peng, Q. Planche, B., Gao, Z., Zheng, M., Choudhuri, A., Chen, T., Chen, C., and Wu, Z. 3d vision-language gaussian splatting. In Proceedings of the International Conference on Learning Representations, 2025.
Kingma, D. and Ba, Adam: A method for stochastic opti- mization. In Proceedings of the International Conference on Learning Reprexentations, 2015.
Qin, M., Li, W., Zhou, J., Wang, H., and Pfister, H. Langsplat: 3d language gaussian splatting. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20051 20051-20060, 2024.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C.,, Gustafson, L.,, Xiao, T., Whitehead, S., Berg, A. C.,, Lo, W.-Y., et al. Segment anything. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 4015-4026, 2023.
Qu, Y.. Dai, S., Li, X.. Lin, J., Cao, L., Zhang, S., and Ji, R. Goi: Find 3d gaussians of interest with an opti- mizable open-vocabulary semantic-spunn hyperplane. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 5328-5337, 2024.
Li, H., Wu, Y., Meng. J., Gao, Q., Zhang, Z., Wang, R., and Zhang, J. Instancegaussian: Appearance-semantic joint gaussian representation for 3d instance-level percep- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025a.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. $748-8766 PMLR, 2021.
Li, Y., Lyu, C., Di, Y., Zhai, G., Lee, H., and Tombari, F. Geogaussian: Geometry-aware gaussian splatting for scene rendering. In European Conference Computer Vision, pp. 441-457. Springer, 2025b.
Ravi, N., Gabeur, V., Hu, Y.-T., Hu, R., Ryali, C., Ma, T. Khedr, H., Rädle, R., Rolland, C., Gustafson, L., et al. Sam 2: Segment anything in images and videos. In Proceedings of the International Conference on Learning Representations, 2025.
Liang, S., Wang, S., Li, K., Niemeyer, M., Gasperini, S., Navab, N., and Tombari, F. Supergseg: Open-vocabulary 3d segmentation with structured super-gaussians. arXiv preprint ar a4Xiv:2412. 10231, 2024.
Liu, C., Ding, H., and Jiang, X. GRES: generalized re- ferring expression segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23592-23601, 2023.
Ren, T., Liu, S., Zeng, A., Lin, J., Li, Cao, H., Chen, J., Huang, X., Chen, Y., Yan, F., et al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint 155,, 2024.

---

ReferSplat: Referring Segmentation in 3D Gaussian Splatting
Shi, J.-C., Wang, M., Duan, H.-B., and Guan, S.-H. Lan- guage embedded 3d gaussians for open-vocahulary scene understanding. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 5333-5343, 2024.
Ye, M., Danelljan, M., Yu, F., and Ke, L. Gaussian grouping: Segment and edit anything in 3d scenes. In European Conference on Computer Vision, pp. 162-179. Springer, 2025.
Yu, Z., Chen, A., Huang, B., Sattler, T., and Geiger, A. Mip- splatting: Alias-free 3d gaussian splatting. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 19447-19456, 2024a.
Shorinwa, O., Tucker, J., Smith, A., Swann, A., Chen, T., Firoozi, R., Kennedy, M. D., and Schwager, M. Splat- mover: Multi-stage, open-vocabulary robotic manipula- tion via editable gaussian splatting. 8th Annual Con- ference on Robot Learning. 2024.
Yu, Z., Sattler, T., and Geiger, A. Gaussian opacity fields: Efficient adaptive surface reconstruction in unbounded scenes. ACM Transactions Graphics (TOG), 43(6): 1-13, 2024b.
Tang, J., Ren, J., Zhou, H., Liu, Z., and Zeng. Dream- gaussian: Generative gaussian splatting for efficient 3d content creation. In Proceedings of the International Conference Learning Representations, 2024.
Zhang, Y., Gong, Z., and Chang, A. X. Multi3drefer: Grounding text description multiple 3d objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 15225-15236, 2023.
Wang, C., Wu, M., Lam, S.-K., Ning, X., Yu, S., Wang, R., Li, W., and Srikanthan, T. Gpsformer: A global perception and local structure fitting-baser transformer for point cloud understanding, In European Conference on Computer Vision, pp. 75-92. Springer, 2024.
Zhou, H.,, Shao, J., Xu, L., Bai, D., Qiu, W., Liu, B., Wang, Y., Geiger, A., and Liao, Y. Hugs: Holistic urban 3d scene understanding via gaussian splatting. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 21336-21345, 2024a.
Wang. C., He, S., Fang, X., Wu, M., Lam, S.-K., and Tiwari, P. Taylor series-inspired local structure fitting network for few-shot point cloud semantic segmentation. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 7527-7533, 2025.
Zhou, S., Chang, H., Jiang, S., Fan, Z., Zhu, Z.. Xu, D., Chari, P., You, S., Wang, Z., and Kadambi, A. Feature 3dgs: Supercharging 3d gaussian splatting to enable dis- tilled feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21676-21685 2024b.
Wu, G., Yi, T., Fang. J., Xie, L., Zhang, X., Wei, W., Liu, W., Tian, Q., and Wang, X. 4d gaussian splatting for real-time dynamic scene rendering. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 20310-20322 2024a.
Zuo, X., Samangouei. P., Zhou, Y., Di, Y., and Li, M. Fmgs: Foundation model embedded 3d gaussian splatting for holistic 3d scene understanding. International Journal of Computer Vision, pp. 1-17, 2024.
Wu, Y., Meng. J., Li, H., Wu, C., Shi, Y., Cheng, X., Zhao, C., Feng, H., Ding, E., Wang, J., et al. Opengaussian: Towards point-level 3d gaussian-bart
open vocabulary understanding. In Advances in Neural Information Pro- cessing Systems, 2024b.
Xu, H., Peng, S., Wang, F., Blum, H., Barath, D., Geiger, A., and Pollefeys, M. Depthsplat: Connecting gaussian splatting and depth. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.
Yang. Z., Wang, J., Tang, Y., Chen, K. Zhao, H., and Torr, P. H. Lavt: Language-aware vision transformer for referring image segmentation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Puttern Recognition, pp. 18155-18165, 2022.
Yang, 7., Gao, X., Zhou, W., Jiao, S., Zhang, Y., and Jin, X. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20331-20341, 2024.
12


Reviews:
[{"review_id": "gQ8799wHPc", "reviewer": "Reviewer_t3nW", "summary": "- This paper introduces Referring 3D Gaussian Splatting Segmentation (R3DGS), a task aimed at segmenting target objects in a 3D Gaussian scene based on natural language descriptions.\n- The proposed method addresses key challenges, including identifying occluded objects in novel views.\n- The authors present Ref-LERF, for the proposed task.\n- The framework integrates 3D Gaussian referring fields, a position-aware cross-modal interaction module, and Gaussian-Text Contrastive Learning to improve spatial reasoning and enhance fine-grained understanding of natural language descriptions.\n\n\n## update after rebuttal\nI maintain my original rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your positive feedback on our work: dataset contribution, SOTA result, relationship modeling, and exhaustive ablations.\n\n>**Q1&Q7: Detailed evidence like video**\n\n**A1:** We have provided additional qualitative video results at **[ReferSplat.mp4](https://anonymous.4open.science/api/repo/ReferSplat-779/file/ReferSplat.mp4)**, which include:\n* Cases involving ambiguous language queries\n* Cases with incomplete language input\n* Performance under different novel views\n* Performance on significant viewpoint and perspective shift\n\nThese videos demonstrate our method’s ability to handle diverse and challenging referring expressions, showcasing its robustness to ambiguity and incomplete descriptions. The qualitative results further validate the effectiveness of our method and its potential for real-world applications.\n\n>**Q2: Computational costs**\n\n**A2:** We conduct experiments on the ramen scene from the Ref-LERF dataset using the same NVIDIA A6000 GPU to compare the computational cost of our ReferSplat against SOTA methods. Results show that ReferSplat achieves significantly lower computational complexity and faster inference speed than LangSplat. While GS-Grouping excels in storage and FPS, ReferSplat outperforms all methods in segmentation performance. ReferSplat also has the shortest training time, thanks to a lightweight preprocessing pipeline that avoids costly operations like language feature compression (LangSplat) or mask association with video tracking methods (GS-Grouping). These results demonstrate that ReferSplat’s compact, efficient design is well-suited for real-world and large-scale 3D applications. We will include these comparisons in the revision.\nMethod|Training↓ |FPS↑|Storage↓|mIoU↑|\n-|-|-|-|-|\nLangSplat|176min|12.4|46MB|13.9|\nGS-Grouping|66min|**54.2**|**2.3MB**|14.4|\n**ReferSplat**|**58min**|26.8|3.3MB|**29.2**|\n\n>**Q3: Difficult with sudden viewpoint changes**\n\n**A3:** This is a broader challenge in the field, not specific to our method. Baselines like LangSplat and GS-Grouping also show performance degradation under significant viewpoint changes and perspective shifts, as shown in the **[ReferSplat.mp4](https://anonymous.4open.science/api/repo/ReferSplat-779/file/ReferSplat.mp4)**. Future work will focus on developing robust multimodal representations and improving global scene understanding to enhance robustness under extreme viewpoint variations.\n\n>**Q4: Elephant visible?**\n\n**A4:** While the elephant is clearly visible in some training views, it is **not directly visible** from the **novel camera viewpoint** during inference in Fig. 1. We highlight this to emphasize a key difference from 2D referring expression segmentation (RES), which relies solely on single-view information and struggles to handle such invisible scenarios. In contrast, the proposed ReferSplat leverages multi-view information to construct a holistic 3D scene representation, enabling robust reasoning even when target objects are occluded or not directly visible. For further clarification, please refer to L128–137 (second column). We will revise the corresponding description to make this point clearer.\n\n>**Q5: Messy Rooms dataset evaluation**\n\n**A5:** Drawing from experience in 2D/3D RES tasks, models trained on sufficiently diverse datasets can generalize well and exhibit strong zero-shot capabilities, as exemplified by models like Grounded-SAM. As shown in Appendix Tab. 11, our model shows promising generalization results under the generalized training setting. In particular, it generalizes well to unseen scenes on the ScanNet dataset, which contains numerous objects and diverse scene layouts (see Reviewer 2Vdk Q4). Building upon this foundation, joint training on large-scale, diverse datasets such as ScanRefer, Ref-LERF, and MultiRefer presents a viable path toward adapting our model to more complex environments like Messy Rooms, where manual annotation and detailed descriptions are impractical. The above analysis highlights the scalability and cross-dataset generalization potential of our approach in real-world applications.\n\n>**Q6: Pseudo mask error**\n\n**A6:** To reduce the impact of errors from pseudo masks, we employ a two-stage optimization strategy (as described in L310-316), iteratively refining mask predictions during training. As shown in Tab. 1 of the main paper, this two-stage strategy outperforms the one-stage pipeline, demonstrating its effectiveness in mitigating the impact of noisy pseudo masks and ensuring robust overall performance. It is worth noting that the default results are based on the one-stage pipeline for a fair comparison with previous methods.", "claims_and_evidence": "The authors extend the existing LeRF dataset by incorporating expressive annotations, providing five descriptions of varying token lengths for each object. Figure 4(b) showcases the dataset’s increased complexity.", "methods_and_evaluation_criteria": "Yes, the proposed baselines and benchmark datasets are logical.", "theoretical_claims": "There is no proof in the manuscript.", "experimental_designs_or_analyses": "Yes, ablation studies in Section 4.3 discuss different design choices.", "relation_to_broader_scientific_literature": "This paper advances the challenge of 3D segmentation by integrating natural language descriptions for object identification, even in cases where objects are occluded or invisible from a single view. To support this task, the authors introduce Ref-LERF, a novel dataset specifically designed for language-guided 3D segmentation. This contribution is valuable to the scientific community, enabling more robust and interpretable scene understanding.", "essential_references_not_discussed": "All the related references are cited in the manuscript.", "other_strengths_and_weaknesses": "**Strengths**\n- **[S1] Dataset Contribution**: The creation of the Ref-LERF dataset provides valuable resources for future research for the proposed task.\n\n- **[S2] Outperforms SOTA methods**: The proposed model achieves state-of-the-art performance on the newly introduced R3DGS task and existing 3D open-vocabulary segmentation benchmarks.\n\n- **[S3]** Unlike retrieval-based matching between rendered semantic features and text embeddings, the relationship is directly modelled in the proposed method.\n\n- **[S4] Exhaustive ablations**: The authors provide thorough details regarding design choices, including the number of input views, the dimension of referring features and selection strategies for positive referring features.\n\n\n**Weaknesses**\n\n- **[W1] Detailed Evidence**: The paper lacks detailed evidence on the model's performance when handling highly ambiguous or incomplete language queries, which may limit its practicality in real-world applications. While Gaussian-Text Contrastive Learning is introduced to address ambiguities, it could still lead to confusion in referring segmentation for different objects. Additional ablations, videos, or novel view evaluations would strengthen the paper's claims and provide clearer validation of its effectiveness.\n\n- **[W2] Training time**: The paper does not thoroughly address the method's training and inference time. The method's complexity, particularly the position-aware cross-modal interaction module, may result in high computational costs, potentially limiting its feasibility for large-scale or real-time 3D environments.\n\n- **[W3] Difficult with sudden viewpoint changes**: The authors state that the model struggles with significant viewpoint changes and perspective shifts. However, how do other baselines perform under these conditions? Is this limitation specific to the proposed method, or is it a broader challenge in the field?", "other_comments_or_suggestions": "-", "questions_for_authors": "- Fig 1. The elephant is clearly visible in the scene, so why do the authors claim that it is not?\n\n- How would the model perform on datasets like MessyRooms, where some scenes contain up to 1,000 objects? Given that manual annotation and descriptions are impractical in such cases, how can this challenge be addressed?\n\n- The approach relies heavily on pseudo-ground truth masks. How does the proposed method ensure that errors in these masks do not negatively impact the model's performance?\n\n- Could the authors provide more qualitative results or supporting videos showcasing the model's performance from different novel views? This would further strengthen the paper's claims and demonstrate the method's effectiveness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G4vHxPgUjZ", "reviewer": "Reviewer_2Vdk", "summary": "The paper introduces ReferSplat, a framework for Referring 3D Gaussian Splatting Segmentation (R3DGS), aiming to segment 3D objects based on **natural language descriptions**, even when occluded or not directly visible. Key contributions include:\n\n1. **R3DGS Task**: A new task requiring 3D multi-modal understanding and spatial reasoning.\n2. **Ref-LERF Dataset**: A dataset with 295 language expressions emphasizing spatial relationships and object attributes.\n3. **ReferSplat Framework**: Combines 3D Gaussian Referring Fields, Position-aware Cross-Modal Interaction (PCMI), and Gaussian-Text Contrastive Learning (GTCL) to align language and 3D spatial features.\n4. **State-of-the-Art Results**: Outperforms existing methods on R3DGS and 3D open-vocabulary segmentation benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your positive feedback on our work: novel, well-motivated, effective PCMI and GTCL design, and well-written.\n\n>**Q1: Diversity comparison to existing datasets**\n\n**A1:** The comparision to ScanRefer and Multi3DRefer is shown in the table below. Due to the limited availability of large-scale 3DGS datasets, our Ref-LERF is built upon the widely used LERF dataset, extended with referring sentence annotations to support the R3DGS task. While the number of scenes is limited, Tab. 5 of the main paper shows that our method achieves strong performance, demonstrating its effectiveness. Furthermore, as discussed in our response to Q4, our method also generalizes well to larger and more diverse datasets, e.g., ScanNet, under the generalized training setting.\n|Dataset|Year|Pub.|#Object|#Expression|#scenes|data format|\n|-|-|-|-|-|-|-|\n|ScanRefer|2020|ECCV|11,046|51,583|800|3D scan|\n|Multi3DRefer|2023|ICCV|11,609 |61,926|800|3D scan|\n|Ref-LERF|2025|-|59|295|4|Multi-view Images|\n\n>**Q2: Evaluation metric**\n\n**A2:** To provide a more comprehensive evaluation, we report mAcc@0.25, which measures the percentage of predictions with IoU > 0.25 and is commonly used in 3D point cloud referring segmentation tasks. This metric reflects performance in practical scenarios, including occlusions. As shown in the table below, ReferSplat achieves a mean mAcc. of 38.4 on Ref-LERF, significantly outperforming LangSplat and demonstrating our method’s robustness. We will include this metric and the results in the revision.\n|Method|ram.|ram.|fig.|fig.|tea.|tea.|kit.|kit.|avg.|avg.\n|-|-|-|-|-|-|-|-|-|-|-|\n|Metric|mIoU|mAcc.|mIoU|mAcc.|mIoU|mAcc.|mIoU|mAcc.|mIoU|mAcc.|\n|LangSplat |12.0|18.4|17.9|25.4| 7.6|10.2| 17.9|27.5| 13.9|20.4|\n|GS-Grouping |27.9|30.3 |8.6|11.1| 14.8|16.9| 6.3|13.8| 14.4|18.0|\n|**ReferSplat**|**35.2**|**50.0**|**25.7**|**31.7**| **31.3**|**33.9**| **24.4**|**37.9**|**29.2**|**38.4**|\n\n>**Q3: Dataset splits**\n\n**A3:** The dataset comprises 772 training images and 22 testing images, with 236 language descriptions used for training and 59 for testing, totaling 295 descriptions. We will add the detailed dataset split information in the revision.\n\n>**Q4: Generalization to unseen scenes**\n\n**A4:** The experiments in our main paper follow a per-scene optimization setup, which naturally limits direct generalization to unseen scenes. To address this, we explore a generalized training paradigm, as shown in **Tab. 11 of the Appendix**. In this setting, the referring feature is no longer per-scene initialized; instead, it is predicted from other Gaussian attributes such as color, opacity, and position, following a feed-forward paradigm. This design enables generalization beyond per-scene optimization. To further validate the generalization capability of our method, we conduct additional experiments on the larger-scale and more diverse ScanNet dataset. We select 30 scenes from the official training split for joint training and 5 scenes from the validation split for evaluation. Language expressions are sourced from ScanRefer. As shown in the table below, our method achieves strong performance across diverse and unseen environments, highlighting its robustness and generalization. We expect that extending our framework with scalable architectures (e.g., MVSplat [a]) will lead to further performance gains and improved applicability in real-world scenarios.\n|Method|scene0011|scene0015|scene0019| scene0025|scene0030|mIoU|\n|-|-|-|-|-|-|-|\n|ReferSplat|15.9|19.7|27.8|18.3|21.4|20.6\n\n[a] MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images, ECCV 2024.\n\n>**Q5&Q6: Computational costs**\n\n**A5:** We conduct experiments on the ramen scene from the Ref-LERF dataset using the same NVIDIA A6000 GPU to compare the computational cost of our ReferSplat against SOTA methods. Results show that ReferSplat achieves significantly lower computational complexity and faster inference speed than LangSplat. While GS-Grouping excels in storage and FPS, ReferSplat outperforms all methods in segmentation performance. ReferSplat also has the shortest training time, thanks to a lightweight preprocessing pipeline that avoids costly operations like language feature compression (LangSplat) or mask association with video tracking methods (GS-Grouping). These results demonstrate that ReferSplat’s compact, efficient design is well-suited for real-world and large-scale 3D applications. We will include these comparisons in the revision.\nMethod|Training↓|FPS↑|Storage↓|mIoU↑|\n-|-|-|-|-|\nLangSplat|176min|12.4|46MB|13.9|\nGS-Grouping|66min|**54.2**|**2.3MB**|14.4|\n**ReferSplat**|**58min**|26.8|3.3MB|**29.2**|", "claims_and_evidence": "- ReferSplat’s superiority over baselines (LangSplat, SPIn-NeRF) is validated through quantitative results (Tables 5-7).\n- Ablation studies (Tables 1-4) confirm the effectiveness of PCMI and GTCL.\n- Pseudo mask generation via confidence-weighted IoU improves mask quality (Table 3).", "methods_and_evaluation_criteria": "- **Strengths**:\n    - The integration of language features into 3D Gaussians via referring fields is novel and well-motivated.\n    - PCMI and GTCL address spatial reasoning and semantic disambiguation effectively.\n    - This paper is well-written.\n- **Weaknesses**:\n    - Ref-LERF’s limited scale (only LERF-OVS dataset, including 4 scenes and 59 objects) and lack of diversity comparison to existing datasets (e.g., Scannet/Scannet++) may limit generalizability.\n    - Evaluation focuses on mIoU but omits metrics like precision/recall for occlusion cases.", "theoretical_claims": "No", "experimental_designs_or_analyses": "- **Strengths**:\n    - Comprehensive ablation studies validate each component.\n    - Failure case analysis (Appendix E) highlights practical challenges.\n- **Weaknesses**:\n    - Dataset splits (train/test) and generalization to unseen scenes are unclear.\n    - without training & rendering time comparsion\n    - without storage comparsion", "relation_to_broader_scientific_literature": "I'm not familiar with Segmentation + 3DGS, so I have no idea whether Refersplat is related to previous papers.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "No", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yjtlBUb82d", "reviewer": "Reviewer_Btgt", "summary": "This paper formulates the Referring 3D Gaussian Splatting Segmentation (R3DGS) task, which focuses on segmenting 3D entities that correspond to a given referring-expression in the form of a language-based query. The R3DGS differs by the currently employed task formulation of open-vocabulary 3D segmentation by its focus on spatial relationships between scene objects, as well as distinguishing properties of objects. The paper proposes a 3D Gaussian Splatting based method, ReferSplat, which learns a referring field enabling segmentation in a multi-modal setting. To evaluate the proposed method, the paper shows results both on the R3DGS task, and the open-vocabulary 3D segmentation task. The paper extends the existing LERF dataset with annotations for referring expressions, and constructs a dataset for the R3DGS task, namely the Ref-LERF dataset, which is then used for the R3DGS evaluations.\n\n### Update after rebuttal\nThe clarifications and discussions provided in the rebuttal have sufficiently addressed my concerns. I am still leaning towards acceptance, and will be keeping my original score (4: Accept).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your positive feedback on our work: meaningful, reasonable, and thorough.\n\n>**Q1: Claim 1**\n\n**A1:** In established 2D/3D referring expression segmentation (RES) tasks, referring segmentation involves segmenting target objects based on free-form natural language expressions that often include spatial relationships or object attributes. In contrast, open-vocabulary segmentation typically focuses on identifying all objects of a given category, usually specified by a single **category name**. As shown in Fig. 4 of our paper, the average sentence length in the LERF-OVS dataset is approximately **1.5 words**, with the vast majority of queries being category names (e.g., “cup”) and very few containing descriptive or relational information (e.g., “tea in a glass”). This highlights a clear difference in the nature of the language inputs between the two tasks.\nTo avoid confusion and better reflect the novelty of our task, we have revised the sentence to:\n“*a new task that focuses on segmenting target objects in a 3D Gaussian scene based on natural language descriptions that often contain spatial relationships or object attributes.*”\n\n>**Q2: Claim 2**\n\n**A2:** While the elephant is clearly visible in some training views, it is **not directly visible** from the **novel camera viewpoint** during inference in Fig. 1. We highlight this to emphasize a key difference from 2D RES, which relies solely on single-view information and struggles to handle such invisible scenarios. In contrast, the proposed ReferSplat leverages multi-view information to construct a holistic 3D scene representation, enabling robust reasoning even when target objects are occluded or not directly visible. For further clarification, please refer to L128–137 (second column). We will revise the corresponding description to make this point clearer.\n\n>**Q3: Claim 3**\n\n**A3:** We agree with your comment and have revised the sentence for better clarity. The updated version is:\n*“Existing methods rely on matching the text query with 2D rendered features instead of performing localization directly in 3D space, which limits their performance in complex scenarios.”*\n\n>**Q4: Related work**\n\n**A4:** Thank you for the suggestion. We will expand the related work to provide more discussion on datasets like ScanRefer and MultiRefer.\n\n>**Q5: Referring expression sentences generation**\n\n**A5:** The referring expressions are manually annotated by human annotators prior to training and are included in our Ref-LERF dataset. \n\n>**Q6: Generalize to new referring expressions**\n\n**A6:** For generalization to new expression, our method uses BERT embeddings for their strong language understanding and generalization, gained via pre-training on diverse text. By aligning 3D Gaussian features with BERT representations via Gaussian-Text modeling, the model can interpret and generalize to unseen object relationship descriptors. We will clarify this in the revision. For generalization to new scenes and new datasets, please refer to our responses to Reviewer 2Vdk Q4 and Reviewer t3nW Q5, respectively.\n\n>**Q7: Overall similarity**\n\n**A7:** Thank you for the insightful question. Our method aggregates per-word similarities, allowing the model to account for the influence of individual terms, including spatial relations like “far away.” Visualizations show that such words activate relevant regions, suggesting the model captures their importance in segmentation decisions. Additionally, we incorporate Gaussian-Text Contrastive Learning on global sentence-level features to enhance **holistic comprehension of the sentence context**, including complex spatial relationships. This combination of **local (word-level) and global (sentence-level) cues** enables the model to better interpret both positive and negative spatial terms. We will clarify this aspect in the revision.\n\n>**Q8: Mask usage**\n\n**A8:** Our method does employ pseudo masks generated by Grounded-SAM during training. The intended point is to emphasize that our method eliminates the need for manually annotated ground-truth masks, which are often costly and impractical. We will revise any ambiguous statements in the paper to more accurately reflect this and avoid confusion.\n\n>**Q9: CLIP result**\n\n**A9:** As suggested, we conduct experiments comparing BERT and CLIP embeddings for language features in R3DGS. Results show that BERT consistently outperforms CLIP. This is likely because CLIP focuses more on noun categories, while referring expressions often involve spatial and attribute-based descriptions. We will include more discussion and results in the revision.\n|Method|ram.|fig.| tea.| kit.|avg.|\n|-|-|-|-|-|-|\n|BERT|35.2| 25.7 |31.3 |24.4 |29.2|\n|CLIP|23.5|23.2|26.2 | 21.0|23.5|\n\n>**Q10: Typos**\n\n**A10:** Thank you! We will carefully proofread and correct any typos or ambiguous phrasing in the revision.\n\n>**Q11: Impact statement**\n\n**A11:** We will add impact statement in the revision.", "claims_and_evidence": "Most claims made in the submission are supported by convincing evidence and thorough analysis. However, there are a few claims that I found not fully-founded. It might be that I interpreted certain aspects incorrectly, but I believe these statements might be a bit problematic:\n\n1. _[L010-013] \"We introduce Referring 3D Gaussian Splatting Segmentation (R3DGS), a new task that focuses on segmenting target objects in a 3D Gaussian scene based on natural language descriptions.\":_ Open-vocabulary segmentation methods based on 3D Gaussian splatting representation already segment target objects based on natural language descriptions, as they take a free-form query text as input. I acknowledge that the referring expression-based segmentation is a different task as it focuses more on inter-object relations and object properties, but this statement is slightly misleading as the novel aspect of the proposed task is not segmenting objects based on \"natural language descriptions\", instead the referring segmentation aspect.\n2. _[L034-037, Column 2] - \"… identify newly described objects, even when occluded or not directly visible in the novel view…\":_ The illustration and the explanation are a bit confusing as these objects are indeed visible in _some_ views. The narrative of the method being able to identify and segment objects not directly visible is not fully founded.\n3. _[L074-078] - \"Since the text query is only introduced during inference, the final predictions solely rely on 2D rendered features in a single-view reasoning framework, limiting the model’s ability to effectively localize objects in 3D space.\":_ The limitation of these methods is indeed due to the fact that they rely on 2D rendered features instead of performing the localization directly in the 3D space. However, this is not due to \"the text query only being introduced during inference\" as the first part of the sentence implies.", "methods_and_evaluation_criteria": "Proposed method is reasonable and has been designed with a meaningful thought progression, successfully targeting the 3D referring expression based segmentation task. Introducing a referring field in the representation is quite meaningful, and it addresses the limitation of existing open-vocabulary 3D Gaussian splatting methods which mainly focus on learning an implicit semantic field, often falling short on identifying inter-object spatial relations.\n\nAs there are no available datasets for evaluating 3D referring expression segmentation task in the context of 3D Gaussian splatting, this paper proposes an extension of the LERF dataset. The proposed dataset, Ref-LERF, aims to provide a reasonable evaluation benchmark for the proposed task of 3D referring expression-based segmentation for 3D Gaussian splatting. In addition to designing this dataset and performing reasonable evaluations on this dataset, the proposed method is also compared against existing benchmarks for open-vocabulary 3D segmentation for completeness. Overall, I think the evaluation is overall meaningful and through, and I have the impression that a reasonable effort was made to obtain fair comparisons.", "theoretical_claims": "I did not identify any theoretical claims with proofs provided in the submission.", "experimental_designs_or_analyses": "As there are no available datasets for evaluating 3D referring expression segmentation task in the context of 3D Gaussian splatting, this paper proposes an extension of the LERF dataset. Experimental analysis is presented on the proposed Ref-LERF dataset. Experimental design is generally meaningful, and I found that the claims made while discussing the experimental results are reasonable.", "relation_to_broader_scientific_literature": "Referring expression segmentation is a crucial task for many robotics applications. While there is a large body of work towards this goal in the 3D scene understanding domain that primarily focus on 3D point cloud representations, I am not aware of any works addressing the same for higher-fidelity representations such as 3D Gaussian splatting-based methods. There is another line of work for addressing natural-language based segmentation of 3D Gaussian splatting representations, particularly in the context of open-vocabulary segmentation. However, the design of such methods as well as the evaluation methodology is generally more fixated on identifying objects based on their semantics. Often, those methods are evaluated using a set of text queries describing the target object- however, to the best of my knowledge none of these methods systematically evaluate for how well the method can identify objects based on their relations to other scene objects. By extending the existing LERF dataset, this submission takes a step towards a relatively less explored aspect of language-guided segmentation of 3D Gaussian splatting representations.", "essential_references_not_discussed": "I found that this submission generally discussed essential references in the context of open-vocabulary 3D Gaussian splatting segmentation. While I understand that this method has a main focus on 3DGS and not on 3D point cloud representations, given the really large body of work for 3D referring segmentation of point clouds, I found the discussion in L134-148 a bit underdeveloped. I think at least the datasets for this task (such as ScanRefer and MultiRefer) can be discussed further, to provide a better context on the relevancy of the proposed RefLERF dataset.", "other_strengths_and_weaknesses": "Strengths:\n- The paper addresses an important topic (3D referring segmentation) that was not explored in the context of 3DGS representations to the best of my knowledge.\n- The paper is generally written with clarity.\n- Qualitative examples are interesting, I appreciated that the examples in Figure 5 generally feature expressions where the target object's semantics are not explicitly stated, showing the strengths of the method for identifying objects mostly based on object-relations.\n\nWeaknesses:\n- If I understand correctly, the method requires the generation of a set of referring expression sentences first, which are then used during training. However, it is not clear to me how or when these sentences are generated. How are these sentences generated?\n- I am unclear about how well the method can generalize to new referring expressions if it is trained with a set of pre-written expressions. I understand that the contrastive module is introduced for this purpose, but I do not understand how it is possible to generalize if the model never sees certain types of object relationship descriptors.\n- L196-200 state that the overall similarity is measured by summing per-word similarities. But how are tokens representing negative relations accounted for? For instance if the query is \"the object far away from the window\", I still suspect that the model will identify an object right by the window. How are such cases handled in the model implicitly?\n- I understand that based on the training referring expression sentences, pseudo-GT 2D masks are generated using Grounded-SAM. This means that the method indeed relies on the use of 2D masks. However there are some statements claiming that the proposed method circumvents using masks. This is not fully founded, and a bit confusing. \n- For extracting word and sentence features, the method employs BERT embeddings (L314, right column) despite using CLIP distillation at another part of the method (L196, right column). I was unable to see any discussion or an ablation regarding the choice of BERT instead of CLIP to extract features from the text input.", "other_comments_or_suggestions": "1. _[Fig. 1 caption] \"The one sanding on the...\"_ should be _\"standing\"_\n2. _[L156] \"Each Gaussian g_i is parameterized its mean position...\"_ should be _\"by its mean position...\"_\n3. _[L250-251] \"The proposed Position-aware Cross-Modal Interaction modules establishes\"_ - should be _\"module establishes\"_\n4. _[L040-042, Column 2] - \"A straightforward baseline for R3DGS is to adapt existing open-vocabulary 3D scene understanding methods by replacing the open-vocabulary class names with natural language expressions.\":_ I agree with this statement in the sense that it accurately identifies how a straightforward baseline can be formed from open-vocabulary 3D scene understanding methods. However, the phrasing \"open-vocabulary class names\" is not very accurate.\n5. The impact statement required by ICML is missing in this submission.", "questions_for_authors": "Please see the questions listed in Strengths and Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tA5mlwCehJ", "reviewer": "Reviewer_naU6", "summary": "This paper introduces a new task—Referring 3D Gaussian Splatting Segmentation (R3DGS), which aims to segment target objects in 3D Gaussian Splatting scenes based on natural language descriptions. The authors construct the dataset specifically for this task, named Ref-LERF, and propose a framework called ReferSplat. The method primarily builds 3D Gaussian Referring Fields, introduces a Position-aware Cross-Modal Interaction module to fuse spatial and textual information, and employs Gaussian-Text Contrastive Learning to enhance the discriminative capability of cross-modal features, achieving state-of-the-art performance on both the R3DGS and 3D open-vocabulary segmentation tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your positive feedback on our work: clever, reasonable, effective PCMI and GTCL design, and comprehensive ablation studies.\n\n>**Q1&Q2: Generalization ability**\n\n**A1:** The experiments in our main paper follow a per-scene optimization setup, which naturally limits direct generalization to unseen scenes. To address this, we explore a generalized training paradigm, as shown in **Tab. 11 of the Appendix**. In this setting, the referring feature is no longer per-scene initialized; instead, it is predicted from other Gaussian attributes such as color, opacity, and position, following a feed-forward paradigm. This design enables generalization beyond per-scene optimization. To further validate the generalization capability of our method, we conduct additional experiments on the larger-scale and more diverse ScanNet dataset. We select 30 scenes from the official training split for joint training and 5 scenes from the validation split for evaluation. Language expressions are sourced from ScanRefer. As shown in the table below, our method achieves strong performance across diverse and unseen environments, highlighting its robustness and generalization. We expect that extending our framework with scalable architectures (e.g., MVSplat [a]) will lead to further performance gains and improved applicability in real-world scenarios.\n|Method|scene0011|scene0015|scene0019| scene0025|scene0030|mIoU|\n|-|-|-|-|-|-|-|\n|ReferSplat|15.9|19.7|27.8|18.3|21.4|20.6\n\n[a] MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images, ECCV 2024.\n>**Q3: GOI result**\n\n**A3:** Thank you for this insightful suggestion. To further validate the effectiveness of our method, we have conducted additional experiments comparing our approach with GOI [b] on the Ref-LERF dataset. The results demonstrate that although GOI outperforms LangSplat, its performance remains notably lower than that of ReferSplat. This highlights that, although both methods utilizing 2D pseudo-mask supervision, the key difference lies in how effectively the connection between 3D Gaussian points and natural language expressions is established within the 3D scene. We will revise the main paper to discuss GOI [b] and update Tab. 5 with the new comparison results shown below.\n|Method|ram.|fig.| tea.| kit.|avg.|\n|-|-|-|-|-|-|\n|Grounded SAM|14.1|16.0|16.9|16.2|15.8|\n|LangSplat|12.0|17.9|7.6|17.9|13.9|\n|SPIn-NeRF| 7.3|9.7|11.7|10.3|9.8|\n|GS-Grouping| 27.9|8.6|14.8|6.3|14.4|\n|GOI|27.1|16.5|22.9|15.7|20.5|\n|**ReferSplat**|**35.2**|**25.7**|**31.3**|**24.4**|**29.2**|\n\n[b] GOI: Find 3D Gaussians of Interest with an Optimizable Open-vocabulary Semantic-space Hyperplane, ACM MM 2024.\n\n>**Q4: Position information**\n\n**A4:** Thank you for your insightful question. Incorporating positional information into the referring feature (Eq. 7) enhances the model’s ability to understand **spatial relationships** described in the referring expressions while also enriches the referring feature with **geometric context** from the 3D scene. Instead of degrading performance, this geometric context helps ensure that segmentation masks accurately cover the complete target object, resulting in improved open-vocabulary segmentation performance.", "claims_and_evidence": "The authors claim that ReferSplat achieves SOTA performance through extensive comparisons with existing methods. The effectiveness of individual modules (e.g., PCMI and GTCL) is demonstrated.", "methods_and_evaluation_criteria": "Methodologically, the authors cleverly incorporate natural language guidance into the conventional 3D Gaussian Splatting framework, enabling the model to capture fine-grained segmentation details of target objects in 3D scenes. The evaluation metrics, such as mean IoU, are standard, and the constructed Ref-LERF dataset provides a reasonable test platform for the task.", "theoretical_claims": "The paper presents several derivations regarding rendering formulas, cross-modal attention mechanisms, and contrastive learning. The overall theoretical derivations are consistent with commonly used methods in the field.", "experimental_designs_or_analyses": "The authors perform extensive comparisons with existing methods (such as LangSplat and Grounded SAM), providing both quantitative and qualitative analyses.\nHowever,", "relation_to_broader_scientific_literature": "This work is closely related to the latest developments in 3D neural rendering.\nHowever, for the R3DGS task, GOI[1] stands out from other methods by incorporating a 2D referring expression segmentation model to assist in 3D localization, making it one of the most comparable works to your proposed approach. However, the current submission does not include a  comparison with GOI in Ref-LeRF dataset. I recommend conducting and reporting such a comparison—or at least providing a detailed discussion regarding how your method differs from GOI—to strengthen the experimental validation and highlight the relative advantages of your proposed framework.\n\n[1] Goi: Find 3d gaussians of interest with an optimizable open-vocabulary semantic-space hyperplane.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "### Strengths\n1. Provides comprehensive ablation studies and quantitative comparisons, with experimental results showing significant performance improvements.\n\n### Weaknesses\n1. The model is trained separately for each scene, which might affect its scalability and generalization ability in large-scale deployments;\n2. The dataset is relatively small, and more scenes will be needed in the future to validate the robustness of the method;\n3. Refer to \"Relation To Broader Scientific Literature*\"", "other_comments_or_suggestions": "No.", "questions_for_authors": "Why does incorporating positional information into the refer feature (CLIP feature) in Eq. (7) not degrade open-vocabulary segmentation performance, but instead improve it?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work initially received unanimously positive reviews in major aspects. It aims for an interesting and well-motivated 3D task while contributing a newly introduced dataset, outperforming previous methods, with more directly applied text embeddings for their referring segmentation in the 3DGS task, with extensive ablation studies. 

Some raised concerns include detailed evidence, time complexity, generalization towards novel views, and natural language expressions. The authors faithfully provided the requested experiments and reasonable explanations, which led to satisfactory responses from all reviewers. AC thinks this quality paper got quality reviews and exceptionally constructive rebuttals to help improve the manuscript.

AC also leave minor comments on visualization and table style. 
* In Figure 5, why is the Grounded-SAM result for "ramen" horizontally flipped? 
* Tables 1-7 have narrow space margins between captions and tables. 
* Missing running titles on the top of pages.

Overall, AC appreciates this well-written and hardworking paper and happily recommends this paper for acceptance to our program.
Decision: Accept (oral)