COS3D: Collaborative Open-Vocabulary 3D Segmentation
Authors: Runsong Zhu
Ka-Hei Hui
Zhengzhe Liu
Qianyi Wu
Weiliang Tang
Shi Qiu
Pheng-Ann Heng
Chi-Wing Fu
Venue: NeurIPS 2025 poster ()
Keywords: Open-vocabulary 3D Segmentation
segmentation
language
Primary area: applications
TL;DR: We present COS3D, a new collaborative prompt-segmentation framework that effectively contributes to integrating complementary language and segmentation cues throughout the entire pipeline.

Abstract: Open-vocabulary 3D segmentation is a fundamental yet challenging task, requiring a mutual understanding of both segmentation and language. However, existing Gaussian-splatting-based methods rely either on a single 3D language field, leading to inferior segmentation, or on pre-computed class-agnostic segmentations, suffering from error accumulation. To address these limitations, we present COS3D, a new collaborative prompt-segmentation framework that contributes to effectively integrating complementary language and segmentation cues throughout its entire pipeline. We first introduce the new concept of collaborative field, comprising an instance field and a language field, as the cornerstone for collaboration. During training, to effectively construct the collaborative field, our key idea is to capture the intrinsic relationship between the instance field and language field, through a novel instance-to-language feature mapping and designing an efficient two-stage training strategy. During inference, to bridge distinct characteristics of the two fields, we further design an adaptive language-to-instance prompt refinement, promoting high-quality prompt-segmentation inference. Extensive experiments not only demonstrate COS3D's leading performance over existing methods on two widely-used benchmarks but also show its high potential to various applications,~\ie, novel image-based 3D segmentation, hierarchical segmentation, and robotics.

COS3D: Collaborative Open-Vocabulary
3D Segmentation
Runsong Zhu'
Ka-Hei Hui
Zhengzhe Liu
Qianyi Wu
Weiliang Tang'
Shi Qiu
Pheng-Am Heng'
Chi-Wing Fu!
1 The Chinese University of Hong Kong
2 Autodesk Al Lab Monash University
3 Lingnan University
Abstract
Open-vocabulary 3D segmentation is fundamental yet challenging task, requiring a mutual understanding of both segmentation and language. However, existing kaussian-splatting -aased based methods rely either on single 3D language field, leading to inferior segmentation, or on cre-compued class-agnostic segmentations, suffer- ing from errоr accumulation. To address these limitations, present COS3D, new collaborative prompt-segmentatiin framework that contributes to effectively integrating complementary language and segmentation cues throughout its entire pipeline. We first introduce the new concept of collaborative field, comprising an instance field and language field, as the cornerstone for collaboration. During training, to effectively construct the collaborative field, our key idea is to capture the intrinsic relationship between the instance field and language field, through a novel instance-to to to-langu feature mapping and designing an efficient two-stage training strategy. During inference, bridge distinct characteristics of the two fields, we further design an adaptive language-to-ingg -o-istaace prompt refinement, pro- moting high-quality prompt-segmentatiin
inference. Extensive experiments not only demonstrate COS3D's leading performance over existing methods on two widely-used benchmarks but also show its high potential to various applications, i.e. novel image-based 3D segmentation, hierarchical segmentation, and robotics.
Introduction
Open-vocabulan 3D segmentation (OV3DS) aims to predict 3D segmentation of scenes according to given natural language queries. Beyond traditional 3D segmentation, which is often restricted to fixed object categories 8 the OV3DS task supports flexible text queries, allowing for diverse semantic categories, physical properties, affordance, and more. This flexibility is crucial making OV3DS practical and valuable tool for applications in fields such as AR, VR, and robotics.
Recent efforts focus transferring 2D vision-languse models (VLMs) to 3D scenes represented by learned radiance fields. These works can be roughly divided into two classes: language-based and regmentation-buseld Specifically, language-baseds methods [ propose to distill language features (e.g., CLIP (44) from the 2D) image space to a 3D language field by leveraging differentiable rendering support OV3DS; see Fig. [ (a). However, directly learning language features through pixel-wise language distillation demonstrates limited distinctiveness, leading to severe artifacts and errors around boundaries in the segmentation results.
On the other hand, segmentation- based methods, e.g., 15 directly decompose the OV3DS task into two sub-tasks: (i) class- agnostic 3D segmentation, followed by (ii) post-selection of the best-matched 3D segments, using a 2D vision-language model [14 [8] see Fig. (b). Though this approach bypasses direct language distillation, faces two major challenges. leading to limited performance. First, without semantic cues, accurately segmenting all objects in a 3D scene is highly
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Text
3D segmentation
Class-agnostic
Text
Language field
Pore Commmnn (nmtnn on Lebt
CO530
vertion)
COS3D (ours)
(CVRR
Language
Mapping
field
Post-selection
Instance field
Langange-azeed
(CVRIZ4)
Segrosttation
Open-vocabulary
Open-vocabulary 3D segmentation
Open vocabblayy
3D segmentation
3D segmentation
Training time (min) (d) Effectiveness and efficiency comparisons
(a) Language- based
(b) Segmentation- based
(c) Collaborative prompt-basen (ours)
Comparing different paradigms. (a) Language-based methods [H3 directly learn 3D language field for open- vecaboocayllary segmentation. (b) Segmentation based methods [15] perform class-agnostic segmentation then post selection. (c) COS3D addresses existing limitations with new collaborative prompt-segmentt instance fields in the training and inference. The solid line (dotted line) indicates inference (training). (d) COS3D clearly outperforms existing methods on both segmentation quality and training efficiency. Note that "fast version" refers result of our approach under short optimization time setting (see while the time and performance of the baselines are sourced from their publications [2 16].
Figure I:
framework that connects knowledge from the language and
Sec 3.34.3
challenging, under- and segmenttttnnoi errors often occur in the class-agnostic segmentation, which further affects the post selection. Second, the hand-crafted matching strategies in the post selection easily introduce additional inaccuracies that further degrade the performance.
Revisiting the existing methods, we attribute their limitations to the lack of integrating language and segmentation information. In particular, these two types of information provide complementary knowledge: segmentation information typically discriminative and boundary-aware whereas language information facilitates high-level understanding of objects and scenes. Fundamentally, to achieve OV3DS requires a mutual understanding of both language and segmentation.
To this end, we introduce COS3D, a new COllaborative approach for wrompt-Segmentttiin of 3D) scenes, in which collectively incorporate segmentation and language cues in our framework; F (c). There are three technical components in COS3D. First, we propose the new concept of collaborative field, comprising an instance field and a language field, as the foundation COS3D. To effectively construct the two fields, our key insight lies in their intrinsic relationship: regions within the same object segment should share similar semantics and exhibit similar language information. Second, we propose modeling the intrinsic relationship through a feature mapping process from a learned, boundary-awand the instance field to implicitly encode the segmentation information, then formulate an instance- to-language mapping learning to facilitate the language-fielle construction. Third, inference, given a text query. we generate the segmentation from the text-aligned language field. Importantly, considering the limited expressivity of the language feature, we leverage the distinct characteristics prompt refinement to exploit the
instance field to the text-aligned language field. Here, we first train
of the instance field and introduce an adaptive language-to-inttane intermediate 3D relevance map from the instance field as a prompt, then design a further refinement on the boundary-aware instance field for prompt segmentation. With these new designs, COS3D is able to arrive at a surprisingly effective and efficient solution; see Fig. (d).
We evaluate COS3D on two standard benchmarks for OV3DS. Both quantitative and qualitative results show that our method significantly outperforms existing approaches. Also, the ablation studies validate the effectiveness of our designs for both training and inference stages. Furthermore, we present three example applications of COS3D, including novel image-based 3D segmentation, explicit hierarchical OV3DS, and robotics, demonstrating its potential and practical values.
Our major contributions are summarized as follows:
We present COS3D, segmentation and language cues, enabling top-quality open-vocabulary 3D segmentation.
collaborative promet-seemeentttion framework that integrates
For training, we propose novel instance-to-langune language mapping with two optional implemen- tations that effectively and efficiently leverage the instance field to enhance the construction of a semantically meaningful language field.
For inference, we propose an adaptive annuuge-to-instane prompt refinement that utilizes the 3D relevance map from the language field to guide the refinement in the instance field.

---

Our method sets new state-of-the-an performance on two standard benchmarks and shows strong potential for image- based segmentation, hierarchical segmentation, and robotics.
2
Related work
Radiance field. Radiance fields have emerged as a powerful representation for supporting 3D scene reconstruction with diverse properties such as geometry, color, and semantics, from only 2D inputs such as multi-view RGB images and extracted feature maps. Neural Radiance Fields (NeRF) [9] model the radiance field using neural networks composed of MLPs, enabling photorealistic novel view synthesis. Subsequent works focus on improving the efficiency of NeRF by introducing explicit 3D structures, such as voxel grids 20 21 and hash grids (2) More recently, 3D Gaussian Splatting (3D-GS) [23)32] is proposed as an alternative representation by modeling the radiance field as a set of explicit 3D Gaussian points. This approach supports splatting based rendering [33), which highly efficient, significantly enhancing its potential for real-time applications. Given these advantages, we adopt 3D-GS as the backbone representation in our 3D segmentation framework.
Open-vocabulary 3D segmentation. Open- -ecebvocyyl 3D scene segmentation has made signifi- cant progress in recent years, empowered by 2D foundation vision-language models (VLMs) (e.g., CLIP [1], LSeg [34], DINO 35 ), together with 3D representations, ranging from point clouds to radiance fields. Farly approaches 43 project 3D point clouds to 2D views to align them with image-based features, enabling zero-shot open-vocabulary 3D segmentation. However, as discussed in [44], point cloud representation suffers from the discrete structure and typically has a lower resolution compared to images, limiting their effectiveness and applications.
To overcome limitations of the point cloud representation, recent methods propose distilling dense VLM features into continuous radiance field representations, enabling high- resolution novel-view synthesis for effective feature alignment and downstream tasks. Specifically, LeRF 9 first introduced the concept of language field distillation into NeRF via a 2D CLIP
46
supervision. Besides, Spnat I0, L.GGaaussias [1I], SplSp [12), and FastI.GS [3] adopt 3D-GS as an explicit scene representation, which integrates language features, for supporting open- vocabulary 3D scene understanding. While these models achieve promising results, the segmentation quality is still limited by the weak expressiveness of the directly learned language features. In contrast, other methods [1517] tackle the task sequentially by first performing class-agnostic 3D segmentation, followed by selecting the best-matched 3D segment using language queries. For instance, OpenGaussian [16] and InstanceGaussian [17] propose to align 3D segmentations with 2D segmentations produced by SAM to leverage 2D CLIP features to enable subsequent text grounding. Besides, Gaussian Grouping [15] employs a 2D vision- langua-l grounding model (i.e., GroundingDINO (18) associate 2D grounding result with 3D segments using handcrafted 2D) matching techniques. However, these methods suffer from error accumulation, which restricts overall performance. To address these limitations, we introduce a novel 3D) prompt-based segmentation framework that collaboratively engage both segmentation cues and language cues, throughout both the training and inference stages, to optimize for high-quality open-vocabulag segmentation.
Promptable segmentation. Promptable segmentation, which aims to generate segmentations based on input prompts by specifying the target be segmented within an image, was introduced by the [47]. Its effectiveness has been widely demonstrated through number of follow-up works 48.52 To adopt it to 3D understanding, some recent methods (53-56) propose to learn the discriminative instance field that encode 3D segmentation information. At the inference, these methods leverage user-click prompts to specify the target, facilitating the production of more accurate segmentation results. Though these methods achieve notable improvement, they require additional manual interaction in screen space as prompt, thereby limiting their applicability in autonomous 3D systems.
Segment Anything Model (SAM)
Going beyond the prior works, realize novel prompt-segmentation framework directly support open-vocabulary 3D queries as prompts for segmentation inference through innovations that actively integrate the segmentation-awe instance field and text-aligned language field.
3 Method
Given set of multi-view posed images, we utilize the 2D foundation models SAM (5 and CL.IP 14 to produce associated 2D instance segmentation masks (K,) } and 2D language feature maps {Fz).
3

---

Sec. Collaborative
Sec. Collaborative filed learning
Instance feature map
Source domain: rendered instance feature
Language hled
Instance field
Renderiing
Grouping
3D
Cached
Ins2Lang mapping
Instance filed
Contrastive
mapping
RGB
segment
Target domain: language feature
Language
SAM
CLIP
Cached
Stage hell learning
Ins2Lang mapping learning
Collaborative wrompt-segmentatiie
Text prompl
relevance map
Adaptive Lang2lns
eg, 'sheep"
prompt refinement
Render
Query feature
segmentation
Figure 2: Overview of our proposed COS3D method. We first introduce collaborative field, compris- ing an instance field and a language field (see See During training, we first learn the instance field to encode instance information and propose novel instance-to-language (Ins2Lang) mapping to construct the language field (see Sec During inference, leveraging the 3D relevance map from the language field, we design adaptive language-tor -onnstagg----nstacee (Lang2Ins) prompt refinement to further guide the instance field, enabling accurate segmentation (see Sec 3.3)
Based on reconstructed 3D Gaussian representations, our objective is to enrich these 3D Gaussians with high-quality, text-guided segmentations that align with the open-vocabulary query. Fig. illustrates the overview of our approach. First, to support effective open-vocabulary segmentation, we propose the new concept of collaborative field, which comprise an instance field and language field, integrating both segmentation and language cues (see Sec. 3.1. Next, to enable collaboration between the instance and language fields throughout the whole pipeline, we further design novel instance-to-languag mapping within a two-stage learning strategy in Sec. 3.2 and an adaptive language-to-instane prompt refinement for collaborative prompt-based inference in Sec, 3.3
3.1 Collaborative field
Specifically, our collaborative field utilize the explicit 3D Gaussian Splatting (3D-GS) 23), as the underlying 3D scene representation. Specifically, the 3D-GS model represents a 3D scene using a collection of explicit 3D Gaussians and leverages differentiable rasterization for efficient rendering. Mathematically, 3D-GS represents 3D scene as set of N Gaussians, (P., qı c.) denotes the center, scale, orientation (quaternion).
3D-GS backbone.
G (g where each opacity, and color coefficients (in spherical harmonics) of the i-th Gaussian. Each 3D Gaussian is projected to the image plane as 2D Gaussian G via tile-based rasterization 23 The color at query pixel u computed using a-blending:
C. = c,a
c,a. Caa.IIaa a, = o.(((,,
(I)
VEN
where N is the set of sorted 2D Gaussians contributing to pixel u. The parameters {gi} are optimized via photometric reconstruction loss.
Collaborative field. To enable effective open-vocabulary 3D segmentation of the scene represented by 3D-GS, propose the collaborative field to fully exploit the complementary strengths of seg- mentation and language information. Specifically. our collaborative field comprise two components: an instance field θ₁ and language field Og. Mathematically, the instance field is obtained by augmenting each Gaussian g with feature vector E Rdd, where dj is the feature dimension (set to 16 by default). Besides, the language field encoded as a high-dimensional language feature vector L E Rdd for each Gaussian g. where d, is the dimension of language feature, i.e., 512 for CLIP 14]. We refer to these representations as collaborative field because they interact during both training and inference. During training, the instance field serves distinctive representation that

---

simplifies learning the language field, allowing us to construct it more effectively and efficiently. At inference time, the language field provides an initial 3D relevance map in response text query, which guides the instance field to refine and produce accurate open-vocabulary segmentation results.
3.2 Collaborative field learning
In our collaborative field learning, the key idea is to introduce the instance- coatece-t-llaaee lo-language mapping to model the intrinsic relationship of two fields. To this end, we propose two-stage training strategy. First, we learn a segmentation- aware instance field supervised by the 2D SAM [47] segmentation. Next, based the learned instance field, we construct the language field by learning mapping function from paired instance and CLIP features extracted from multi-view images. Once trained, this mapping function is applied to the instance field to generate language field, completing the construction of the collaborative field.
Stage 1: instance field learning. For the instance field 0j we adopt contrastive learning to optimize the rendered features. Similar to color rendering, we apply differentiable rasterization to efficiently render the instance feature I, at each pixel as: I, ΣiEN La;; [ - at), 0; =
(u). Then, we apply widely-used InfoNCE loss in existing works 54 55 58 59
for supervision:
Lins
1
Σ Σ log S. EQ
exp (sim (I.., I,) En,ea exp (sim (Iu, I.)
S2
where similarity kernel function sim uses the dot product operation here and Q is the of pixel samples. In specific, S2y, 9 denotes the pixel samples with the same instance ID j. according to the 2D instance segmentation K,, I and I, represent the mean instance features (centroids) for Ω2 and Q2 respectively. Notably, the instance segmentation mask IC, is automatically generated using SAM by creating grid of point prompts across the image, following common practices 16 12 II]. By minimizing this loss across training views, the instance field learns to produce discriminative and view-consistent features that capture 3D instance-level information.
Stage 2: instance-to-language (Ins2Lang) mapping learning, Based on the learned discriminative instance field, we introduce Ins2Lang mapping transfer the instance feature source domain to the language feature target domain. Mathematically, the mapping function & is defined as
= LI I O1,
(3)
where I denotes the instance feature and L denotes the corresponding language feature. To learn the mapping function Φ, we first construct training pairs between the instance features and their corresponding language features. Specifically, for each individual view, we render the multi-view consistent instance feature map, enabling
directly the 2D language feature map Fr. from CLIP as the corresponding pair. Moreover, since the CLIP features are inherently patch-wise, and reduce redundancy, we utilize the SAM mask group rendered instance feature maps and CLIP feature maps by averaging the features with the same 2D instance ID. This process results in segment- wise training feature pairs, denoted as (I" L 2m=1 where M is the total number of pairs. Based on the training pairs, we further provide two implementation strategies, ie., shallow MLPs and kernel regression for the mapping function.
)
i) Shallow MLPs. We can choose shallow MLPs, denoted as & Peewwrrk to represent the instance-
m= to supervise the learning of & network Specifically, use the following mapping loss: Lmmapping L Notably, the mapping learning is highly efficient, requiring less than three minutes on a single GPU.
to-language mapping. Then, we utilize the prepared mapping pair (I Lmm 31
$ network (Im) )
ii) Kernel regression. We can also utilize traditional kernel regression, denoted as 4 to represent the mapping function. In particular, we adopt the widely used Nadaraya-Wan -atson estimator [60] because of its simplicity and learning-free nature. Mathematically. the kernel regression function
defined as: 4 (I) = 1 (- (-11---11
/ (_
exp 1-r"' where a is a 2
20t
hyperparameter that controls the bandwidth of the kernel function, and is set to 0.1 by default.
Based on the learned Lnng mapping t we can obtain our language feature field e by calculating the corresponding language feature (LL) for each Gaussian.
5

---

Discussion. We introduce the Ins2Lang mapping within a two-stage training strategy for language field construction, offering the advantages of effectiveness and efficiency. (i) Effectiveness: In contrast to existing approaches that directly learn language features from scratch, our method constructs the language field via a mapping function from the learned instance field. This strategy leverages the discriminative, segmentation-awar features already captured by the instance field, enabling more semantically meaningful and spatially coherent language representations. As a result, it significantly improves the quality of open-vocabulary segmentation. (ii) Efficiency: Unlike approaches that directly optimize language features for each Gaussian point, requiring extensive supervision and per-point updates, our method employs shared mapping function that generalizes across all Gaussians. This significantly reduces the number of parameters and the training overhead. Moreover, instead of using dense pixel-level supervision, we construct training pairs at the segment (patch) level using SAM masks, which further lowers redundancy and improves learning efficiency. Together, these design choices result in highly efficient training process. In addition, the experimental comparisons with other alternatives are provided in our ablation (see Sec. 4.3).
3.3 Collaborative prompt-segmeentation inference
During inference, we further design an adaptive language-to-ing to-instance prompt refinement, enabling a collaborative wompp--eementaaie inference. As illustrated at the bottom of Fig. 2 given query text, we utilize the text-aligned language field to generate 3D relevance map as a prompt and introduce the adaptive prompt refinement in the boundary-aware instance field for producing accurate 3D Gaussian segmentation.
3D relevance map in language field. Based on the text-aligned language field, for the text query Bex.. we first generate the 3D relevance map, which indicates the correspondence between the input text and 3D regions. Specifically, we first utilize the CLIP [4] text encoder to obtain the corresponding language feature L Then, we compute the 3D point-level relevance R as: exp(L-Lue)
where Leanon is the CLIP embedding of a predefined canonical phrase 99 Intuitively, we can obtain segmentation by identifying Gaussian points S with high rele- vance, using a predefined threshold T (set to 0.5 by default), following common practices
R = min;
16]
to-nssag-e (Lang2Ins) prompt refinement. Directly extracting the segmen- tation by the above process easily produces inferior segmentation (see Fig. (b)). To address this, we treat the relevance map as a prompt to guide the instance field in refining segmentation via an adaptive Lang2Ins prompt refinement process. Particularly, starting from the initial high-relevance Gaussian point set S, we aim to obtain a refined Gaussian point set St that represents 3D Gaussian segmentation results.
Adaptive
Concretely, for each center Gaussian point g' E S, we leverage the learned instance field to define local neighborhood set Sg', consisting of points whose instance features have cosine similarity above a threshold T with that of g'. Here, threshold T is based on the statistical value from instance field. Considering the presence of errors in S, and the risk that expansion from noisy points may include undesired objects, we further perform an adaptive filtering operation for Syy based on a region-level relevance. We define this region-level relevance as the opacity-weighted average of relevance scores in Sgr (Ewes, Ow Rew BBw)/(Ess, ow), where ou and R., are the opacity and relevance of point w E ESgr. We process the regions descending order based on the relevance score of their center points. A region is included in the final segmentation only if its region score exceeds the threshold т (as defined earlier). By applying this process to the initial set S, we gradually aggregate retained expanded point sets, producing the final segmentation S,. More details on the algorithm and automatic threshold generation are provided in Supp.
[a] Reference Query: "pumplán"
[b) w/o Lang2ins prompt refnement
(c) w/ Lang2ins prompt refinement
Figure 3: Visual comparisons on LeRF 9
Unlike the existing methods 10-12 that solely rely on the relevance map, bounded by the limited expressivity of language features, our adaptive Lang2Ins prompt refinement further leverages the discriminative instance field to aggregate neighboring points with spatial and semantic coherence, thus enabling boundary-aware segmentation. Further, helps adaptively filter noisy points using robust region-wise relevance. As shown in Fig. (c), this algorithm significantly enhances the Gaussian segmentation quality, enabling clean object rendering. Moreover, the proposed adaptive Lang2Ins prompt refinement is efficient, adding small overhead to the query time (sec Sec.
Discussion.

---

3.4 Implementation details
We adopt the official implementation 3D-GS 23 with a default 30K training iterations as our base architecture. For the instance field and instance-fo-languag mapping (e.g.. MLPs version), also set the training iterations to 30K by default, following common practice as in [1O]. For kernel regression version in mapping. the function is directly formulated without requiring training. All experiments are conducted on single RTX-4090 GPÚ. More details are provided in Supp.
4 Experiments
4.1 Results on LeRF dataset
Settings. (i) Task: For open-voce vocabulary text queries, we first select the corresponding Gaussians
and render them into multi-view 2D images. (ii) Dataset and metrics: Following OpenGaussian [[6]
and LpngSplat [TO,, we evaluate our method on the LeRF dataset [9]. After rendering the selected
objects into 2D views, we compute mean Intersection over Union (mloU) and mean Accuracy (mAcc). (iii) Baselines: As this task requires explicit 3D point-level segmentation, we compare our method with other explicit Gaussian-bases LEGaussians [T], and segmentation- based method OpenGaussian [16]. Moreover, we provide the quantitative comparison with the most recent works, ie., InstanceGaussian [[7] and Dr. Splat [12]
methods, including language- methods LangSplat [6]],
Results. Quantitative results are presented in Tab. demonstrating that our proposed method achieves significantly improved results compared to existing language-based and segmentation- based baselines. The qualitative results in Fig. 4 further show that the rendered objects using method contain more complete boundaries and significantly fewer artifacts. Note that Instance- Gaussian 117 and Dr. Splat [12 are not open-
open-sourced, sourced, which prevents further visual comparisons.
Table 1: Performance of Gaussian segmentation in 3D space from text query on LeRF 99. Follow- ing [6,, we report mloU and mAce. The performance all prior works is sourced from [6 17
fqurines
Methed
miloU"
InloU
LangSplat
CVPR CVPR
9,66
10,16
6.93
11.38
20.34
9.18
9.09
1EConssinn
Langaage
16.21
23.82
17.99
23.21
19.27
27.12
15.79
26.76
11.78
18:18
Dr Splat [2]
CVPR 25
43.58
63.87
$3.36
90.36
$7.00
76.27
24.70
35.21
39.07
63.64
OpenGiusssan 16
Neut CVPR'25 25
Seymertation
38.36
$1.45
39.29
55.36
60.44
76.27
31.01
42.25
22.70
31.82
Y
Segmentation
45.30
58.44
Ours ohallow MLPs) Ours (Semel regresioon
Collaborative prompt
49.75
70.60
$3.90
76.79
66.91
$8.14
36.1
49.30
41.55
68:15
Collaborative prompt
$0.76
72.08
68.03
82.14
66.07
91.83
35,86
46.48
42.10
68.18
Query: "green chair"
Query: "green apple"
Query: "yellow pouf"
Query: "plate"
Query: "knife"
Query: "egg"
Reference
LangSplat
OpenGausslan
Ous
Reference
LangSplat
OpenGaussian
Ous
Open-vocabulary 3D Gaussian segmentation on the LeRF dataset. Our method outperforms previous open-sourced SOTA methods (i.e., LangSplat, OpenGaussian) in accurately identifying the 3D objects corresponding to text queries with fewer artifacts. Here, we present our results using kernel regression for visual comparison, and more results are provided in Supp.
Figure

---

Table 2: Performance of Gaussian segmentation on the ScanNetv2 [62] dataset compared to base-
lines 10
16
based on text query. The performance of all prior work been sourced from [6
19 classes
15 classes
10 classes
Methods
Type
mloU 1
mAce. 1
mloU
mAcc.
mloU
mAcc.
LangSplat
10
Language
3.78
9.11
5.35
13.20
8.40
22.06
LEGaussians
Language
3.84
10.87
9.01
22.22
12.82
28.62
OpenGaussian 16
Segmentation
24.73
41.54
30.13
48.25
38.29
55.19
Ours (shallow MI.Ps)
Collahorative prompt
26.72
39.89
31.02
46.30
37.28
55.41
Ours (kernel regression)
Collaborative prompt
32.47
49.05
35,95
54.35
44.32
Query
Query "tollet"
Query 'bed'
Query "door"
Reference mesh
OpenGaussian
Ours
GT
Reference mesh
OpenGausiin
Ours
GT
Figure Open- vocabulary Gaussian segmentation on ScanNetv2 dataset. Our method outperforms the previous open-sourced SOTA approach (i.e., OpenGaussian) in accurately identifying 3D objects for various text queries. addition, we use green boxes indicate regions accurate predictions and red boxes to indicate regions of missing predictions. Here, we present our results using kernel regression for visual comparison, and more results are provided in Supp.
4.2 Results on ScanNetv2 dataset
In this task, we focus on direct 3D evaluation without rendering processing. Specifically, the model takes text queries input and selects the corresponding Gaussian points. (ii) Dataset and metrics: Following the protocol established by OpenGaussian [16]. we adopt 19, 15, and 10 categories from ScanNetv2 [62] as text queries. Moreover, we evaluate performance using mloU and mAce for the 10 scenes selected by OpenGaussian [16]. (iii) Baselines: We compare our method against recent Gaussian-based approaches, including LangSplat [IO] LEGaussians [1], and OpenGaussian [16]. following prior work [16). Note that the latest baselines (i.e., InstanceGaussian and Dr. Splat) use different evaluation protocols or segmentation inputs, which hinder direct evaluation using the results from their papers. We provide additional comparisons in Supp.
Settings. (i) Task:
Quantitative results, as shown in Tab. 2 consistently demonstrate that our method signifi- cantly outperforms Gaussian-basse based methods. The 3D visualization results are presented in Fig. illustrating that our method achieves accurate and complete 3D Gaussian point-level segmentation for various queries, especially challenging scenarios (e.g., the dense "chair" query shown in Fig. 5)
Results.
4.3 Ablation study
The influence of learning designs for collaborative field. For collaborative field learning, propose an restanc-to-languaae mapping design within a two-stage learning strategy. Alternatively, there are two other technically feasible training solutions for our proposed collaborative field. One straightforward approach is the one-stage joint learning of instance and language branches, where the instance field is also influenced by the mapping loss, leading costly optimization and additional risk that the supervision for the mapping may make the instance feature space less discriminative. Another alternative is parallel learning, where the two branches are trained independently. failing to fuse the instance field and the language field. As shown in Tab. compared to alternatives, both of our two implementations for the proposed two-stage instance-to-language mapping design not only improve final performance but also significantly reduce the required training time. Moreover, we find that our kernel regression implementation achieves the best performance. We attribute this to choice of discriminative instance features the source domain, which makes the mapping process

---

inherently an easy regression task, and the traditional kernel regression method is well-suited for such a case. Thus, we use the kernel regression implementation for subsequent analysis. Moreover, we provide a more detailed analysis of kernel regression and the MLP counterpart in Supp-
The influence of collaborative prompt-segmentation inference. To analyze the influence of collaborative coompt-base segmentation inference, we compare our results with two other alterna- tives. Specifically, the first inference alternative conducts the class- agnostic 3D segmentation by clustering 7 and uses a similar strategy as OpenGaussian [16] to select the 3D segment results, and the second inference alternative solely utilizes the language branch in our collaborative field to generate the segmentation results [10] TT. The comparisons are presented in Tab. f indicating that collaborative prompt-based inference significantly improves segmentation by integrating language and instance knowledge, with only a slight increase in inference time per query.
Table 3: Ablation of learning on LeRF 9
Table 4: Ablation of inferences LeRF 9
Learning solution / (Mean)
mloU'
mAce
Training time
Inference solution/ / (Mean)
m
mAcc 1
Query time
Parallel learning
Joint leaming
49.15
43.84
69.19
59.81
165 min
95 min
Instance branch
44.07
59.83
0.12
Language branch
48.99
71.31
0.13
Our (shallow MI.Ps)
49.75
70.60
53 min
Collaborative prompt (ours)
50.76
72.08
0.22
Our (kernel regression)
50.76
72.08
30
Training efficiency. To analyze the training efficiency of our method, we compare performance under different training times. As training the instance field with the default 30K optimization steps requires the majority of the total training time (45 out of 50 minutes), we conducted experiments with shorter training times by reducing the number of optimization steps (i.e., 3K and 6K). The results, presented in Tab. [5] demonstrate that our method converges quickly and achieves significantly better performance than baselines even with less training time, highlighting our superior training efficiency.
The influence of different 2D foundation vision-language models (VLMs). We utilize CLIP and SAM [47] as the default 2D language and segmentation models in our main experiments, [7] ensure that the improvements are attributed to the proposed collaborative fields design. Furthermore, we conducted an ablation study comparing different 2D foundation VL.Ms (e.g.. CL.IP [1 vs. SigI.IP 63). SAM [47 vs.
following common practice in recent baselines [T0 IT 16 12
SAM2 [64], Semantic SAM [48). The results, shown in the Tab. 6 demonstrate that our framework compatible with different 2D foundation models. Additionally, we empirically observed that using
more advanced models (e.g.. SAM2 (64] for the segmentation model and SigLIP 63 for the language model) can lead to performance improvements.
Table 5: Training efficiency analysis on
LeRF 9
We report mean mloU.
Table 6: Comparisons various 2D foundation VLMs LeRF (.. Model A is the default implementation for our proposed COS3D.
Method
Training time
mIoU 1
I Langsplat
10
240 min
9.66
LEGaussian
240 min
16.21
Model
Segmentation
Language
mloU1 1
mAce
Dr.Splat 12
10 min
43.58
OpenCiaussia
16
60 min
38.36
A
SAM
4
CLIP [4
50.76
72.08
Instance Gaussian
45.30
B
SAM
SigL.IP 63
51.08
73.79
Ours (3k instance)
min
50.16
C
SAM2
CLIP
51.94
75.05
Ours (6k for instance)
15 min
50.24
D
Semantic SAM
48
CLIP
14
49.93
70.94
Ours (default)
50 min
50.76
4.4 Applications
Novel image-based 3D segmentation. Beyond text queries, our method inherently supports 3D segmentation using a novel image as queries. Specifically, given a novel query image, we utilize the CLIP vision backbone to extract visual features and apply our inference algorithm to obtain the 3D segmentation. As shown in Fig. 6 (a), our method enables accurate image-based segmentation when the query image contains similar, but not identical, object to those in the original 3D scene.

---

(a) Novel image-based based 3D segmentation
Image query
Reference
Query results
Image query
Reference
Query results
(b) Explicit hierarchical open-vocabulary extension
Reference
"Sheep" "
"Sheep head"
Reference
*Cup"
*Cup handle"
(c) Real-world robotic application
Box
HeadBbnnd
Our Gaussian segmentation results
Start
End
Captured posed images
Task: "Grasp the headphone and place it into the box."
Figure 6: (1) We illustrate the novel image-based 3D segmentation results. (b) We perform explicit hierarchical open-voca vocabulary 3D segmentation on LERF 19. (c) Following prior works [65 we leverage method to provide accurate 3D segmentation for real-world robotic grasping, enabling the successful execution of grasp manipulation. We provide the video demo in Supp.
Explicit hierarchical open-vocabulary extension. Furthermore, our method can be naturally extended support explicit hierarchical queries. Inspired by Elick-Gaussiar (54), we apply our mapping function to construct hierarchical language field based on two-level feature representation capture hierarchical information. As shown Fig. (b). our approach produces accurate 3D seg- mentation results across coarse and fine-grained queries, enabling explicit hierarchical understanding.
Real-world robotic application. We further demonstrate the applicability of COS3D in real-world robotic scenarios. Specifically, we leverage our method to provide effective 3D perception for robotic grasping tasks, building upon prior work [65)67]. As illustrated in Fig. 6 (c), the accurate open- vocabulary segmentation produced by our method assists the robotic arm in completing grasping operations. More details and the video demo are provided in Supp.
Conclusion
We presented COS3D, a new collaborative 3D prompt-xegmentation approach for open- opcaluocyb queries. We introduce the new concept of collaborative field comprising the instance and language fields. To achieve collaboration, we model the instance-tollanguae mapping during training and design an adaptive language-to-insaacce prompt refinement during inference. Extensive experimental results manifest effectiveness of our method over the state of the art.
Limitations. Although COS3D provides effective open-vocabulary 3D segmentation, it has the following limitations. First, COS3D lacks reasoning capabilities for segmentation, as the text- aligned language field struggles with relational or multi-object queries. Moreover, following recent approaches [10 12 [T], COS3D adopts the offline setting, and extending COS3D the online setting would be beneficial. We provide more discussion these potential extensions in Supp.
10

---

Acknowledgements
This study was funded by the InnoHK initiative of the Innovation and Technology Commission of the Hong Kong Special Administrative Region Government via the Hong Kong Centre for Logistics Robotics; and Hong Kong Innovation and Technology Fund under Project MHP//92/22.
References
[1] R. Qi, H. Su, K. Mo, and Guibas, "PointNet: Deep learning on point sets for 3D classification and segmentation," in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652-6666 0600 2017.
[2] C. Qi, I.. Yi, H. Su, and I.. J. Guibas, "PointNet++:: Deep hicrarchical feature learning on point sets in a metric space," Advances in neural information processing systems, vol. 30, 2017.
[3] G. Qian, Y. Li, H. Peng, Mai, H. Hammoud, M. Elhoseiny, and Ghanem, "PointNext: Revisiting PointNet++ with improved training and scaling strategies," Advances in neural information processing systems, vol. 35, pp. 23192 232044, 2022.
[4] D. Robert, H. Raguet, and L. Landrieu, "Efficient 3D semantic segmentation with superpoint transformer." in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 17195-17204, 2023.
[5] G. Riegler, A. Osman Ulusoy, and A. Geiger, "OctNet: Learning deep 3D representations at high resolu- tions," in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3577-3586, 2017.
[6] Y. Siddiqui, I.. Porzi, S. R. Buld, N. Müller, M. Nießner, A. Dai, and P. Kontschieder, "Panoptic lifting for 3D scene understanding with neural fields," in Proceedings of the IEEFJCVF Conference on Computer Vision and Pattern Recognition, pp. 9043-9052 9052, 2023.
[7] Y. Bhalgat, I. Laina, J. Henriques, A. Zisserman, and A. Vedaldi, "Contrastive Lift: 3D object instance segmentation by fow-fast contrastive fusion," arXiv preprint (xXx:23066.0666 2023.
[8] R. Zhu, Qiu, Q. Wu, K.-H. Hui, P.-A. Heng, and C.-W. Fu, "PCF-Lift: Panoptic lifting by probabilistic contrastive fusion," in European Conference on Computer Vision, pp. 92-108, Springer, 2024.
[9] Kerr, M. Kim, K. Goldberg, A. Kanazawa, and M. Tancik, "LERF: Language embedded radiance fields," in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19729-19733 2023.
[10] Qin, W. Li, J. Zhou, H. Wang. and H. Pfister, "LangSplat: 3D language Gaussian splatting," " in Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20051-20060, 2024.
[11] J.-C. Shi, M. Wang. H.-B. Duan, and S.-H. Guan, "Language embedded 3D Gaussians for open- vocabuacyy scene understanding," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5333 5343,, June 2024.
[12] K. Jun-Seong, G. Kim, K. Yu-Ji, Y.-. F. Wang, J. Choe, and T.-H. Oh, "Dr. Splat: Directly referring 3D Gaussian splatting via direct language embedding registration," arXiv preprint 25Xiv:2502.16665 2025.
[13] Y. Ji, H. Zhu, J. Tang, W. Liu, Z. Zhang, X. Tan, and Y. Xie, "FastLGS: Speeding up language embedded Gaussians with feature grid mapping." in Proceedings of the AAAI Conference on Artificial Intelligence. vol. 39, pp. 3922-3930, 2025.
[14] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., "Learning transferable visual models from natural language supervision," in International conference machine learning, pp. 8748 8763, PMLR, 2021.
[15] M. Yc, M. Danelljan, F. Yu, and L. Kc, "Gaussian grouping: Segment and edit anything in 3D scenes," arXiv preprint arXiv:2312.00732 2023.
[16] Y. Wu, J. Meng, H. Li, C. Wu, Y. Shi, X. Cheng, C. Zhao, H. Feng, F. Ding, J. Wang, et al,, "OpenGaussian: Towards point-level 3D Gaussian-based open vocabulary understanding." arXiv preprint 2406022406..0058 2024.
[17] H. Li, Y. Wa, J. Meng, Q Gao, Z. Zhang, R. Wang, and J. Zhang, "InstanceGaussian: sppearrance-sematiic joint Gaussian representation for 3D instance-level perception." arXiv preprint arXiv:2411.19235. 2024.
11

---

[18] S. Liu, Z. Zeng. T. Ren, F. Li, H. Zhang, J. Yang. Q. Jiang, C. Li, J. Yang. H. Su, et al., "Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection," in European Conference on Computer Vision. pp. 38-55, Springer, 2024.
[19] B. Mildenhall, P. Srinivasan, M. Tancik, T. Barron, R. Ramamoorthi, and R. Ng. "NeRF: Representing scenes as neural radiance fields for view synthesis," Communications of the ACM, vol. 65, no. 1. pp. 99-106, 2021.
[20] A. Chen, Xu, A. Geiger, J. Yu, and H. Su, "TensoRF: Tensorial radiance fields." in Eurapean Conference on Computer Vision, pp. 333-350, Springer, 2022.
[21] L. Liu, J. Gu, K. Zaw Lin, T.S. Chua, and C. Theobalt, "Neural sparse voxel fields." Advances in Neural Information Processing Systems, vol. 33, pp. 15651-15663, 2020.
[22] T. Müller, A. Evans, C. Schied, and A. Keller, "Instant neural graphics primitives with a multiresolution hash encoding," " ACM transactions on graphicy (TOG), vol. 41, no. 4, pp. 1-15, 2022.
[23] B. Kerbl, G. Kopanas, I. Leimkühher. and G. Drettakis, "3D Gaussian splatting for real-time radiance field rendering," " ACM Transactions on Graphics. vol. 42, no. 2023.
[24] T.-X. Xu, W. Hu, Y.K. Lai, Shan, and S.HH. Zhang, "Texture Disentangling the geometry and texture for 3D Gaussian splatting editing," arXiv preprint M..i::24033100500 2024.
[25] 7. Liang, Q. Zhang, W. Hu, Feng, I. Zhu, and K. Jia, "Analytic-Splattinn Anti-aliased 3D Gaussian splatting via analytic integration." arXiv preprint urXiv:2403.11055 2024.
[26] Z. Zhang, W. Hu, Y. Lao, T. He, and H. Zhao, "Pixel-GS" GS: Density control with pixel-aware gradient for 3D Gaussian splatting," arXiv preprint arXiv:2403. 243:2003.55530, 2024.
[27] K. Cheng, X. Long, K. Yang, Y. Yao, W. Yin, Y. Ma, W. Wang, and X. Chen, "GaussianPro: 3D Gaussian splatting with progressive propagation," in Forry-first International Conference Machine Learning, 2024.
[28] B. Huang. Z. Yu, A. Chen, A. Geiger, and S. Gao, "2D Gaussian splatting for geometrically accurate radiance fields," ACM SIGGRAPH 2024 Conference Papers. pp. 1-11, 2024.
[29] Z. Yu, A. Chen, B. Huang, T. Sattler, and A. Geiger, "Mip-Splatting: Alias-free 3D Gaussian splatting," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19447-19456, 2024.
[30] O. Seiskari, J. Ylilammi, V. Kaatrasalo, P. Rantalankila, M. Turkulainea, J. Kannala, E. Rahtu, and A. Solin, "Gaussian splatting on the move: Blur and rolling shutter compensation for natural camera motion," in European Conference on Computer Vision, pp. 160 177, Springer, 2024.
[31] D. Zhang, C. Wang, W. Wang, P. Li, M. Qin, and H. Wang, "Gaussian in the wild: 3D Gaussian splatting for unconstrained image collections," in European Conference on Computer Vision, pp. 341-359, Springer, 2024.
[32] Kulhanek, S. Peng. Z. Kukelova, M. Pollefeys, and Sattler, "WildGaussians: 3D Gaussian splatting in the wild," arXiv preprint arXiv:2407.08447, 2024
[33] G. Kopanas, J. Philip, T. Leimkühler, and G. Drettakis, Print-based neural rendering with per-view optimization," in Computer Graphics Forum, vol. 40, pp. 29-43, Wiley Online Library, 2021.
[34] B. Li, K. Q. Weinberger, S. Belongie, V. Koltun, and R. Ranftl, "Language-driven semantic segmentation," arXiv preprint S.XXiv:2201.0354: 2022.
[35] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, "Emerging properties in self-supervised vision transformers." in Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650 9660, 2021.
[36] R. Ding, J. Yang, C. Xue, W. Zhang, S. Bai, and X. Qi, "PLA: Language-driven open-vocabulary 3D scene understanding."" " in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 7010-7019,201
[37] K. M. Jatavallabhula, A. Kuwajerwala, Q Gu, M. Omama, T. Chen, A. Maalouf, S. Li, G. Iyer, S. Saryazdi, N. Keetha, et al. "ConceptFusion: Open-set set multimodal 3D mapping," arXiv preprint arXiw:2302.07241 2023.
12

---

[38] L. Jiang, S. Shi, and B. Schiele, "Open-vocabulary 3D semantic segmentation with foundation models," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21284-21294, 2024.
[39] M. Liu, Y. 7hu, H. Cai, S. Han, Z. Ling, F. Porikli, and H. Su, "PartSLIP: Low-shot part segmentation for 3D point clouds via pretrained image-language models," " Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 21736-21746, 2023.
[40] S. Peng. K. Genova, C. Jiang. A. Tagliasacchi, M. Pollefeys, T. Funkhouser, et al., "OpenSeene: 3D scene understanding with open vocabularies." in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 815 815-824, 2023.
[41] J. Yang, R. Ding, W. Deng. 7. Wang, and X. Qi, "RegionPIC:: CC: Regional point-language contrastive learning for open-world 3D scene understanding,"" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19823-19832. 2024.
[42] Zhang, Dong, and Ma, "CLIP-FO3D Learning free open- 3D scene representations from 2D dense CLIP," in Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 2048- 2059, 2023.
[43] M. E. A. Boudjoghra, A. Dai, J. Lahoud, H. Cholakkal, R. M. Anwer, S. Khan, and F. S. Khan, "Open-YOl 3D; Towards fast and accurate open-vocabulary 3D instance segmentation." arXiv preprint arXiv:2406.0255 2024.
[44] F. Enngelmann, F. Manhardt, M. Niemeyer, K. Tateno, M. Pollefeys, and F. Tombari, "OpenNeRF: Open 3D neural scene segmentation with pixel-wise features and rendered novel views," arXiv preprint arXiv:2404.0m 033600. 2024.
[45] Y. Bhalgat, I. Laina, F. Henriques, A. Zisserman, and A. Vedaldi, "N2F2: Hicrarchical seene understand- ing with nested neural feature fields," in European Conference on Computer Vision, pp. 197-214, Springer, 2024.
[46] J. Guo, X. Ma, Y. Fan, H. Liu, and Q Li, "Semantic Gaussians: Open-vocabulary scene understanding with 3D Gaussian splatting," arXiv preprini arXiv:2403.156 15624, 2024.
[47] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollar, and R. Girshick, "Segment anything," in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4015-4026, Octoher 2023.
[48] F. Li, H. Zhang, P. Sun, X. Zou, S. Liu, J. Yang, C. Li, L. Zhang, and J. Gao, "Semantic-SAM:: Segment and recognize anything at any granularity," arXiv preprint A.Xiv:3307.04760 2023.
[49] Y. Xiong, B. Varadarajan, L. Wu, X. Xiang, F. Xiao, C. Zhu, X. Dai, D. Wang, F. Sun, F. landola, al., "EfficientSAM: Leveraged masked image pretraining for efficient segment anything," in Proceedings of the JEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 161 11-16121, 2024.
[50] L. Ke, M. Ye, M. Danelljan, Y.-W. Tai, C.-K. Tang, F. Yu, et al., "Segment anything in high quality." Advances in Neural Information Processing Systems, vol. 36, pp. 29914 229944, 2023.
[51] C. Zhang, D. Han, Y. Qiao, J. U. Kim, S.H. Bac, S. Lee, and C. S. Hong, "Faster segment anything: Towards lightweight SAM for mobile applications," arXiv preprint arXiv:2306. 14289, 2023.
[52] C. Zhang, D. Han, S. Zheng, J. Choi, T.-H. Kim, and C. S. Hong, "MobileSAMv2: Faster segment anything to everything." arXiv preprint arXiv:2312.0957: 09579, 2023.
[53] J. Cen, Z. Zhou, J. Fang, W. Shen, L. Xie, D. Jiang, X. Zhang, Q. Tian, et al., "Segment anything in 3D with NeRFs," Advances in Neural Information Processing Systems, vol. 36, pp. 25971 2023.
[54] S. Choi, H. Song, Kim, T. Kim, and H. Do, :auusa-a: Interactive segmentation any 3D Gaussians," " arXiv preprint arXiv:2407. 11793, 2024.
[55] H. Ying, Y. Yin, J. Zhang, F. Wang, T. Yu, R. Huang, and L. Fang, "OmniSeg3D: Omniversal 3D segmentation via hicrarchical contrastive lcarning," " in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 20612-20622, 2024.
[56] C. M. Kim, M. Wu, J. Kem, K. Goldberg, M. Tancik, and A. Kanazawa, "GARField: Group anything with radiance fields." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
pp.
13

---

[57] A. Kirillov, K. He, R. Girshick, C. Rother, and P. Dollár, "Panoptic segmentation." in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9404-9413, 2019.
[58] M. C. Silva, M. Dahaghin, M. Toso, and A. Del Bue, "Contrastive Gaussian clustering: Weakly supervised
3D scene segmentation." arXiv preprint arX1:2240.12784, 2024.
[59] R. Zhu, S. Qiu, Z. Liu, K.-HH Hui, Q. Wu, P.-. Heng, and C.-W. Fu, "Rethinking end-to-end 2D to 3D scene segmentation in Gaussian splatting." arXiv preprint arXiv:2503. 14029, 2025.
[60] E. A. Nadaraya, "On estimating regression," " Theory of Probability & Its Applications, vol. 9, no. 1, pp. 141-142, 1964.
[61] H. Li, R. Qin, Z. Zou, D. He, B. B. Dai, D. Zhang, and J. Han, "LangSurf: Language-embedded surface Gaussians for 3D scene understanding." arXiv preprint arXiv:2412. 17635,, 2024.
[62] A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner, "Scannet: Richly- hinhly-annotated
3D reconstructions of indoor scenes," " in Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5828-5839, 2017.
[63] X. Zhai, B. Mustafa, A. Kolesnikov, and T.. Beyer, "Sigmoid loss for language image pre-training," in Proceedings of IEEE/CVF international conference on computer vision, pp. 11975-11986 2023.
[64] N. Ravi, V. Gabeur, Y.-T. Hu, Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, Rolland, L. Gustafson, al., "SAM 2: Segment anything images and videos," " arXiv preprint arXiv:2408.007/ 2024.
[65] Y. Zheng, X. Chen, Y. Zheng, Gu, R. Yang, Jin, P. Li, Zhong, Z. Wang, L. Liu, et al., "Gaus- sianGrasper: 3D language Gaussian splatting for open- vocabulary robotic grasping." IEEE Robotics and Automation Letters, 2024.
[66] Tang, J. Pan, Y.-H. Liu, M. Tomizuka, L. E, Li, C.-W. Fu, and M. Ding, "GeoManip: Geometric constraints as general interfaces for rohot manipulation," arXiv preprint arXiv:2501.09783, 2025.
[67] A. Rashid, S. Sharma, C. M. Kim, J. Kerr, Y. Chen, A. Kanazawa, and K. Goldberg. "Language embedded radiance fields for zero-shot task-oriented grasping," in 7th Annual Conference on Robot Learning, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We clarify our contributions in Sec. | and experimentally verify them in
Sec. 4
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
We discuss the current limitations regarding 3D reasoning segmentation for handling complicated queries in See.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer:
[NA]
Justification:
This paper does not involve theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We disclose the experimental information in Sec. 3,4 Sec. the supp.
Sec.
4.2 and
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Both data and code our work will be publicly available on github.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ praips.cc/ubblicgguuuee CoodeSumeeieeeeee ssi iessonPolll mollicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experimental settings and details are presented in Sec. 4.1 Sec. 4.2 and the supp.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We follow the common practice of existing works to report the experimental results, particularly OpenGaussian (published in NeurIPS 2024)
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide machine information in Sec. 3.4 We provide the training time and inference time in the ablation study Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics atttps://neurrs wttp:ppp.cpppbpbbpe httpc///pcbccee/tccsssieeeeeeeee
Answer: [Yes]
Justification:
We have already confirmed with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss possible societal impacts of it in Sec. 4.4 about its robitics applica-
tions.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not involve the release of data or models that pose high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We apply LeRF and ScanNetv2 dataset and 3D-GS, OpenGaussian codebase for our method implementation and testing, all of them are publicly available and have already used in published papers.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
21


Reviews:
[{"review_id": "FhDarrlT4X", "reviewer": "Reviewer_S9Jm", "summary": "This paper proposes COS3D, a framework for open-vocabulary 3D segmentation over reconstructed Gaussian scenes. The method builds a collaborative representation consisting of two aligned fields: an instance field that captures segmentation features trained from 2D instance masks (e.g., via SAM), and a language field that encodes CLIP-aligned features via a two-stage instance-to-language mapping. During inference, a text or image prompt produces a relevance map in the language field, which is adaptively refined using the instance field to generate accurate object-level 3D segmentations. COS3D outperforms prior methods on LeRF and ScanNet V2 benchmarks and supports flexible queries and applications such as hierarchical segmentation and robotic grasping.", "questions": "1. Could COS3D be extended to handle gradually reconstructed 3D scenes, where the collaborative fields are updated incrementally as new observations arrive?\n2. How would the current architecture support relational or multi-object queries, such as “all mugs on the shelf” or “the red chair next to the table”? Are such extensions feasible without major redesign?\n3. Have the authors conducted any indirect comparisons or qualitative assessments against recent methods like InstanceGaussian or Dr.Splat, given their similar use of Gaussian scene representations?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength:**\n1. COS3D introduces a dual-field architecture with an explicit mapping from instance to language space, enabling effective fusion of geometry-aware segmentation cues and CLIP-aligned semantics. This separation allows targeted supervision and mutual refinement between fields.\n2.The two-stage training procedure decouples segmentation learning and language alignment, resulting in better mIoU and faster convergence than joint or parallel alternatives (e.g., 50.76 vs. 43.84 mIoU; 50 min vs. 165 min training). This makes COS3D practical for scalable deployment.\n3. COS3D achieves state-of-the-art results on both synthetic (LeRF) and real-world (ScanNet V2) datasets, and generalizes to multiple query modalities (text, image) and tasks (e.g., robotics). Inference is fast (~0.22s/query), enabling interactive or embodied use.\n\n**Weaknesses:**\n1. COS3D relies on precomputed 3D Gaussian scenes and high-quality 2D masks from SAM. It is not directly applicable to raw point clouds, online reconstruction, or streaming sensor input, which may restrict use in dynamic or large-scale outdoor environments.\n2. The system handles only one object query at a time and cannot support relational, spatial, or multi-object reasoning (e.g., “the chair next to the table”). The authors acknowledge this but do not provide architectural insights into potential extensions.\n3. Due to unavailability of code, methods like InstanceGaussian [1] and Dr.Splat [2] are not included in ScanNet V2 evaluations. While understandable, this omission limits clarity on how COS3D ranks among the latest Gaussian-based 3D segmentation methods.\n\n**Reference:**\n1. Haijie Li et al. InstanceGaussian: Appearance-Semantic Joint Gaussian Representation for 3D Instance-Level Perception. CVPR 2025.\n2. Jun-Seong Kim et al. Dr. Splat: Directly Referring 3D Gaussian Splatting via Direct Language Embedding Registration. CVPR2025.", "was_revised": true, "final_justification": "The rebuttal has addressed my concerns regarding the choice of 3DGS over raw point clouds, the potential for online reconstruction, and the lack of comparisons with InstanceGaussian and Dr. Splat. The additional results on ScanNetV2 clarify the performance advantage over these baselines. I therefore maintain my positive recommendation.", "rebuttal": "We thank the reviewer for appreciating our work (e.g., achieving state-of-the-art results, being fast, making COS3D practical for scalable deployment, supporting flexible queries and applications) and for their inspiring and thoughtful comments, which have helped us improve our paper. We are pleased to address the issues raised in the review.\n\n**Q1:** …Gaussian splatting…not directly applicable to raw point clouds, online reconstruction ...    \n**A1:**\n- **Raw point clouds:** Thanks. Our COS3D adopts the 3D Gaussian splatting (3DGS) representation reconstructed from multi-view image inputs, with the following considerations:  First, Our 3DGS model can be derived from multi-view images, which are easy to acquire, widely available, and offer higher resolution along with rich textural information. In contrast, raw point clouds typically require specialized equipment for collection and are more expensive. Second, compared to raw point clouds, the 3D Gaussian representation enables the utilization of 2D foundation models like CLIP and SAM which contain rich world-level knowledge; thus, the 3DGS representation improves our model’s capability on the open-vocabulary segmentation.  Besides, the 3D Gaussian splatting is a commonly-used 3D representation of most recent open-vocabulary 3D segmentation (OV3DS) approaches  [1, 2, 3, 4, 5].\n\n- **Extension to online reconstruction**: Thank you. Following recent approaches [1, 2, 3, 4, 5], we evaluate our model on standard offline benchmarks. It is an interesting direction to incorporate an online setting. To do so, one possible solution is to integrate our COS3D with existing online Gaussian reconstruction methods [6, 7,8, 9] to update our collaborative fields incrementally. We acknowledge that efficient online OV3DS requires additional research efforts, and we leave this for future work.\n\n**Q2:** Discussion on potential extension to relational or multi-object queries.  \n**A2:** Thanks. This work currently focuses on the OV3DS task, and the proposed collaborative framework (COS3D) significantly surpasses previous state-of-the-art methods. We plan to explore relational reasoning in future work, as noted in our limitations. While extending our framework to support relational reasoning remains an open question, COS3D provides a solid foundation. For instance, the recent study [10] demonstrates that reasoning 3D segmentation (e.g., querying \"the red chair next to the table\") can be achieved to some extent by employing an LLM as an agent in combination with an OV3DS approach as a tool. We will provide more discussion on the potential extension in the revised version.\n\n\n**Q3:** More comparisons with InstanceGaussian and Dr. Splat on ScanNetv2.  \n**A3:** Thank you for your reminder. Since InstanceGaussian and Dr. Splat were not open-sourced at the submission deadline, we did not include their results on ScanNetv2. Recently, we observed that InstanceGaussian and Dr. Splat were open-sourced thus we include more comparisons here. Considering the inconsistent evaluation protocols (as mentioned in the main paper, L282–L284) and input (i.e., segmentation mask) in these baselines, we ran the official codes of InstanceGaussian and Dr. Splat on ScanNetv2 using the same input and evaluation protocol as ours, following OpenGaussian, to ensure fair comparisons. The results presented in the following table demonstrate that our methods outperforms InstanceGaussian and Dr.Splat in all metrics.\n\n\n**Table 1.** Comparison with the latest baselines on ScanNetv2.  Our * are sourced from Table 2 in our main paper.\n| Method             | 19 classes        |                   | 15 classes        |                   | 10 classes        |                   |\n|--------------------|:----------------:|:-----------------:|:----------------:|:-----------------:|:----------------:|:-----------------:|\n|                    | mIoU $\\uparrow$             | mAcc $\\uparrow$              | mIoU $\\uparrow$            | mAcc $\\uparrow$              | mIoU $\\uparrow$             | mAcc $\\uparrow$             |\n| Instance Gaussian [5]  | 30.21            | 47.88             | 33.77            | 51.63             | 42.32            | 59.78             |\n| Dr. Splat [4]          | 23.35            | 40.17             | 30.06            | 49.13             | 38.44            | 55.33             |\n| Ours *               | 32.47            | 49.05             | 35.95            | 54.35             | 44.32            | 63.66             |\n\n\n[1] LangSplat: 3D Language Gaussian Splatting. CVPR2024.  \n[2] LEGaussians: Language Embedded 3D Gaussians for Open-Vocabulary Scene Understanding. CVPR2024.  \n[3] OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understanding. NeurIPS2024.  \n[4] Dr. Splat: Directly Referring 3D Gaussian Splatting via Direct Language Embedding Registration. CVPR2025.  \n[5] InstanceGaussian: Appearance-Semantic Joint Gaussian Representation for 3D Instance-Level Perception. CVPR2025.  \n[6] Gaussian Splatting SLAM. CVPR2024.  \n[7] Gaussian-SLAM: Photo-realistic Dense SLAM with Gaussian Splatting. ArXiv.   \n[8] RGB-Only Gaussian Splatting SLAM for Unbounded Outdoor Scenes. ICRA 2025.  \n[9] Outdoor Monocular SLAM with Global Scale-Consistent 3D Gaussian Pointmaps. ICCV 2025.  \n[10] LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agent. ICRA2024."}, {"review_id": "27Wiq17VkG", "reviewer": "Reviewer_etuw", "summary": "This paper proposes COS3D, a collaborative prompt-based framework for open-vocabulary 3D segmentation using 3D Gaussian Splatting. It introduces a collaborative field comprising an instance field and a language field. During training, an instance-to-language mapping is used to construct the language field efficiently. During inference, an adaptive language-to-instance prompt refinement is applied to improve segmentation quality. The method outperforms previous approaches on LeRF and ScanNetv2, demonstrating strong performance and practical applicability.", "questions": "1. The authors state that the proposed method lacks reasoning ability. Have the authors considered replacing CLIP with more advanced language models such as Qwen or GPT in the main pipeline, beyond the experiments in the supplementary material? Would such replacement lead to improved reasoning capability and segmentation performance?\n2. The paper provides two implementation versions but lacks deeper analysis in the experiment section. Could the authors provide deeper insights into when each version is expected to perform better?\n3. The kernel regression approach fixes the hyperparameter σ. Could the authors provide an ablation study on σ to validate its influence on performance?\n4. The training time comparison in Table 5 is not sufficient, as only one method reports its training time. Could the authors include training time for most baselines to better support the claim of COS3D’s superior training efficiency?\n5. There is a minor typo in line 318: Table 5 is incorrectly referred to as “Table 4.3” in the main text.", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The idea of constructing a collaborative field comprising both language and instance features is quite novel and addresses the limitations of prior approaches that rely solely on either.\n2. The proposed method achieves strong performance improvements over state-of-the-art on LeRF and ScanNetv2 benchmarks in both mIoU and mAcc.\n3. The paper is well-organized and includes informative visualizations and ablation studies.\n\nWeaknesses:\n1. The authors mention that the proposed method lacks reasoning ability. While Qwen is used in the supplementary materials, the main method still relies on CLIP for language features. It would be valuable to explore whether replacing CLIP with more advanced language models like GPT or Qwen could enhance the reasoning capacity of the language field. I am particularly curious whether such substitutions could lead to performance gains.\n2. While both MLP and kernel regression versions are implemented, the kernel regression appears hand-crafted with hyperparameters fixed. Moreover, the authors do not sufficiently analyze when or why one version might fail or perform better in the experiment section.\n3. The training time comparison in Table 5 is insufficient. For example, one of the baseline methods, Instance Gaussian, lacks reported training time, and the training time of other competing methods in Table 1 and 2 is not reported either. This provides little support to COS3D’s claimed superior training efficiency.", "was_revised": true, "final_justification": "The rebuttal clear my doubts and I would keep my rating.", "rebuttal": "We are encouraged to see that the reviewer finds the proposed method novel, effective in addressing the limitations, achieving strong performance improvements, and well-organized. We sincerely thank the reviewer for their constructive comments (e.g., discussions on extensions and additional analysis), which help enhance the quality of our paper. We are pleased to address the issues raised in the review.\n\n\n**Q1:** Potential extension to reasoning ability.   \n**A1:** Thanks for your question. This work currently focuses on the open-vocabulary 3D segmentation task, and we plan to explore reasoning ability in our future work, as noted in our limitations. Moreover, incorporating reasoning abilities using advanced LLM models is a promising direction. Specifically, it is technically feasible to replace the CLIP[1] feature with the vision encoder feature from a V-LLM model, such as Qwen-vl2.5 [2]. However, how to effectively integrate the learned 3D Qwen feature fields with the LLM backbone to handle the reasoning queries and enhance segmentation quality remains an open question, as the relevance between the 3D Qwen feature fields and the text queries cannot be directly modeled using feature distance (e.g., cosine similarity) as in CLIP-style models (e.g., CLIP[1], SigLIP[3] ). One possible solution is to render the 3D Qwen feature field into a 2D feature map based on a reference view, allowing it to be fed into the LLM backbone along with reasoning queries. It is an interesting research problem to design comprehension solutions that enable interaction between 3D LLM-style feature fields and reasoning queries. We will provide more discussion on the future work in the revised version.\n\nIn our supplementary robotic demo, we incorporate reasoning ability to some extent using a two-stage strategy: first, the Qwen-VL model parses complex commands into short queries; then, our COS3D model predicts the 3D segmentation results based on the queries.\n\n\n**Q2:** Analysis between kernel regression and the MLP counterpart.  \n**A2:** Thanks for your interesting question. Both kernel regression and the MLP counterpart are means to achieve our instance-to-language feature mapping function, and the improved performance achieved by these two strategies consistently demonstrates the effectiveness of our feature mapping design.\n\nMoreover, we provide a discussion from the perspectives of accuracy and memory efficiency:\n\n- **Accuracy**: We observe that the kernel regression version tends to achieve better performance than the MLP counterpart in our experiments. We attribute this to the selection of discriminative instance features as the source domain, which makes the mapping learning process inherently an easy regression task, where the traditional kernel regression method is extremely suitable. Moreover, kernel regression benefits from its training-free property, bypassing the unstable optimization process required by the MLP counterpart. \n\n\n- **Memory efficiency:** On the other hand, the MLP counterpart, consisting of small layers, offers better GPU memory efficiency. Specifically, the computational complexity of kernel regression depends on the size of the training data, resulting in higher memory requirements. \n\nTo illustrate this, we report the following table comparing the accuracy and GPU memory usage of kernel regression and MLP, using an identical input batch size of 100K.\n\n**Table 1.** Trade-off between accuracy and memory efficiency on LeRF.\n\n| Type              | mIoU $\\uparrow$   | mAcc $\\uparrow$   | GPU Memory $\\downarrow$  |\n|-------------------|-------|-------|------------|\n| MLP               | 49.75 | 70.60 | 819M       |\n| Kernel regression | 50.76 | 72.08 | 6863M      |\n\n\nWe will incorporate more discussion in the revised version.\n\n\n**Q3:** Hyperparameter analysis in kernel regression.  \n**A3:** Thanks. For kernel regression, we set this hyperparameter to 0.1 across all experiments, and it performs well without requiring additional tuning. Additionally, we conducted an ablation study on $\\sigma$, with the results presented in the following table. Overall, the results remain stable under moderate changes to $\\sigma$, and our method consistently outperforms the latest state-of-the-art approach, InstanceGaussian [4].\n\n\n**Table 2.** Quantitative comparisons for different $\\sigma$ values in kernel regression.\n\n| $\\sigma$            | mIoU $\\uparrow$  | mAcc $\\uparrow$ |\n|------------------|:-----:|:------:|\n| 0.05             | 51.00 | 71.38  |\n| 0.08             | 50.55 | 70.59  |\n| 0.10 (default)   | 50.76 | 72.08  |\n| 0.15             | 50.55 | 71.23  |\n| 0.20             | 50.58 | 71.25  |\n| InstanceGaussian[4]             | 45.30 | 58.44  |\n\n\n**Q4:** Training times.  \n**A4:** Thank you. We originally plotted the training time of most baselines in the main paper Fig. 1(d) and included partial information in the main paper Tab. 5 due to space constraints. For clarity, we have organized the complete information into the table below. We will update the full table in the supplementary materials.\n\n**Table 3.** Training efficiency analysis on LeRF. We report mIoU.\n\n| Method                   | Training time $\\downarrow$  | mIoU $\\uparrow$  |\n|--------------------------|:-------------:|:-----:|\n| Langsplat[5]                | 240 min       | 9.66  |\n| LEGaussians[6]              | 240 min       | 16.21 |\n| Dr. Splat [7]               | 10 min        | 43.58 |\n| OpenGaussian[8]             | 60 min        | 38.36 |\n| InstanceGaussian[4]         | -             | 45.30 |\n| Ours (3K for instance)   | 8 min         | 50.16 |\n| Ours (6K for instance)   | 15 min        | 50.24 |\n| Ours (default)           | 50 min        | 50.76 |\n\n**Q5:** Typo.  \n**A5:** Thank you for pointing out the typo in L318. We will correct this in the revised version.\n\n[1] Learning transferable visual models from natural language supervision. ICML 2021.   \n[2] Qwen2.5-VL Technical Report. ArXiv.    \n[3] Sigmoid Loss for Language Image Pre-Training. ICCV 2023.  \n[4] InstanceGaussian: Appearance-Semantic Joint Gaussian Representation for 3D Instance-Level Perception. CVPR 2025.  \n[5] LangSplat: 3D Language Gaussian Splatting. CVPR 2024.  \n[6] LEGaussians: Language Embedded 3D Gaussians for Open-Vocabulary Scene Understanding. CVPR 2024.  \n[7] Dr. Splat: Directly Referring 3D Gaussian Splatting via Direct Language Embedding Registration. CVPR 2025.  \n[8] OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understanding. NeurIPS 2024."}, {"review_id": "QIBiQRKBUo", "reviewer": "Reviewer_yYMN", "summary": "The paper COS3D: Collaborative Open-Vocabulary 3D Segmentation proposes novel take on the problem of 3D segmentation by using both a instance field and a language field, both of which are connected though a learned mapping (MLP or kernel). In addition, the authors propose an inference strategy to improve segmentation quality by proposing an adaptive language-to-instance prompt refinement. Finally, the method is evaluated across a set of 3D segmentation benchmarks.", "questions": "My main remaining issue is the lack of ablations for the 2D foundation models. How does the performance change if you use other VLMs / Segmentation Models, for example the ones cited above?\nAdditionally, how is SAM used/prompted to generate the 2D instance masks?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- The proposed technical novelties are interesting and effective\n- The concepts of creating instance and language fields, taken together as the collaborative field, is sound and seems to work well\n- I appreciate the technical discussion parts after each of the methods subsections\n- The mechanism for adaptive prompt refinement makes sense and further improves the overall method\n- The method is somewhat ablated\n\nWeaknesses:\n- I’m completely missing ablations on how robust the approach is when using VLMs alternative to CLIP or segmentation models other than SAM. Since the method relies heavily on these off-the-shelf models, I think ablating 2 other choices per model would be appropriate. For VLMs, I would suggest SigLIP [1] or BLIP [2]. For the segmentation model, SAM2 [3] and Semantic SAM [4] at different granularities would be interesting.\n- L. 161 states „First, we learn a segmentation-aware instance field supervised by the 2D SAM segmentation“, but in the following section Stage 1 paragraph, there is no mention of how SAM is used/prompted to generate the 2D instance masks. The authors should be more clear in this section.\n- Several typos in Figure 2 where the authors write filed instead of field \n- Same in Figure 4, the authors write „Ous“ instead of „Ours“\n\n[1] Zhai, Xiaohua, et al. \"Sigmoid loss for language image pre-training.\" Proceedings of the IEEE/CVF international conference on computer vision. 2023.\n[2] Li, Junnan, et al. \"Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.\" International conference on machine learning. PMLR, 2022.\n[3] Ravi, Nikhila, et al. \"Sam 2: Segment anything in images and videos.\" arXiv preprint arXiv:2408.00714 (2024).\n[4] Li, Feng, et al. \"Segment and Recognize Anything at Any Granularity.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.", "was_revised": true, "final_justification": "The authors have addressed my concerns sufficiently. Therefore, I believe the paper is ready for acceptance and raise my score.", "rebuttal": "We thank the reviewer for their helpful comments and suggestions (e.g., ablation on different VLMs), which have helped enhance the quality of our paper. We are pleased to address the issues raised in the review.\n\n**Q1:** Ablation on different VLMs.  \n**A1**: Thank you for your suggestions. We utilize CLIP and SAM as the default 2D language and segmentation models in our experiments, following common practice in recent baselines [1, 2, 3, 4, 5], to ensure that the improvements are attributed to the proposed collaborative fields design.   \n\nHowever, our methods can adopt other foundation models. Based on your suggested alternatives, we conducted an ablation study comparing different 2D foundation models (e.g., CLIP [6] vs. SigLIP [7], SAM [8] vs. SAM2 [9] & Semantic SAM [10]). The results, shown in the following table, demonstrate that our framework is compatible with different 2D foundation models. Furthermore, we empirically observed that using more advanced models (e.g., SAM2 for segmentation, SigLIP for language) can lead to performance improvements. We sincerely appreciate your insightful suggestions, which contributed to these valuable observations!  \n\nWe will include ablation results on different 2D foundation models in the revised version of our paper.\n\n\n**Table 1.** Ablation on 2D foundation models on LeRF. \n\n| Model           | Segmentation   | Language | MIoU $\\uparrow$  | mAcc $\\uparrow$   |\n|------------------|:--------------:|:--------:|:-----:|:------:|\n| COS3D (default)  | SAM [8]            | CLIP [6]     | 50.76 | 72.08  |\n| COS3D            | SAM [8]            | SigLIP [7]   | 51.08 | 73.79  |\n| COS3D            | SAM2 [9]           | CLIP [6]     | 51.94 | 75.05  |\n| COS3D            | Semantic SAM [10]   | CLIP [6]     | 49.93 | 70.94  |\n\n**Q2:** Procedure for SAM instance mask generation.  \n**A2**: Thanks. To generate the 2D instance masks from SAM, we follow the automatic mask generation strategies commonly used in prior works [1, 2, 3, 4, 5]. Specifically, we create a grid of point prompts across the image and obtain the final segmentation results using the same hyperparameters as in OpenGaussian[3]. Further details can be found in the supplementary materials (Supp. L26–L28). We will clarify this procedure in Sec. 3.2 paragraph (“Stage 1: instance field learning.”) for the revised version.\n\n\n**Q3:** Typos.  \n**A3:**  Thank you for pointing out the typos in Fig. 2 and Fig. 4.  We will update them in the revised version.\n\n[1] LangSplat: 3D Language Gaussian Splatting. CVPR 2024.  \n[2] LEGaussians: Language Embedded 3D Gaussians for Open-Vocabulary Scene Understanding. CVPR 2024.  \n[3] OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understanding. NeurIPS 2024.  \n[4] Dr. Splat: Directly Referring 3D Gaussian Splatting via Direct Language Embedding Registration. CVPR 2025.  \n[5] InstanceGaussian: Appearance-Semantic Joint Gaussian Representation for 3D Instance-Level Perception. CVPR2025.  \n[6] Learning transferable visual models from natural language supervision. ICML 2021.   \n[7] Sigmoid Loss for Language Image Pre-Training. ICCV 2023.  \n[8] Segment Anything. ICCV 2023.  \n[9] SAM 2: Segment Anything in Images and Videos. ICLR 2025.  \n[10] Semantic-SAM: Segment and Recognize Anything at Any Granularity. ECCV 2024."}, {"review_id": "c7Jvn0dNXD", "reviewer": "Reviewer_EzJB", "summary": "This paper proposed COS3D, a new open-vocabulary 3D segmentation technique that collaborates on language and segmentation cues. COS3D utilizes a \"collaborative field\" consisting of an instance field (encoding object boundaries) and a language field (oriented with textual cues). A two-step framework is introduced: training a discriminative instance field first and then projecting these instance feature maps to language feature maps. A text prompt is utilized for initial segmentation at inference time, which is then improved with an adaptive prompt strategy. Various extensive experiments exhibit large improvements on standard benchmarks, evidencing efficacy and efficiency.", "questions": "1. How sensitive is COS3D’s adaptive refinement to the thresholds employed? Can the authors give an analysis or advice on how to choose these parameters?\n\n2. Could the authors present quantitative results on non-category queries such as materials like \"metal,\" and abstract concepts like \"small\" to better substantiate the open-vocabulary claim?\n\n3. Kernel regression is superior to its MLP counterpart at all times. Should the authors elaborate on practical trade-offs such as scalability and complexity of computation for these implementations?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n\nQuality: The methods are technically solid and well-justified. Experiments make many interesting comparisons and clearly exhibit better performance relative to strong baselines. Ablations are extensive and insightful.\n\nClarity: In general, writing and structure are clear. Key ideas—collaborative field, two-stage training, adaptive inference—are explicated sensibly and supported by useful visualizations.\n\nSignificance: The approach is highly relevant to open-vocabulary 3D segmentation and demonstrates significant improvement from existing efforts. Useful applications toward robotics and hierarchical segmentation enhance its utility.\n\nNovelty: The collaborative strategy, specifically the instance-language mapping and adaptive prompt tuning, is an original contribution.\n\nWeaknesses:\n\nAlthough the authors claim open-vocabulary capability, evaluation primarily utilizes fixed object classes. To make stronger assertions, performance would need to be demonstrated on broader queries such as attributes, materials, or abstract concepts.\n\nThe algorithm is tailored to scenes for 3D Gaussian Splatting; generalization to other 3D representations is neither obvious nor would it enhance applicability.\n\nAdaptive refinement utilizes hyperparameters (thresholds) whose sensitivity is unexplored and questions practical robustness.", "was_revised": true, "final_justification": "Thanks for addressing my questions, keep the scores.", "rebuttal": "We thank the reviewer for appreciating our work (e.g., original contribution, proposing a technically solid and well-justified method, conducting extensive experiments, and maintaining a clear structure) and for providing insightful comments (e.g., non-category queries and the trade-off between kernel regression and the MLP counterpart). We are pleased to address the issues raised in the review.\n\n\n**Q1:** Non-category queries.  \n**A1:** Thank you for the insightful suggestion. Note that the standard benchmarks (e.g., LeRF, ScanNet) commonly used in existing open-vocabulary baselines do not include evaluations for non-category queries. But we have provided some qualitative examples with non-category queries (e.g., materials: fleece, affordance: drinkable, abstract: music) in Supp. Fig. 1 and Supp. Fig. 4. These results demonstrate the capability of our approach to understand non-category queries to some extent. Both the quantitative comparisons on standard evaluations and the qualitative examples with non-category queries consistently highlight the effectiveness of our proposed COS3D.\nWe agree that building a benchmark for non-category queries and designing comprehensive quantitative evaluations is an interesting and valuable direction, which we leave for future work.\n\n**Q2:** 3D Gaussians Splatting (3DGS) representations.    \n**A2:** Thanks. Our COS3D adopts 3DGS as the 3D representation with the following considerations. First, 3DGS has become a widely used and popular representation for real 3D scenes to support efficient rendering. Moreover, it has also been adopted by recent baselines [1, 2, 3, 4, 5] for the open-vocabulary 3D segmentation task.\n- **Generalization to other 3D representations:** We believe our methods have the potential to generalize to other 3D representations, such as point clouds. For instance, recent advancements (e.g., VGGT [6]) can connect the original 2D image with 3D representations (e.g., point (cloud) maps), enabling the use of 2D foundation models to provide open-vocabulary segmentation for 3D point clouds. We leave further explorations in future work.  \n\nWe will include these discussions regarding further generalization in the revised version.\n\n**Q3:** Hyperparameter analysis in adaptive refinement.    \n**A3:** Thank you for your questions. There are two hyperparameters in the adaptive refinement algorithm: the region expansion threshold $\\mathcal{T}$  and the widely used relevance threshold $\\tau$.\n\n- **The region expansion threshold $\\mathcal{T}$:** Instead of using a fixed value for this hyperparameter, we implement a hyperparameter generation strategy as described in Algorithm 1 (see Supp Page 2). In this approach, the final expansion threshold is determined based on the calculated mean segmentation feature distances. Additionally, we conducted experiments with different expansion thresholds by applying permutations (ranging from -20% to 20%) to the mean feature distances. The results, shown in the table below, demonstrate that COS3D achieves stable performance within a reasonable range of region threshold values and consistently outperforms the latest SOTA method (InstanceGaussian[5]). \n\n**Table 1.** Quantitative comparisons of different region expansion threshold $\\mathcal{T}$ values.  We conduct the experiments under different expansion thresholds by adding a moderate permutation (from -20% to 20%) to the mean feature distances.\n\n| Perturbation  | mIoU $\\uparrow$  | mAcc $\\uparrow$   |\n|---------------|:-----:|:-----:|\n| -20 \\%        | 49.96 | 72.44 |\n| -10 \\%        | 49.90 | 72.44 |\n| 0.0 (default) | 50.76 | 72.08 |\n| +10 \\%          | 49.68 | 71.28 |\n| +20 \\%        | 49.87 | 72.18 |\n| InstanceGaussian[5]             | 45.30 | 58.44  |\n\n\n- **The relevance threshold $\\tau$:** This hyperparameter has been widely used and proven to be relatively robust in existing methods [1, 2, 3, 4, 5]. Therefore, we simply set it to a fixed value of 0.5 throughout our experiments.\n\n**Q4:** Practical analysis between kernel regression and its MLP counterpart.  \n**A4:** \nThis is a great question. First, both kernel regression and MLP are approaches used to implement our instance-to-language mapping function. The improved performance achieved by these two strategies demonstrates the effectiveness of the mapping design.\n\nWe selected these two implementations primarily because of their distinct theoretical properties. Specifically, kernel regression is a traditional fitting technique that provides strong interpretability (e.g., transparent kernel functions) and intuitive locality. In contrast, the MLP counterpart can handle flexible data patterns (both local and global) and requires only the storage of model parameters after training, making it more compact.\n\nIn practice, we observed a trade-off between accuracy and memory efficiency. On the one hand, kernel regression achieves better accuracy compared to the MLP counterpart. As discussed in the ablation section of the main paper, we attribute this to the discriminative instance features chosen as the source domain, which make the mapping process inherently an easy regression task. In such cases, the traditional kernel regression method is well-suited.\n\nOn the other hand, the MLP counterpart, consisting of small layers, offers better GPU memory efficiency. Specifically, the computational complexity of kernel regression depends on the size of the training data, resulting in higher memory requirements. To illustrate this, we report the following table comparing the accuracy and GPU memory usage of kernel regression and MLP, using an identical input batch size of 100K.\n\n**Table 2.** Trade-off between accuracy and memory efficiency on LeRF.\n\n| Type              | mIoU $\\uparrow$   | mAcc $\\uparrow$   | GPU Memory $\\downarrow$  |\n|-------------------|-------|-------|------------|\n| MLP               | 49.75 | 70.60 | 819M       |\n| Kernel regression | 50.76 | 72.08 | 6863M      |\n\n\n\n\n\n[1] LangSplat: 3D Language Gaussian Splatting. CVPR2024.  \n[2] LEGaussians: Language Embedded 3D Gaussians for Open-Vocabulary Scene Understanding. CVPR2024.  \n[3] OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understanding. NeurIPS2024.  \n[4] Dr. Splat: Directly Referring 3D Gaussian Splatting via Direct Language Embedding Registration. CVPR2025.  \n[5] InstanceGaussian: Appearance-Semantic Joint Gaussian Representation for 3D Instance-Level Perception. CVPR2025.  \n[6] VGGT: Visual Geometry Grounded Transformer. CVPR2025."}]

Author rebuttal:

Meta-review: The method proposes a novel open vocabulary method for radiance fields that relies on two aligned fields, one centered on identifying instances and a language field. The paper received a positive assessment from all reviewers, which highlighted the novelty and effectiveness of the method. Therefore, I recommend accepting the paper. However, several concerns and improvements were suggested during the reviewer-author discussion that should be included in the final version of the paper, such as additional comparisons, extended analysis and discussion of MLP vs kernel method, hyperparameter analysis, training times, and ablation studies on different VLMs.
Decision: Accept (poster)