GOI: Find 3D Gaussians of Interest with an Optimizable Open-vocabulary Semantic-space Hyperplane

Authors: yansong  qu
shaohui  dai
liujuan  cao
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-19 08:14:26

Abstract: 3D open-vocabulary scene understanding, crucial for advancing augmented reality and robotic applications, involves interpreting and locating specific regions within a 3D space as directed by natural language instructions.
To this end, we introduce GOI, a framework that integrates semantic features from 2D vision-language foundation models into 3D Gaussian Splatting (3DGS) and identifies 3D Gaussians of Interest using an Optimizable Semantic-space Hyperplane.
Our approach includes an efficient compression method that utilizes scene priors to condense noisy high-dimensional semantic features into compact low-dimensional vectors, which are subsequently embedded in 3DGS.
During the open-vocabulary querying process, we adopt a distinct approach compared to existing methods, which depend on a manually set fixed empirical threshold to select regions based on their semantic feature distance to the query text embedding. This traditional approach often lacks universal accuracy, leading to challenges in precisely identifying specific target areas. Instead, our method treats the feature selection process as a hyperplane division within the feature space, retaining only those features that are highly relevant to the query. We leverage off-the-shelf 2D Referring Expression Segmentation (RES) models to fine-tune the semantic-space hyperplane, enabling a more precise distinction between target regions and others. This fine-tuning substantially improves the accuracy of open-vocabulary queries, ensuring the precise localization of pertinent 3D Gaussians.
Extensive experiments demonstrate GOI's superiority over previous state-of-the-art methods.

Main paper: 
 introduction: 
 The field of computer vision has witnessed a remarkable evolution in recent years, driven by advancements in artificial intelligence and deep learning. A critical aspect of this progress is the enhanced ability of computer systems to interpret and interact with the threedimensional world. The growing complexity in technology use has spurred a significant demand for advanced 3D visual understanding. This evolution brings to the fore the significance of the open-vocabulary querying task [5, 28, 34] -the capacity to process and respond to user queries formulated in natural language, enabling a more natural and flexible interaction between users and the digital world. Such advancements hold the potential to enhance how human navigate and manipulate complex three-dimensional data [14, 19, 47] , bridging the gap between human cognitive abilities and computerized processing [7, 21] . Due to the scarcity of large-scale and diverse 3D scene datasets with language annotations, earlier Methods [21, 29] distill the multimodal knowledge from off-the-shelf vision-language models, such as CLIP [42] and LSeg [24] , into Neural Radiance Fields (NeRF) [35] . However, NeRF's implicit representation limits its speed and accuracy, hindering practical application. Recently, the 3D Gaussian Splatting (3DGS) [20] has emerged as an effective representation of 3D scenes, with explorations in constructing semantic fields [40, 49, 60] . This semantics lifting approach requires pixel-aligned features, whereas CLIP encodes a image into one global semantic feature. [21, 30, 49] utilize a multi-scale feature pyramid that incorporates CLIP embeddings from image crops. However, this approach results in blurred semantic boundaries, even with DINO [3] features, leading to unsatisfactory query results.In this work, we introduce 3D Gaussians Of Interest (GOI). We utilize the vision-language foundation model APE [48] to extract pixel-aligned semantic features from multi-view images. GOI leverages these semantic features to reconstruct a 3D Gaussian semantic field. Due to the explicit nature of 3DGS, embedding high-dimensional features into each 3D Gaussian is computationally demanding. To address this, we introduce the Trainable Feature Clustering Codebook (TFCC), which compresses noisy highdimensional features using scene priors. This approach significantly reduces storage and rendering costs while maintaining the informational capacity of each feature. Moreover, current open-vocabulary query strategies call for setting a fixed empirical threshold to ascertain features proximate to the query text. This, however, results in a failure to precisely query the targets. We introduce the Optimizable Semantic-space Hyperplane (OSH) to address this issue. OSH is fine-tuned by the Referring Expression Segmentation (RES) model, which aims to identify binary segmentation masks in 2D RGB images for text queries and is recognized for its robust spatial and localization capabilities. The OSH enhances GOI's spatial perception for more precise phrasal queries like "the table under the bowl", aligning query results more closely with target regions. Additionally, we have meticulously expanded and annotated a subset of the Mip-NeRF360 [1] dataset, tailored for the open-vocabulary query task. Owing to our method's proficient 3D open-vocabulary scene understanding, it is practical for a range of downstream applications, notably scene manipulation and editing.In summary, the main contributions of our work include:• We propose GOI, an innovative framework based on 3D Gaussian Splatting for accurate 3D open-vocabulary semantic perception. The Trainable Feature Clustering Codebook (TFCC) is further introduced to efficiently condense noisy high-dimensional semantic features into compact, low-dimensional vectors, ensuring well-defined segmentation boundaries.• We introduce the Optimizable Semantic-space Hyperplane (OSH), which eschews the fixed empirical threshold for relative feature selection due to its limited generalizability. Instead, OSH is finetuned for each text query with the off-the-shelf RES model to precisely locate target regions.• Extensive experiments demonstrate that our method outperforms the state-of-the-art methods, achieving substantial improvements in mean Intersection over Union (mIoU) of 30% on the Mip-NeRF360 dataset [1] and 12% on the Replica dataset [50] . 
related work 2.1 neural scene representation: 
 Recent methods in representing 3D scenes with neural networks have made substantial progress. Notably, Neural Radiance Fields (NeRF) [35] have excelled in novel view synthesis, producing highly realistic new viewpoints. However, NeRF's reliance on a neural network for complete implicit representation of scenes leads to tedious training and rendering times. Many subsequent methods [6, 12, 18, 36, 43, 44] have concentrated on improving its performance. In order to enhance the quality of surface reconstruction, [10, 13, 33, 52, 53] uses the signed distance function (SDF) for surface expression and uses a novel volume rendering scheme to learn an SDF representation. On the other hand, some approaches [8, 9, 39, 41, 55, 56] have explored the combination of implicit and explicit representations, utilizing traditional geometric structures, such as point clouds or mesh, to enhance NeRF's performance and to enable more downstream tasks. Kerbl et al. proposed 3D Gaussian Splatting (3DGS) [20] , which greatly accelerates the rendering speed of novel view synthesis and achieves high-quality scene reconstruction. Unlike NeRF that represents a 3D scene implicitly with neural networks, 3DGS represent a scene as a set of 3D Gaussian ellipsoids, and accomplish efficient rendering by rasterizing the Gaussian ellipsoids into images. The technique adopted by 3DGS, which entails encoding scene information into a collection of Gaussian ellipsoids, provides distinct advantages [25, 26, 54] . It permits easy manipulation of specific parts in the reconstructed scene without significantly affecting other components. We have extended the 3DGS to achieve open-vocabulary 3D scene perception.Foundation Models (FM) are becoming an impactful paradigm in the content of AI. They are typically pre-trained on vast amounts of data, possess numerous model parameters, and can be adapted to a wide range of downstream tasks [2] . The efficacy of 2D visual foundation models is evident in multiple visual tasks, such as object localization [31] and image segmentation [15] [16] [17] . The incorporation of multimodality substantially amplifies the perceptual ability of these models. For instance, CLIP [42] , by using contrastive learning, aligns the outputs of text encoders and image encoders into the unified feature space. Similarly, SAM [22] showcases immersive capabilities as a promptable segmentation model, delivering competitive, even superior zero-shot performance vis-à-vis earlier fully-supervised models. DINO [4, 37] , a self-supervised Vision Transformer (ViT) model, is trained on vast unlabeled images. The model deciphers a semantic representation of images, encompassing components such as object boundaries and scene layouts. Moreover, recent efforts are focused on leveraging existing pretrained models, thereby pushing the limit of Foundation Models. Grounding DINO [32] represents an open-set object detector executing target detection based on textual descriptions. It utilizes CLIP and DINO as basic encoders, and proposes a tight fusion approach for better synthesizing of visual-language information. Grounded SAM [45] integrates Grounding DINO with SAM, facilitating the detection and segmentation for arbitrary queries. APE [48] is a universal visual perception model designed for diverse tasks like segmentation and grounding. Rigorously designed visual-language fusion and alignment modules enable APE to detect anything in an image swiftly without heavy cross-modal interactions.Earlier works, such as Semantic NeRF [59] and Panoptic NeRF [11] , introduced the transfer of 2D semantic or panoptic labels into 3D radiance fields for zero-shot scene comprehension. Following this, [23, 51] capitalized on pixel-aligned image semantic features, which they lifted to 3D, rather than relying on pre-defined semantic labels. Vision-language models like CLIP exhibited impressive performance in zero-shot image understanding tasks. A subsequent body of work [21, 23, 30] proposed leveraging CLIP and CLIP-based visual encoders to extract dense semantic features from images, with the aim of integrating them into NeRF scenes.The recently proposed 3D Gaussian Splatting has achieved leading benchmarks in areas of novel view synthesis and reconstruction speed. This advancement has made the integration of 3D scenes with feature fields more efficient. LangSplat [40] , LEGaussians [49] , Feature 3DGS [60] , Gaussian Grouping [57] explored the integration of pixel-aligned feature vectors from 2D models like LSeg, CLIP, DINO and SAM into 3D Gaussian frameworks so as to enabling 3D open-vocabulary query and localization of scene areas. 
problem definition and method overview: 
 Given a set of posed images 𝐼 = {𝐼 1 , 𝐼 2 , . . . , 𝐼 𝐾 }, a 3D Gaussian scene 𝑆 can be reconstructed using the standard 3D Gaussian Splatting technique [20] based on 𝐼 . Our method expands 𝑆 with openvocabulary semantics, enabling us to precisely locate the Gaussians of interest based on a natural language query.We begin by recapping the vanilla 3D Gaussian Splatting (Sec. 3.2). Figure 2 illustrates the overview pipeline of our method. Initially, we utilize a frozen image encoder, well-aligned with the language space, to process each image 𝐼 𝑘 and derive the 2D semantic feature maps 𝑉 = {𝑉 1 , 𝑉 2 , . . . , 𝑉 𝐾 } (Sec. 3.3). To integrate these 2D high-dimensional feature maps into 3DGS, while ensuring minimal storage and optimal computational performance, Trainable Feature Clustering Codebook (TFCC) is proposed (Sec. 3.4). We expand 3DGS to reconstruct 3D Gaussian Semantic Field (Sec. 3.5) . Following this, we explain how to utilize the RES model to optimize the Semantic-space Hyperplane, thereby achieving accurate open-ended language queries in 3D Gaussians (Sec. 3.6).3D Gaussian Splatting utilizes a set of 3D Gaussians, essentially Gaussian ellipsoids, which bears a significant resemblance to point clouds, to model the scene and accomplish fast rendering by efficiently rasterizing Gaussians into images, given cameras poses. Specifically, each 3D Gaussian is parameterized by its centroid 𝑥 ∈ R 3 , a 3D anisotropic covariance matrix Σ in world coordinates, an opacity value 𝛼, and spherical harmonics (SH) 𝑐. In the rendering process, 3D Gaussians are projected on to the 2D image plane, which transforms 3D Gaussian ellipsoids into 2D ellipses. Σ is transformed to Σ ′ in camera coordinates:EQUATIONwhere 𝑊 denotes the world-to-camera tranformation matrix and 𝐽 is the Jacobian matrix for the projective transformation. In practical, Σ is decomposed into a rotation matrix 𝑅 and a scaling matrix 𝑆: This decomposition is to ensure that Σ is physically meaningful during the optimization. To summarize, the learnable parameters of the 𝑖-th 3D Gaussian are represented by 𝜃 𝑖 = {𝑥 𝑖 , 𝑅 𝑖 , 𝑆 𝑖 , 𝛼 𝑖 , 𝑐 𝑖 }.EQUATIONA volumetric rendering process, similar to NeRF, is then employed in the rasterization to compute the color 𝐶 of each pixel.EQUATIONwhere 𝐺 denotes a set of 3D Gaussians sorted by their depth, and 𝑇 𝑖 represents the transmittance, defined as the cumulative product of the opacity values of Gaussians that superimpose on the same pixel, computed through 𝑇 𝑖 = 𝑖 -1 𝑗=1 (1 -𝛼 𝑗 ).Prior research has broadly employed CLIP for feature lifting in the 3D radiance field, owing to its superior capability in managing openvocabulary queries. [23, 60] use LSeg [24] to extract pixel-aligned CLIP features. However, LSeg proves inadequate in recognizing long-tail objects. To compensate for CLIP's limitation for yielding only image-level features, methodologies such as [21, 40, 49] adopt a feature pyramid approach, using cropped image encoding to represent local features. These methods extract pixel-level features from the CLIP model, but the encoded feature maps lack geometric correspondence to the scene objects. As such, pixel-aligned DINO features are introduced and predicted simultaneously with the CLIP features, thus bounding CLIP with the object geometry. Leveraging the success of SAM, [27, 40] utilizes SAM explicitly to constrain the object-level boundaries of the features. However, using multiple models for feature extraction substantially increases the complexity for training and image prepocessing. We leverage the Aligning and Prompting Everything All at Once model (APE) [48] , which is also able to align the features of vision and language. In APE, a fixed language model formulates language features, and a visual encoder is trained from scratch. The core of the visual encoder, derived from the DeformableDETR [61] , provides APE with formidable detection and localization capacities. Additionally, APE possesses efficient modules for vision-language fusion and alignment. These modules diminish cross-modal interaction and thus reducing computational costs. Therefore, APE presents a robust solution for feature lifting. For this purpose, we make minor modifications to the APE model to extract pixel-aligned features with fine boundaries efficiently (~2s per image). We treat the encoded pixel-aligned feature maps as the pseudo ground truth features, denoted as 𝐺𝑇 .We extract APE feature maps from all training viewpoints and embed them into each 3D Gaussian to reconstruct a 3D semantic field. During the open-vocabulary querying process, we use the pretrained APE language model to encode the language prompts.Due to APE being trained on mass data and the need to align text and image features, it results in a higher feature dimensionality (256). As the previous works [40, 49] have mentioned, directly lifting high-dimensional semantic features into each 3D Gaussian results in excessive storage and computational demands. The semantics of a single scene cover only a small portion of the original CLIP feature space. Therefore, leveraging scene priors for compression can effectively reduce these costs. On the other hand, due to the inherent multi-view inconsistency of encoded 2D semantic feature map, Gaussians tend to overfit each training viewpoint, inheriting this inconsistency and causing discrepancies between 3D and 2D within an object. Therefore, we introduce the Trainable Feature Clustering Codebook (TFCC), which leverages scene priors to compress the semantic space of a scene into an 𝑁 length codebook. This approach effectively reduces redundant and noisy semantic features while preserving sufficient scene information and clear semantic boundaries.We introduce a low-dimensional semantic feature, symbolized as 𝑓 , into each 3D Gaussian, capitalizing on the redundancy of highdimensional semantics across the scene and dimensions to facilitate efficient rendering. To create a 2D semantic representation, we employ a volumetric rendering process similar to color rendering (Sec. 3.2) onto the low-dimensional semantic feature.EQUATIONf is the pixel-wise low-dimensional feature. We utilize an MLP as a feature decoder to obtain logits 𝑒, which are subsequently activated by the Softmax function to find the corresponding TFCC entry's index. This process acquires the feature 𝑣 in the high-dimensional semantic space for each f . Given that volumetric rendering is essentially a process of weighted averages, the 3D Gaussian feature 𝑓 and the rendered 2D pixel-wise feature f are fundamentally equivalent. The low-dimensional feature f and 𝑓 can both be recovered to semantic feature 𝑣 through the MLP decoder D and the TFCC T with 𝑁 entries,EQUATIONwhere 𝑒 = D ( f ) and 𝑒 ∈ R 𝑁 . Thus, both 2D and 3D features can be restrained to a compact and finite semantic space. Initially in the semantic field optimization, we focus on learning the TFCC from 𝐺𝑇 features. To enhance reconstruction efficiency, we adopt 𝑘-means clustering through 𝐺𝑇 feature maps 𝑉 for the codebook initialization. Also, we find some resemblance between the learning of TFCC and the contrastive pre-training from CLIP: Features in the codebook are to align with the 𝐺𝑇 features, and each 𝐺𝑇 feature, denoted as 𝑣 𝑔𝑡 , is assigned to one TFCC entry with the highest similarity. However, the assignment of a pixel feature to a particular entry is not predetermined, rather it pivots on similarity. Therefore, we devise a self-supervised loss function aimed at reducing the self-entropy of the clustering process.EQUATIONwhere 𝑝 𝑗 = Softmax cos 𝑣 𝑔𝑡 , T [ 𝑗] • 𝑞 and 𝑞 is the annealing temperature. To accelerate the process, we also optimize the entry with the highest similarity, introducing a loss similar to [49] ,EQUATIONEQUATIONThus, the loss in optimizing the TFCC isEQUATIONSubsequently, we undertake a joint optimization of the lowdimensional features f and the MLP decoder D. Ideally, the feature recovered from low-dimensional feature should closely correlate with the 𝐺𝑇 feature 𝑣 𝑔𝑡 . As a result, we impose a stronger constraint geared towards aligning the entries' logits of the low-dimensional features with the assigned 𝐺𝑇 entry 𝑑,EQUATION)Finally, to bolster the robustness of this procedure, we introduce an end-to-end regularization, directly optimizing the cosine similarity of 2D semantic feature and corresponding ground truth,EQUATIONThe comprehensive loss function designated for our semantic field reconstruction process is represented as L,EQUATIONThanks to the vision-language models like CLIP and APE, which align features well in image and text spaces. Our 3D Gaussian semantic field, once trained, supports open-vocabulary 3D queries with any text prompt. Most existing methods enable open-vocabulary queries by computing the cosine similarity between semantic and text features, defined as follows: cos(𝜃 ) =𝜙 𝑖𝑚𝑔 •𝜙 𝑡𝑒𝑥𝑡 ∥𝜙 𝑖𝑚𝑔 ∥ ∥𝜙 𝑡𝑒𝑥𝑡 ∥, where 𝜙 𝑖𝑚𝑔 and 𝜙 𝑡𝑒𝑥𝑡 represent the image and text features, respectively. After normalizing the features, the score can be simplified as 𝑆𝑐𝑜𝑟𝑒 = 𝜙 𝑖𝑚𝑔 • 𝜙 𝑡𝑒𝑥𝑡 . The higher the score, the greater the similarity between the two features. By manually setting an empirical threshold 𝜏, regions with score exceeding 𝜏 are retained, thus enabling open-vocabulary queries. The aforementioned process can be conceptualized as a hyperplane separating semantic features into two categories: features of interest and features not of interest, based on the queried text feature and 𝜏. The hyperplane is represented as follows:𝑊 𝑥 + b = 0. (13) Here 𝑊 denotes the queried text feature, 𝑥 represents semantic features and b is the bias derived from 𝜏. However, the empirical parameter 𝜏 is not universally applicable to all queries, often resulting in an inability to precisely locate target areas. Consequently, we propose the Optimizable Semantic-space Hyperplane (OSH). Utilizing a RES model, such as Grounded-SAM [45] , we obtain a 2D binary mask of the target area and optimize the hyperplane via one-shot logistic regression. This optimization ensures that the classification results of the hyperplane more closely align with the target area of the query.As shown on the right side of Figure 2 , From a specific camera pose, an RGB image and a feature map are obtained through the rgb and semantic feature rendering processes described in Sec. 3.5, respectively. For a text query 𝑡, the text encoder of APE generates a text embedding 𝜙 𝑡𝑒𝑥𝑡 , which is used as the initial weight of the hyperplane 𝑊 𝑥 + 𝑏 = 0. The Feature Map is classified by the hyperplane, resulting in the prediction of a binary mask 𝑚. The text query 𝑡 and the RGB image are processed by the RES Model to generate a binary mask m of the target area as the pseudo-label. This mask is subsequently used with 𝑚 in logistic regression to optimize 𝑊 and 𝑏. We fine-tune the OSH with the objective:EQUATIONwhere 𝑃 denotes all samples, 𝜎 (•) denotes Sigmoid function. Following the one-shot logistic regression, the optimized Semantic-space Hyperplane can be represented byEQUATION)Note that the parameters of the 3D Gaussians remain frozen during this process. The red lines in Figure 2 indicate operations that occur only during the initial query with a new text prompt. Subsequently, the OSH can delineate regions of interest in both 2D rendered views and 3D Gaussians. Specifically, for a semantic feature 𝐹 , derived from a 2D semantic feature map at pixel 𝑝 or from a 3D Gaussian 𝑔 , if 𝑊 ′ 𝐹 + 𝑏 ′ > 0, it indicates that 𝐹 is sufficiently close to the queried text, warranting retention of 𝑝 or 𝑔 in the query results set. 
implementation details: 
 Our method is implemented based on 3D Gaussian Splatting [20] .We modified the CUDA kernel to render semantic features on the 3D Gaussians, ensuring that the extended semantic features also support gradient backpropagation. Our model, based on a 3D Gaussian Scene reconstructed via vanilla 3D Gaussian Splatting, can be trained on a single A100-40G GPU in approximately 10 minutes. 
experiments 5.1 evaluation setup: 
 Datasets. To assess the effectiveness of our approach, we conduct experiments on two datasets: The Mip-NeRF360 dataset [1] and the Replica dataset [50] . Mip-NeRF360 is a high-quality real-world dataset that contains a number of objects with rich details. It is extensively used in 3D reconstruction. We selected four scenes (Room, Bonsai, Garden, and Kitchen), both indoors and outdoors, for our evaluations. Additionally, we designed an open-vocabulary semantic segmentation test set under these scenes. We manually annotated a few prominent objects in each scene, providing 2D masks and descriptive phrases, such as "sofa in dark green". Replica is a 3D synthetic dataset that features high-fidelity indoor scenes. Each scene comprises RGB images along with semantic annotations. We conducted experiments on four commonly used scenes from the Replica dataset: office0, office1, room0, and room1. For a given viewpoint image, our evaluation concentrates on assessing the effectiveness of single-query results within an open-vocabulary context rather than obtaining a similarity map for all vocabularies in a closed set and deciding mask regions based on similarity scores [27, 30, 60] . Therefore, in designing our experiments, we drew inspiration from the methodologies of refCOCO and refCOCOg [58] .For each semantic ground truth in the Replica test set, we cataloged the class names present and sequentially used these class names as text queries to quantitatively measure the performance metrics.Baseline Methods and Evaluation Metrics. To assess the accuracy of open-vocabulary querying results, we employ mean Intersection over Union (mIoU), mean Pixel Accuracy (mPA), and mean Precision (mP) as evaluation metrics. Additionally, to evaluate model performance metrics, we measure the training duration and the rendering time.We conduct a comparative evaluation of our approach in contrast with LangSplat [40] , Gaussian Grouping [57] , Feature 3DGS [60] ,and LERF [21] .Qualitative Results. We present the qualitative results produced by our method alongside comparisons with other approaches. Figure 3 offers a detailed showcase of the open-vocabulary query performance on the Mip-NeRF360 test data. It especially highlights the utilization of phrases that describe the appearance, texture, and relative positioning of different objects.LERF [21] generates imprecise and vague 3D features, which hinder the clear discernment of boundaries between the target region and others. Feature 3DGS [60] employs a 2D semantic segmentation model LSeg [24] as its feature extractor. However, like LSeg, it lacks proficiency in handling open-vocabulary queries. It frequently queries all objects related to the prompt. Gaussian Grouping [57] leverages the instance mask via SAM [22] to group 3D Gaussians into 3D instances devoid of semantic information. It uses Grounding DINO [32] to pinpoint regions of interest for enabling 3D openvocabulary queries. However, this approach leads to granularity issues, often identifying only a fraction of the queried object, such as the major part of "green grass" or the flower stem from the "flowerpot on the table". LangSplat [40] uses SAM to generate object segmentation masks and subsequently employs CLIP to encode these regions. However, this strategy results in CLIP encoding only object-level features, leading to an inadequate understanding of the correlations among objects within a scene. For instance, when querying "the tablemat next to the red gloves", it erroneously highlights the "red gloves" rather than the intended "tablemat".Our methodology effectively uses semantic redundancy to cluster features into a TFCC, enabling efficient encoding of diverse object features. Consequently, this approach precisely pinpoints objects such as the sofa, grass, and road while maintaining accurate boundaries. Our strategy further excels at discerning the intricate interrelationships among various objects within a scene. Unlike LangSplat, we encode entire images with the image encoder to integrate scene-level information into the semantic features. Additionally, we deploy dynamically optimize a semantic-space hyperplane, effectively filtering out unnecessary objects from the 3D Gaussians of Interest. For instance, in the cases of "flowerpot on the table" and "the tablemat next to the red gloves", we successfully segment the primary subjects of the phrase rather than the secondary objects. "flowerpot on the table" "green grass" "the tablemat next to the red gloves" "asphalt road" Moreover, Table 3 underscores the effectiveness of our approach. We detail the image pre-processing time for extracting semantic To discover each component's contribution to 3D open-vocabulary scene understanding in our proposed pipeline, a series of ablation experiments are conducted for the Mip-NeRF360 dataset [1] , using the same 2D semantic feature maps extracted from the APE [48] image encoder. We employ the approach of lifting compressed lowdimensionality semantic features into 3D Gaussians as our baseline. This is contrasted with results from models not utilizing the TFCC module, those not employing the OSH module, and the results from the complete model.As illstrated in Table 4 , OSH and TFCC are critical to the effectiveness of our approach; without them, there would be a significant deterioration in performance (-27% ~-12% mIoU). As shown in Figure 4 , the baseline model (top-right) struggles due to its scattered features, making it difficult for the model with the OSH module (bottom-middle) to identify a suitable hyperplane. In contrast, the model with TFCC (bottom-left) demonstrates more clustered features and distinct semantic boundaries.Our method can be applied to a variety of downstream tasks, with the most direct application being the editing of 3D scenes. As shown in the figure 5 , we use the text query "flowerpot on the table" to locate the 3D Gaussians of interest. Our method enables the highlighting of target areas, localized deletion, and movement. Furthermore, by integrating with Stable-Diffusion [46] , We can employ the Score Distillation Sampling (SDS) [38] loss function to achieve high-quality 3D generation tasks in specific areas. The query text is used to locate the 3D Gaussians of interest (GOI). "A beautiful vase" is used as the prompt for the 3D inpainting process after locating the GOI. 
conclusion: 
 In this paper, we introduce GOI, a method for reconstructing 3D semantic fields, capable of delivering precise results in 3D openvocabulary querying. By leveraging the Trainable Feature Clustering Codebook, GOI effectively compresses high-dimensional semantic features and integrates these lower-dimensional features into 3DGS, significantly reducing memory and rendering costs while preserving distinct semantic feature boundaries. Moreover, moving away from traditional methods reliant on fixed empirical thresholds, our approach employs an Optimizable Semantic-space Hyperplane for feature selection, thereby enhancing querying accuracy. Through extensive experiments, GOI has demonstrated improved performance over existing methods, underscoring its potential for downstream tasks, such as localized scene editing. 
ground truth: 
 Ours LangSplat Feature3DGS LERF "Bowl on the table" Text EmbeddingFigure 1 : We propose GOI, an innovative approach to 3D open-vocabulary scene understanding based on 3D Gaussian Splatting [20] . In the top row, we emphasize our key contribution: the Optimizable Semantic-space Hyperplane (OSH). Instead of relying on a manually set, fixed empirical threshold for relative feature selection, which frequently lacks universal accuracy, OSH is fine-tuned for each query to accurately locate target regions in response to natural language prompts. The bottom row showcases our superior performance in open-vocabulary querying compared to other approaches.subsequently embedded in 3DGS. During the open-vocabulary querying process, we adopt a distinct approach compared to existing methods, which depend on a manually set fixed empirical threshold to select regions based on their semantic feature distance to the query text embedding. This traditional approach often lacks universal accuracy, leading to challenges in precisely identifying specific target areas. Instead, our method treats the feature selection process as a hyperplane division within the feature space, retaining only those features that are highly relevant to the query. We leverage off-the-shelf 2D Referring Expression Segmentation (RES) models to fine-tune the semantic-space hyperplane, enabling a more precise distinction between target regions and others. This fine-tuning substantially improves the accuracy of open-vocabulary queries, ensuring the precise localization of pertinent 3D Gaussians.Extensive experiments demonstrate GOI's superiority over previous state-of-the-art methods. The dataset, model, and code are available at https://quyans.github.io/GOI-Hyperplane/. 
 
 Appendix:  


Decision: The reviewers all recommend accepting this paper because of the design novelty and sufficient experimental validation. AC also agrees with the reviewers and recommends an acceptance of the submission.

Reviews: review: paper_summary: paper_summary: This paper focuses on 3D open-vocabulary semantic perception and introduces GOI, an innovative framework based on 3D Gaussian Splatting for accurate semantic analysis. GOI integrates semantic features from 2D vision-language foundation models into 3D Gaussian Splatting (3DGS), and it identifies 3D Gaussians of Interest using an Optimizable Semantic-space Hyperplane. This manuscript also includes an efficient compression method that utilizes scene priors to condense noisy, high-dimensional semantic features into compact, low-dimensional vectors. These vectors are then embedded in 3DGS. Additionally, the framework enhances feature management through the Trainable Feature Clustering Codebook (TFCC), which organizes and optimizes the feature space for improved processing efficiency and accuracy.
strength_weakness: strength_weakness: This paper presents a method that achieves comparable results on the benchmark Mip-NeRF360 Replica. It conceptualizes the feature selection process as a division of the feature space via hyperplanes, retaining only those features that are highly relevant to the query. This approach overcomes the limitations of traditional methods, which often lack universal accuracy and face challenges in precisely identifying specific target areas.
limitations: limitations: I am not an expert in this area, so I will refer to the opinions of other reviewers to determine my final rating. 
I have the following questions: 
It would be beneficial to see results on ScanNet200. 
Could you also provide outcomes when directly using CLIP features?
paper_summary: paper_summary: The paper proposes a feature distillation method using a learnable codebook inside 3D gaussian splatting for open vocabulary querying in 3D scenes. The paper also introduces Optimizable Semantic-space Hyperplane (OSH), which do away with the need for a manual threshold  in pixel localization. OSH can be finetuned for each text query and is more generalizable .
strength_weakness: strength_weakness: The learnable codebook usage inside gaussian spplatting for avoiding noisy features is a novel component. Optimizable Semantic-space Hyperplane (OSH) proposed in the work do away with the need for a manual threshold  in pixel localization.
limitations: limitations: Distilling knowledge from vision-language Models and constructing  semantic  fields inside  3D Gaussians has been explored in a number of previous works.  The paper mentions one of its contributions as having a lifting approach with  pixel aligned semantic fields . Section 3.5 in the paper discusses how 3D Gaussian Semantic field is obtained by volumetric rendering process similar to color rendering.  However this part is fundamentally similar to the 3D Gaussian semantic field creation discussed in Feature 3DGS [1]. 

The major difference from [1] is that the authors have used a Trainable Feature Clustering Codebook citing  ‘redundant and noisy high dimensional semantic features’ and ‘computational needs’ . How is the size, N,  of codebook fixed ?  What is the impact of varying ‘N’ ?  Also there is no ablation study presented to showcase the impact on accuracy and computation if the codebook is not used in the existing dataset (Replica).  

Additionally the impact of OSH module is showcased only on Mip-NeRF360 annotations which I assume the authors have contributed. Ablation studies of OSH and TFCC on Replica  is also needed to establish the proposed method’s usefulness.

Table 3 showcase the training time advantage of the proposed method. The major difference in timing is attributed to the fact the paper starting with a pre-trained model of 3D gaussians. It is not clear if this can be adopted by other methods as well or is there any special design considerations that is specific to this paper that allows such an initialization and subsequent saving of training time.

[1] Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields

Minor Comments:
Line 283 -  we utilize an frozen image encoder  we utilize a frozen image encoder
paper_summary: paper_summary: The paper introduces a novel framework for 3D open-vocabulary scene understanding. The method leverages semantic features extracted from 2D vision-language models, integrating them into a 3D Gaussian splatting architecture. This integration facilitates the identification and localization of 3D Gaussians of Interest (GOI) based on natural language prompts. The paper claims substantial improvements over existing methods, particularly in terms of accuracy and the ability to handle open-vocabulary queries.
strength_weakness: strength_weakness: + Innovative Approach: The combination of 3D Gaussian splatting with semantic features from 2D models is innovative. This approach allows for more precise localization and understanding of 3D scenes from natural language inputs.  
&nbsp;
+ Optimizable Semantic-space Hyperplane (OSH): The introduction of OSH as a method to fine-tune the feature selection process dynamically in response to natural language queries is a significant advancement. It addresses the common issue of fixed thresholds that cannot adapt to various queries.  
&nbsp;
+ Extensive Evaluation: The paper presents a thorough experimental setup, including comparisons with several state-of-the-art methods. The use of multiple datasets and extensive metrics (mIoU, mPA, mP) to evaluate their method adds credibility to their claims.  
&nbsp;
+ Practical Applications: The paper does well to discuss potential practical applications, particularly in augmented reality and robotic navigation, which are areas of significant interest within the community.
limitations: limitations: - Complexity and Computation Requirements: The method appears to be computationally intensive, involving multiple components such as feature extraction, 3D Gaussian splatting, and the optimization of a semantic-space hyperplane. The practicality of deploying such a system in real-time applications may be limited.  
&nbsp;
- Dependency on Training Data: The performance of the system heavily relies on the quality and diversity of the annotated 3D scene datasets. The paper does not address how the system would perform with less curated or noisy data.  
&nbsp;
- Generalization Concerns: While the results are promising, the generalization of the model to a wider array of scenes and languages (beyond the tested datasets and English language prompts) remains unclear.  
&nbsp;
- Hyperparameter Sensitivity: The method involves several layers of optimization and feature selection, which may be sensitive to hyperparameter settings. The paper does not discuss the sensitivity analysis or the robustness of the method against variations in hyperparameters.