Gaga: Group Any Gaussians via 3D-aware Memory Bank

## Abstract

Abstract We introduce Gaga , a framework that reconstructs and segments open-world 3D scenes by leveraging inconsistent 2D masks predicted by zero-shot segmentation models.
Contrasted to prior 3D scene segmentation approaches that heavily rely on video object tracking, Gaga utilizes spatial information and effectively associates object masks across diverse camera poses.
By eliminating the assumption of continuous view changes in training images, Gaga demonstrates robustness to variations in camera poses, particularly beneficial for sparsely sampled images, ensuring precise mask label consistency.
Furthermore, Gaga accommodates 2D segmentation masks from diverse sources and demonstrates robust performance with different open-world zero-shot segmentation models, enhancing its versatility.
Extensive qualitative and quantitative evaluations demonstrate that Gaga performs favorably against state-of-the-art methods, emphasizing its potential for real-world applications such as scene understanding and manipulation.

###### Keywords:

## 1 Introduction

Effective open-world 3D segmentation is essential for scene understanding and manipulation. Despite notable advancements in 2D segmentation techniques, exemplified by Segment Anything (SAM) [] and EntitySeg [], extending these methodologies to the realm of 3D encounters the challenge of ensuring consistent mask label assignment across multi-view images.
Specifically, masks of the same object across different views may have different mask label IDs, as the multi-view images are processed by the 2D segmentation model individually.
Naively lifting these inconsistent 2D masks to 3D introduces ambiguity and leads to inferior results in 3D scene segmentation. Hence, we argue that it is crucial to assign each mask a multi-view consistent universal mask ID before lifting them to 3D. We refer to this task as mask association.

Figure: Figure 2: Comparison of rendered segmentation. Previous methods [, ] adopt an off-the-shelf video object tracker for mask association. Results on the ScanNet dataset [] show that they frequently misidentify objects, especially when similar objects are present in the scene (*e.g*., the leather sofas), and struggle to handle significant changes in camera perspective. In contrast, Gaga integrates 3D information to precisely locate objects and associate 2D masks, leading to multi-view consistent class-agnostic segmentation and precise instance segmentation rendering. We adopt the ground truth panoptic segmentation of the ScanNet dataset for comparison as it is visually the same as class-agnostic segmentation.
Refer to caption: x2.png

Prior research efforts [, ] built upon the recent advance in 3D reconstruction, Gaussian Splatting [], attempt to solve this task by treating multi-view image datasets as video sequences and adopting an off-the-shelf video object tracking method [].
Nevertheless, this design relies on the assumption of minimal view changes between multi-view images, a condition that may not consistently hold in real-world 3D scenes. Consequently, these approaches struggle with similar objects or occluded objects that intermittently disappear and reappear in the sequence, as shown in Fig. 2.

As such, we analyze the fundamental disparity between the 3D mask association and video object tracking task: the utilization of inherent 3D information. Specifically, masks of the same object across different views shall correspond to the same group of 3D Gaussians. Hence, we can assign two masks from different views with the same universal mask ID if there is a large overlap between the two groups of 3D Gaussians that are splatted to them.

Based on this intuition, we propose Gaga, a framework that groups any 3D Gaussians and renders consistent 3D segmentation across different views.
Given a collection of posed RGB images, we first employ Gaussian Splatting to reconstruct a 3D scene and extract 2D masks using an open-world segmentation method. Subsequently, we iteratively build a 3D-aware memory bank that collects and stores Gaussians grouped by category. Specifically, for each input view, we project each 2D mask into 3D space using camera parameters and search the memory bank for the category with the largest overlap with the deprojected mask. Depending on the degree of overlap, we either assign the mask to an existing category or create a new one. Finally, following the mask association process described above, we leverage the consistent 2D masks to learn a feature on each Gaussian for rendering segmentation.

Our approach, Gaga, is capable of 1) synthesizing novel view images of RGB and segmentation with inherent 3D consistency; 2) grouping 3D Gaussians based on their 2D segmentation masks and providing accurate 3D instance segmentation for scene manipulation; 3) accommodating any 2D segmentation methods without additional mask pre-processing. Our contributions are summarized as follows:

- •
We propose a framework that reconstructs and segments 3D scenes using inconsistent 2D masks generated by open-world segmentation models.
- •
To resolve the inconsistency of 2D masks across views, we design a 3D-aware memory bank that collects Gaussians of the same semantic group. This memory bank is then employed to align 2D masks across diverse views.
- •
We show that the proposed method can effectively leverage any 2D segmentation masks, making it easily applicable for synthesizing novel view images and segmentation masks.
- •
We conduct comprehensive experiments on diverse datasets and challenging scenarios, including sparse input views, to demonstrate the effectiveness of the proposed method both qualitatively and quantitatively.

## 2 Related Work

Segment and Track Anything in 2D.
Segment Anything (SAM) [] and EntitySeg [] demonstrate the effectiveness of large-scale training in image segmentation, thus establishing a pivotal foundation for open-world segmentation methods. Subsequent studies [, , ] further extend the applicability of SAM to video data by leveraging video object segmentation algorithms to propagate the masks generated by SAM. Conversely, acquiring data for training their 3D counterparts poses a challenge, given that existing 3D datasets with annotated segmentation [, ] primarily focus on indoor scenarios.

NeRF-based 3D Segmentation.
Neural Radiance Fields (NeRFs) [] model scenes as continuous volumetric functions, learned through neural networks that map 3D coordinates to scene radiance. This approach facilitates the capture of intricate geometric details and the generation of photorealistic renderings, offers novel view synthesis capabilities.

Semantic-NeRF [] initiates the incorporation of semantic information into NeRFs and enables the generation of semantic masks for novel views. Note that lifting semantic segmentation masks to 3D does not face the challenge of ambiguous mask IDs across views. Building upon it, subsequent researches expand the scope by introducing instance modeling and matching instance masks relying on 3D bounding boxes [, ], solving cost-based linear assignment during training [, ] or directly training instance-specific MLPs []. However, most of these methods are developed based on ground truth segmentation and tailored for scene modeling within specific domains. They often entail high computational costs and lack substantial evidence of their performance in open-world scenarios.

Leveraging SAM’s open-world segmentation capability, SA3D [] endeavors to recover a 3D consistent mask by tracing 2D masks across adjacent views with user guidance. Similarly, Chen et al. [] distill SAM encoder features into 3D and query the decoder. In contrast, Gaga achieves multi-view consistency without user intervention, and offers segmentation for all objects rather than an instance. Concurrent work GARField [] densely samples SAM masks and trains a scale-conditioned affinity field supervised on the scale of each mask deprojected to 3D. Whereas Gaga does not require the preprocessing process to obtain densely sample masks of different scales. Meanwhile, [] groups Gaussians as clusters and multiple 3D instances may have the same group ID, which differs from the 3D segmentation task.

Gaussian-based 3D Segmentation.
As an alternative to NeRF and its variants [, , ], Gaussian Splatting [] has recently emerged as a powerful approach to reconstruct 3D scenes via real-time radiance field rendering. By representing the scene as 3D Gaussians, Gaussian Splatting achieves photorealistic novel view synthesis with high reconstruction quality and efficiency. Additionally, manipulating 3D Gaussians for scene editing is more straightforward than NeRF’s representation.

SAGA [] renders a 2D SAM feature map and uses a SAM guidance loss to learn 3D segmentation from the ambiguous 2D masks. Similar to [], this method requires user input and only provides segmentation for one instance at a time. Feature 3DGS [] distills LSeg [] and SAM features to 3D Gaussians and decodes rendered features to obtain segmentation. However, it fails to provide consistent segmentation across views. Gaussian Grouping [] and CoSSegGaussians [] use 2D segmentation masks as pseudo labels and use a video object tracker [] to associate masks across different views before lifting them to 3D. However, in scenarios with significant changes in camera poses between frames, such approaches struggle to maintain accuracy.

## 3 Proposed Method

### 3.1 Preliminaries

Gaussian Splatting.
Recently, Gaussian Splatting [] has significantly advanced the 3D representation field by combining the benefits of implicit and explicit 3D representations. Specifically, a 3D scene is parameterized as a set of 3D Gaussians $\{G_{i}\}$. Each Gaussian $G_{i}=\{p_{i},s_{i},q_{i},\alpha_{i},c_{i}\}$ is defined by its position $p_{i}=\{x,y,z\}\in\mathbb{R}^{3}$, scale $s_{i}\in\mathbb{R}^{3}$, orientation $q\in\mathbb{R}^{4}$, opacity $\alpha\in\mathbb{R}$ and color features $c$ encoded by spherical harmonics (SH) coefficients.

For image rendering, Gaussian Splatting employs the splatting rendering pipeline, wherein 3D Gaussians are projected onto the 2D image space using the world-to-frame transformation matrix corresponding to each camera pose. Gaussians projected to the same coordinates $(x,y)$ are blended in depth order and weighted by their opacities to produce the color $c_{x,y}$ of each pixel:

$$ $c_{x,y}=\sum_{i\in N}c_{i}\alpha_{i}\prod_{j=1}^{i-1}(1-\alpha_{j}).$ (1) $$

Identity Encoding.
Identity encoding $e_{i}$ [] aims to assign a universal label to each 3D Gaussian for segmentation rendering. It is a 16-dimensional feature attached to each Gaussian, which is subsequently decoded to a segmentation mask ID $m_{x,y}$ for each pixel $(x,y)$ through a classifier $L$, *i.e*., a combination of a linear and a SoftMax layer:

$$ $m_{x,y}=\arg\max(L(\sum_{i\in N}e_{i}\mathbf{\alpha}_{i}\prod_{j=1}^{i-1}(1- \mathbf{\alpha}_{j}))).$ (2) $$

The resulting mask IDs are supervised using 2D segmentation masks.

### 3.2 Group Any Gaussians via 3D-aware Memory Bank

Figure: Figure 3: Overview of Gaga. Gaga reconstructs 3D scenes with Gaussian Splatting and adopts an open-world model to generate 2D segmentation masks. To eliminate the 2D mask label inconsistency, we design the 3D-aware mask association process, where a 3D-aware memory bank is employed to assign a consistent group ID across different views to each 2D mask based on the 3D Gaussians projected to that mask (Sec. 3). Specifically, we find the corresponding Gaussians projected to each 2D mask and assign the mask with the group ID in the memory bank with the maximum overlapped Gaussians (Eq. 3). After the 3D-aware mask association process, we use masks with multi-view consistent group IDs as pseudo labels to train an identity encoding on each 3D Gaussian for segmentation rendering.
Refer to caption: x3.png

Given a set of posed images, we aim to reconstruct a 3D scene with semantic labels for segmentation rendering. To this end, we first leverage Gaussian Splatting for scene reconstruction. We then employ an open-world 2D segmentation method such as SAM [] or EntitySeg [] to predict class-agnostic segmentation for each input image.
However, because the segmentation model processes each input image independently, the resulting masks are not naturally multi-view consistent. To resolve this issue, [, ] assume that nearby input views are similar and apply a video object tracker to associate inconsistent 2D masks of different views. Yet, this assumption may not hold for all 3D scenes, especially when the input views are sparse.

Gaga is inspired by the fundamental disparity between the task of mask association across multiple views and tracking objects in a video: the incorporation of 3D information. To reliably generate consistent masks across different views, we propose a method that leverages 3D information without relying on any assumptions about the input images. Our key insight is that masks belonging to the same instance in different views shall correspond to the same Gaussian group in the 3D space. Consequently, these Gaussians should be grouped together and assigned an identical group ID.

Figure: Figure 4: Illustrations of finding corresponding Gaussians. (a) Motivation to choose front $x\%$ of Gaussians. We select $x\%$ of Gaussians which are closest to the camera frame because many Gaussians splatted to mask in view 1 represent objects from behind, as shown in view 2. (b) The significance of mask partition. We color the Gaussians in the 3D-aware memory bank based on their groups, displayed as "3D Gaussian Groups" in columns 1 and 3. When images aren’t partitioned (row 1), the front $x\%$ of Gaussians concentrate in a confined area, failing to accurately represent the shape of the mask, resulting in mismatched masks (columns 2, 4).
Refer to caption: x4.png

Corresponding Gaussians of a Mask. Based on this intuition, we first associate each 2D segmentation mask with its corresponding 3D Gaussians.
Specifically, we splat all 3D Gaussians onto the camera frame given the camera pose of each input image.
Subsequently, for each mask within the image, we identify which 3D Gaussians are projected within that mask.
Those Gaussians should be identified as representatives of the mask in 3D and can be used as guidance for us to associate masks from different views.

Notably, segmentation masks typically describe the shape of foreground objects under the current camera pose.
However, as Fig. 4 (a) shows, a significant proportion of Gaussians do not contribute to the pixels in the 2D segmentation mask, as they represent objects situated behind.
To address this, we select the front $x\%$ percentage of 3D Gaussians that are closest to the camera frame as the corresponding Gaussians of the mask.
Here, $x$ serves as a hyperparameter that can be adjusted based on the nature of the current 3D scene.
As shown in Fig. 4 (b) row 1, selecting corresponding Gaussians based on the entire mask will inaccurately represent the shape of the mask for masks of large regions, and fail to associate masks across different camera poses.
To resolve this issue, we propose a strategy wherein we partition an input image into patches and calculate corresponding Gaussians within each patch. Specifically, we begin by dividing the image into 32$\times$32 patches. Subsequently, we identify the collection of top $x\%$ percentage of 3D Gaussians that are closest to the camera frame within each patch to be the corresponding Gaussians of mask $m$, denoted as $\mathcal{G}(m)$. As demonstrated in Fig. 4 (b) row 2, this simple strategy effectively improves the consistency of associated masks across different views.

3D-aware Memory Bank.
Next, to collect and categorize 3D Gaussians into groups and use them to associate masks across different views, we introduce a 3D-aware Memory Bank (see Fig. 3).
Given a set of images, we initialize the 3D-aware Memory Bank by storing the corresponding Gaussians of each mask in the first image into an individual group and label the mask with a group ID the same as its mask label.
For each 2D mask of the subsequent image, we first determine its corresponding Gaussians as outlined above. We then either assign these Gaussians to an existing group within the memory bank or establish a new one if they do not share similarities with existing groups in the memory bank. The details of this assignment process are elaborated in the following.

Group ID Assignment via Gaussian Overlap.
To assign each mask a group ID, we aim to find if the current mask has a significant similarity with any group in the memory bank.
Here, we define the similarity between two sets of 3D Gaussians based on their shared Gaussians ratio. Specifically, given the 3D Gaussians corresponding to a 2D mask $m$ (denoted as $\mathcal{G}(m)$ as described above) and the Gaussians of group $i$ (denoted as $\mathcal{G}_{i}$) in the memory bank, we identity their shared Gaussians (*i.e*., Gaussians of the same indices) as $\mathcal{G}(m)\cap\mathcal{G}_{i}$, we then compute the overlap as the ratio of the number of shared Gaussians to the number of all corresponding Gaussians of mask $m$:

$$ $Overlap(m,i)=\frac{\#(\mathcal{G}(m)\cap\mathcal{G}_{i})}{\#\mathcal{G}(m)}.$ (3) $$

If group $i$ has the highest overlap with mask $m$ among all groups in the memory bank, and this overlap value is above a threshold, we assign the group ID of mask $m$ as $i$ and add the non-overlapped Gaussians in the $i_{th}$ group.

$$ $\mathcal{G}_{i}=\mathcal{G}_{i}\cup\mathcal{G}(m).$ (4) $$

We establish a new group ID $j$ if none of the existing groups contains an overlap with mask $m$ above the overlap threshold.
We add $\mathcal{G}(m)$ into this new group in the 3D-aware memory bank and assign mask $m$ with the new group ID $j$.
Note that we ensure each Gaussian will only be added to one group in the memory bank by recording the indices of all Gaussians that already exist in the memory bank.

### 3.3 3D Segmentation Rendering and Downstream Application.

After the group ID assignment process, masks projected by the same group of Gaussians are supposed to have the same group ID across different views.
Similar to [], we use those associated masks as pseudo labels and lift them to 3D by training the identity encoding. As we already obtain pre-trained Gaussians, we fix the other properties (e.g., location, opacity, etc.) of Gaussians for efficiency.

Our segmentation-aware 3D Gaussians can be readily used for various downstream applications. For instance, we can render segmentation masks of novel views that have consistent mask color for the same object across different camera poses.
Gaussians can also be selected by their identity encoding for scene editing tasks including removal, color-changing, position translation, *etc*., as demonstrated in Sec. 4.6.

## 4 Experiments

### 4.1 Experimental Setup

Datasets.
We experiment with various datasets across diverse scenarios to demonstrate the performance of Gaga. For quantitative comparison, we use a scene understanding dataset LERF-Mask [], along with two indoor scene datasets: Replica [] and ScanNet [].
LERF-Mask is based on the LERF dataset [] and annotated with tasks and ground truth by the author of [].
It contains 3 scenes: figurines, ramen, and teatime. For each scene, 6-10 objects are selected as text queries, and Grounding DINO [] is utilized to select the mask ID from the rendered segmentation.
Indoor datasets Replica and ScanNet are commonly used to evaluate 3D scene understanding methods, as ground truth semantic segmentation and panoptic segmentation are provided. We employ 8 scenes from Replica rendered by the author of [], each consisting of 180 training images and an equal number for testing.
We utilize 7 scenes in ScanNet, processed similarly to []. Each scene contains over 300 training images and around 100 testing images. Note that all annotated segmentation masks are unavailable during training and only accessible during evaluation as ground truth.
We present visual comparison results on the commonly used scene reconstruction dataset, MipNeRF 360 []. Additionally, we showcase the robustness of Gaga against variations in training image quantity by sparsely sampling Replica and MipNeRF 360 datasets. For all experiments, the quantitative and qualitative results are conducted on the test set, *i.e*., novel view synthesis results.

Evaluation Metrics.
mIoU and boundary IoU (mBIoU) are used for evaluation on the LERF-Mask dataset.
For Replica and ScanNet datasets, we evaluate using ground truth panoptic segmentation, disregarding class information.
To handle differences between predicted and ground truth mask labels, we calculate the best linear assignment based on IoU.
Moreover, with IoU = 0.5 as the criterion, we report precision and recall to further evaluate the accuracy of predicted masks.

Implementation Details.
We use SAM [] the default version and EntitySeg [] with the Hornet-L backbone
to obtain open-world 2D segmentation.
We preprocess the generated raw masks following the method outlined in [], prioritizing those with higher confidence scores by ranking them accordingly.
Masks with confidence scores below 0.5 are discarded.
For all experiments, we train the vanilla Gaussian Splatting for 30K iterations and train the identity encoding for 10K iterations with all other parameters frozen.
We choose the front 20% 3D Gaussians that are closest to the camera frames as the corresponding Gaussians of a mask.
We set the overlap threshold for declaring a new group ID as 0.1.
For fair comparisons, we train Gaussian Grouping [] for 40K iterations, with all parameters for training 3D Gaussians with their identity encoding remaining the same as the default setting in [] and [].
We use the official transcript of [] to obtain camera poses and initial point clouds.

### 4.2 Open-vocabulary 3D Query on LERF-Mask Dataset

**Table 1: Quantitative results for open-vocabulary 3D query tasks on LERF-Mask dataset. Gaga outperforms previous approaches, showcasing favorable performance in terms of mIoU and BIoU with both segmentation models. * denotes the results are reported in [].**
| Model | 2D Seg. Method | mIoU(%) | mBIoU(%) |
| --- | --- | --- | --- |
| LERF []* | / | 37.17 | 29.30 |
| Gaussian Grouping [] | EntitySeg | 54.10 | 50.90 |
| Gaga(Ours) | 62.44 | 60.28 |  |
| Gaussian Grouping[]* | SAM | 72.79 | 67.58 |
| Gaga(Ours) | 74.71 | 72.19 |  |

Figure: Figure 5: Visual comparison on LERF-Mask dataset. Our rendered segmentation exhibits fewer artifacts and delivers more accurate instance segmentation results. Visualization of the rendered feature field via PCA demonstrates that Gaga obtains superior identity encoding features.
Refer to caption: x5.png

We experiment on the LERF-Mask dataset following the evaluation process in []. Tab. 1 illustrates that Gaga achieves superior results in mIoU and mBIoU compared to previous methods [, ]. especially when utilizing EntitySeg as the 2D segmentation method, resulting in approximately a 9% advantage for both mIoU and mBIoU.
In Fig. 5, we present visualizations of the rendered segmentation, rendered feature field, and 3D query results. Gaga yields more precise segmentation with fewer artifacts and empty regions, indicating that our 3D-aware mask association method provides multi-view consistent 2D segmentation with less label ambiguity as pseudo labels for training the identity encoding. The visualized identity encoding feature field of Gaga exhibits much cleaner results, further supporting this observation. Hence, Gaga can provide a more precise instance segmentation for queried 3D objects.

**Table 2: Quantitative results on Replica and ScanNet datasets. Gaga performs well with both 2D segmentation methods on two datasets. Notice that the performance of Gaussian Grouping varies significantly with different 2D segmentation methods, whereas Gaga consistently delivers stable performance.**
|  | 2D Seg. | Replica | ScanNet |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Model | Method | IoU(%) | Precision(%) | Recall(%) | IoU(%) | Precision(%) | Recall(%) |
| Gaussian Grouping [] | EntitySeg | 35.90 | 14.07 | 31.57 | 39.54 | 6.88 | 36.56 |
| Gaga(Ours) | 41.08 | 63.06 | 46.14 | 42.56 | 33.89 | 47.63 |  |
| Gaussian Grouping [] | SAM | 21.76 | 25.00 | 19.72 | 34.24 | 18.70 | 32.61 |
| Gaga(Ours) | 46.50 | 41.52 | 52.50 | 44.87 | 18.61 | 45.94 |  |

Figure: Figure 6: Qualitative results on Replica and ScanNet datasets. Gaga provides high-quality segmentation masks that are more similar to the ground truth. Gaussian Grouping often covers the same object with different masks (rows 1, 4, 6), creates large empty regions (rows 1-4), and misidentifies similar instances (rows 5, 6).
Refer to caption: x6.png

### 4.3 3D Segmentation on Replica and ScanNet Datasets

Tab. 2 presents the quantitative comparison results on the Replica and ScanNet datasets. Gaga exhibits better performance on both datasets regardless of the 2D segmentation model utilized, showcasing its stability across different datasets and models.
Qualitative results are shown in Fig. 6. Rows 1-3 depict the visualizations from the Replica dataset, while rows 4-6 showcase results from the ScanNet dataset.
Gaussian Grouping [] frequently assigns different mask IDs to the same object, resulting in inconsistent mask colors and empty regions. Rows 5 and 6 illustrate that Gaussian Grouping struggles to distinguish similar objects, whereas our proposed Gaga accurately identifies each object by leveraging 3D information.

### 4.4 3D Segmentation with Limited Data on Replica Dataset

**Table 3: Quantitative results on Replica dataset with limited training data. Gaga consistently outperforms Gaussian Grouping with both 2D segmentation methods. The percentage of IoU drop indicates that Gaga exhibits greater robustness against reductions in training data.**
| 2D Seg. Method | EntitySeg | SAM |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Model | Training Data | IoU(%) $\uparrow$ | IoU Drop(%) $\downarrow$ | IoU(%) $\uparrow$ | IoU Drop(%) $\downarrow$ |
| Gaussian Grouping [] | 30% | 28.42 | 20.85 | 17.02 | 21.78 |
| Gaga(Ours) | 37.98 | 7.57 | 41.79 | 10.11 |  |
| Gaussian Grouping [] | 20% | 24.56 | 31.35 | 16.02 | 26.38 |
| Gaga(Ours) | 37.25 | 9.33 | 40.27 | 13.40 |  |
| Gaussian Grouping [] | 10% | 20.62 | 42.56 | 13.97 | 35.78 |
| Gaga(Ours) | 31.93 | 22.27 | 35.61 | 23.40 |  |
| Gaussian Grouping [] | 5% | 10.00 | 72.15 | 6.77 | 68.87 |
| Gaga(Ours) | 20.59 | 49.88 | 22.79 | 50.98 |  |

Figure: Figure 7: Qualitative results on Replica dataset with limited training data. The visualizations depict samples when using only 5% of the training data (9 training images). Even with limited data, Gaga consistently produces high-quality segmentation. In contrast, Gaussian Grouping struggles to track objects accurately and leaves significant empty regions.
Refer to caption: x7.png

To demonstrate the robustness of Gaga against changes in training image quantity, we sparsely sample the Replica training set with ratios of 0.3, 0.2, 0.1, and 0.05.
As depicted in Tab. 3, Gaga consistently exhibits superior performance in terms of IoU, with approximately a 10% advantage using EntitySeg and a 20% advantage using SAM.
Remarkably, when utilizing SAM, Gaga surpasses fully trained Gaussian Grouping with just 5% of the training data (22.79% *vs*. 21.76%).
We also compute the IoU drop compared to using all training images as follows:

$$ $IoU\ Drop(x)=\frac{IoU(100\%)-IoU(x\%)}{IoU(100\%)},$ (5) $$

where $IoU(x\%)$ denotes the IoU achieved when $x\%$ of the training data is used. Compared to Gaussian Grouping, Gaga exhibits less sensitivity to decreases in the number of training images, as evidenced by smaller values in IoU drop. Visualization results are shown in Fig. 7. With just 5% of the training data, Gaga can still deliver accurate segmentation masks, whereas Gaussian Grouping fails to provide masks for a significant portion of objects due to inaccurate tracking.

### 4.5 3D Segmentation on MipNeRF 360 Dataset

Figure: Figure 8: Qualitative results on the MipNeRF 360 dataset. Rows 1-3 show that Gaga provides superior segmentation with finer details (rows 1, 2), fewer artifacts (row 1), and more consistent instance segmentation across different views (bicycle in row 3). Rows 4-6 present results with sparsely sampled data. While Gaussian Grouping can not accurately track objects (bicycle in row 4, bulldozer in row 5, stump and flowers in row 6), Gaga consistently provides precise segmentation.
Refer to caption: x8.png

We further showcase the performance of Gaga on diverse scenarios, *i.e*., the MipNeRF 360 dataset, with SAM as 2D segmentation method. We provide visualization comparison with Gaussian Grouping [] in Fig. 8, rows 1-3. We display two images for each scene to assess the consistency across different views. Gaga offers more detailed segmentation, while segmentation masks generated by Gaussian Grouping exhibit severe artifacts. Additionally, inconsistency across two views exists in the rendering results of Gaussian Grouping, as shown in row 3.

We sparsely sample the MipNeRF 360 dataset with sample step = 3 and visualize the results in Fig. 8, rows 4-6. Similar to results in Fig. 7, Gaussian Grouping can not accurately track objects with limited training data, resulting in empty regions in rendered segmentation. Conversely, Gaga maintains accurate segmentation, even for those tiny leaves in the "stump" scene in row 6.

Figure: Figure 9: Scene manipulation results on MipNeRF 360 and Replica datasets. Gaga accurately identifies the cushion part of the footstool, whereas Gaussian Grouping colors it entirely. For object removal and translation tasks, Gaga generates more precise 3D entities with fewer artifacts, resulting in better visual performance.
Refer to caption: x9.png

### 4.6 Application: Scene Manipulation

Gaga achieves high-quality and multi-view consistent 3D segmentation, beneficial for tasks like scene manipulation, as we can accurately segment the Gaussians of a 3D object and edit their properties. Using a pre-trained 3D Gaussian model with identity encoding, we employ the classifier trained with identity encoding to predict mask labels for each 3D Gaussian. Subsequently, we select 3D Gaussians sharing the same mask label as the target object and edit their properties for tasks like object coloring, removal, and position translation.

We demonstrate the effectiveness of this application on MipNeRF 360 and Replica datasets (see Fig. 9). In the "room" scene of MipNeRF 360, we change the color of the cushion in the footstool and remove the stuffed animal on the armchair. Gaga accurately identifies 3D Gaussians representing the cushion, while Gaussian Grouping [] fails, colors the entire footstool maroon along with part of the sofa and some floating Gaussians. Gaga also effectively groups and removes the entire part of the stuffed animal on the sofa with minimal artifacts, whereas Gaussian Grouping leaves many floating Gaussians. Similar results are observed in the experiment involving the position shifting of a chair in the "office 3" scene of the Replica dataset. Notice that Gaussian Grouping also creates artifacts at positions far away from the target object.

**Table 4: Ablation study on different mask association methods. Our mask association method with 3D-aware memory bank surpasses the previous video tracker baseline on both IoU, Precision, and Recall.**
| Baseline | IoU (%) | Precision (%) | Recall (%) |
| --- | --- | --- | --- |
| w/o. Mask Association | 8.81 | 3.19 | 2.16 |
| Video Object Tracker [] | 21.76 | 25.00 | 19.72 |
| Memory Bank (w. All Gaussians) (Ours) | 42.26 | 40.19 | 45.95 |
| Memory Bank (w/o. Mask Partition) (Ours) | 46.08 | 27.88 | 50.67 |
| Memory Bank (Ours) | 46.50 | 41.52 | 52.50 |

Figure: Figure 10: Visual comparison of different mask association methods. Gaga with 3D-aware memory bank achieves a superior visual quality, and closer to the ground truth. Notice that the Video Object Tracker baseline mislabels the wall and floor, Memory Bank (w. All Gaussians) mislabels the floor, and Memory Bank (w/o. Mask Partition) baseline creates artifacts on the table and chairs.
Refer to caption: x10.png

### 4.7 Ablation Study on Mask Association Method

We conduct ablation studies to evaluate the effectiveness of the proposed mask association method on the Replica dataset. The baselines for comparison include:

- 1.
w/o. Mask Association: Lifting inconsistent 2D masks to 3D.
- 2.
Video Object Tracker: [] is employed as a representative method.
- 3.
Memory Bank (w. All Gaussians): Associating masks in the same manner as Gaga, except that it selects all Gaussians splatted to the mask as its corresponding Gaussians.
- 4.
Memory Bank (w/o. Mask Partition): Associating masks in the same manner as Gaga, except that it does not partition the image and masks into patches.
- 5.
Memory Bank: *i.e*., Gaga.

Quantitative results in Tab. 4 indicate that Gaga with the 3D-aware memory bank achieves superior performance with a 24.74%, 16.52%, and 32.78% improvement on IoU, precision on recall, respectively, compared to the previous method with video object tracker. Comparison with the Memory Bank (w. All Gaussians) and Memory Bank (w/o. Mask Partition) baselines demonstrate the effectiveness of our well-designed process for finding corresponding Gaussians of each mask. We also show the visual comparison in Fig. 10.

## 5 Conclusions

We introduce Gaga, a framework that reconstructs and segments open-world 3D scenes by utilizing inconsistent 2D masks predicted by zero-shot segmentation models.
Gaga employs a 3D-aware memory bank to store the indices of pre-trained 3D Gaussians and establishes mask association across different views by identifying the overlap between Gaussians that are projected to each mask.
Results on various datasets demonstrate that Gaga outperforms previous methods with superior segmentation accuracy, multi-view consistency, and reduced artifacts.
Additionally, application in scene manipulation highlights Gaga’s high segmentation accuracy and practical utility.

## Appendix

In this appendix, we provide further experimental results, including a qualitative comparison with GARField [], additional results on scene manipulation and sparse view setting in Sec. B. We then delve into more experimental details of the datasets, metrics, and implementation in Sec. C. Additional ablation studies are shown in Sec. D and limitations are discussed in Sec. E.

## A Supplementary Video

Please watch the supplementary demo video for a comprehensive introduction and visual comparison between our method Gaga and the current state-of-the-art methods. The video features additional qualitative comparisons and an animation illustration of our method.

## B Supplementary Experiments

Figure: Figure 11: Qualitative comparison with GARField on Replica dataset. Gaga renders higher-quality RGB and segmentation masks in significantly less time. It’s worth noting that in the segmentation masks generated by GARField, the same colors are used multiple times for different masks, meaning one mask label may contain multiple groups representing different 3D instances.
Refer to caption: x11.png

### B.1 Results Compared with GARField

We provide comparison results with GARField in Fig. 11. GARField follows a hierarchical grouping pipeline. It extracts densely sampled segmentation masks from SAM [] and trains a feature field using contrastive loss for grouping. If two rays fall into the same SAM mask, their features will be pulled together. Otherwise, features are pushed apart.

We use the default setting to train GARField. For a fair comparison, Gaga also uses the 2D segmentation masks provided by SAM. Visualization results show that Gaga provides segmentation masks with better quality and multi-view consistency. Whereas GARField does not provide multi-view consistent segmentation, and it also has inferior RGB rendering results.

After training, GARField employs a hierarchical grouping pipeline to cluster each pixel into groups and generates segmentation masks. This hierarchical structure comprises 41 levels, and it takes approximately 20 minutes to output segmentation masks for a single image. In contrast, Gaga renders segmentation for one image under 0.5 seconds.

### B.2 Additional Results on Scene Manipulation

Figure: Figure 12: Scene manipulation results on MipNeRF 360 and LERF-Mask datasets. Gaga accurately identifies the flowerpot without affecting the color of the plant. Notice that Gaussian Grouping creates a cyan region on the wooden door behind. For the object removal and duplication tasks, Gaga can also provide more accurate results with fewer artifacts.
Refer to caption: x12.png

We provide additional results for the downstream scene manipulation task to further demonstrate the prospect of applying Gaga to real-world scenarios. On the "counter" scene of the MipNeRF 360 dataset [], we change the color of the flowerpot to cyan and duplicate the glass jar. Gaussian Grouping [] can not differentiate the plant and flowerpot, whereas Gaga generates a more accurate segmentation mask. Additionally, Gaga produces a clearer boundary and avoids artifacts on the iron tray when duplicating the glass jar.

In the "figurines" scene of the LERF-Mask dataset [], we transform the yellow duck to blue and remove the red toy chair. Gaga precisely changes only the duck’s color without affecting other objects, and achieves a more thorough removal of the red toy chair.

### B.3 Additional Results on Sparsely Sampled Replica Dataset

Figure: Figure 13: Qualitative results on the sparsely sampled Replica dataset. We showcase the novel view synthesis segmentation rendering results provided by Gaussian Grouping and Gaga as the percentage of training images employed decreases from 100% to 5%. Gaussian Grouping cannot correctly track the sofa under sparse views and fails to differentiate ceiling and wall, whereas Gaga consistently provides high-quality segmentation results.
Refer to caption: x13.png

We provide additional qualitative results for the experiment on the sparsely sampled replica dataset in Fig. 13. As the number of training images decreases, Gaussian Grouping produces more empty regions, *e.g*., the sofa and the wall, due to difficulties in accurate tracking under sparse views. Whereas Gaga exhibits a more robust performance against reductions in the number of images.

## C Experimental Details

### C.1 Details on Datasets

Replica Dataset [].
We select 8 scenes from the entire Replica dataset the same as []. We use the rendered results provided by authors of [] and follow their data processing process: for each scene, we uniformly select 20% images as training data and 20% images as test data from all rendered RGB images. This results in 180 training images and 180 test images for each scene.

Sparsely Sampled Replica Dataset.
For the same 8 scenes as the previous experiment, we randomly sample 30%, 20%, 10%, and 5% of the total 180 training images, resulting in 54, 36, 18, and 9 training images for each task, respectively. The number of test images remains at 180.

ScanNet Dataset [].
DM-NeRF [] selects 8 scenes from the entire ScanNet dataset. Each scene has approximately 300 images for training and about 100 images for testing. We utilize 7 out of the 8 scenes, excluding "scene 0024_00" due to the subpar 3D reconstruction results in both Gaussian Splatting [] and Gaussian Grouping [].

MipNeRF 360 Dataset [].
We downsample the images by a factor of 4 to accommodate the large size of the original images. For novel view synthesis evaluation, we set the sample step at 8, the same as the setting in [].

**Table 5: Selected scenes in Replica and ScanNet datasets. We select 8 scenes from the Replica dataset following [], and 7 scenes from the ScanNet dataset following [].**
| Dataset | Scene Name |  |  |  |
| --- | --- | --- | --- | --- |
| Replica [] | office 0 | office 1 | office 2 | office 3 |
| office 4 | room 0 | room 1 | room 2 |  |
| ScanNet [] | scene 0010_00 | scene 0012_00 | scene 0033_00 | scene 0038_00 |
| scene 0088_00 | scene 0113_00 | scene 0192_00 |  |  |

We employ the official script from Gaussian Splatting [] for colmap to acquire camera poses and the initial point cloud. Consequently, the actual number of images utilized in the experiment might be lower than expected due to colmap process failures. Please refer to Tab. 5 for the scene names used in the Replica and ScanNet datasets.

### C.2 Details on Evaluation Metrics

Given the disparate mask label assignments between the ground truth segmentation and the predicted segmentation for 3D objects, we find the best linear assignment between the labels based on IoU for quantitative evaluation. Subsequently, we employ IoU > 0.5 as the criterion for precision and recall calculations. We outline the pseudocode for the evaluation procedure in Algorithm 1.

Figure: Algorithm 1 Evaluation Metrics Input pred_masks and gt_masks are represented in binary format with shape ($n_{image}$, $n_{mask}$, $h$, $w$), where $n_{image}$ is the number of test images, $n_{mask}$ is the number of predicted or ground truth masks, $h$, $w$ are the height and width of test images. We use scipy.optimize.linear_sum_assignment to solve the linear assignment problem.

### C.3 Further Implementation Details

For training vanilla 3D Gaussians, we maintain the same parameter setting as []. To train the identity encoding, we freeze all the other attributes of Gaussians and use the same parameter setting as []. The identity encoding has 16 dimensions, and the rendered 2D identity encoding is in the shape of 16 $\times$ $h$ $\times$ $w$, where $h$ and $w$ represent the height and width of the image. The same classifier is utilized for predicting mask ID given the 2D identity encoding and selecting Gaussians for editing given the 3D identity encoding. It has 16 input channels and the number of output channels equals the number of groups in the 3D-aware memory bank after the mask association process for all images. All datasets are trained on a single NVIDIA RTX 6000 Ada GPU.

## D Supplementary Ablation Studies

We conduct additional ablation studies on three parameters involved in the process of mask association and find corresponding Gaussians of a mask. These ablation studies are performed on the Replica dataset [], utilizing SAM [] as the 2D segmentation model. Parameters denoted with * are used as the default setting. We also provide additional visual comparison results for the mask association methods utilized by Gaussian Grouping [] and Gaga in Sec. 14.

### D.1 Percentage of Front Gaussians

**Table 6: Ablation study on the percentage of front Gaussians. Results for selecting 10%, 20%, 30%, and 100% of front Gaussians as corresponding Gaussians of a mask are presented below. Gaga demonstrates stable performance across varying parameters, showcasing its robustness.**
| Perc. Front Gaussians (%) | IoU (%) | Precision (%) | Recall (%) |
| --- | --- | --- | --- |
| 10 | 46.42 | 39.57 | 51.54 |
| 20 * | 46.50 | 41.52 | 52.50 |
| 30 | 45.73 | 42.31 | 50.88 |
| 100 | 42.26 | 40.19 | 45.95 |

We present the ablation study on the percentage of front Gaussians selected as corresponding Gaussians in Tab. 6. We choose 10%, 20%, 30%, and 100% (*i.e*., selecting all Gaussians splatted to the mask as its corresponding Gaussians) as candidate parameters. The default setting (20%) has a better performance in general. Gaga shows stable performance for all candidate parameters, indicating its robustness and it does not rely on cautious parameter selection.

### D.2 Number of Image Patches During Partition

**Table 7: Ablation study on image partition. We partition the entire image and its masks into patches to prevent the selected corresponding Gaussians from concentrating in a confined region. Comparison results show that Gaga can perform well as long as the partition process is employed.**
| Num. Patches | IoU (%) | Precision (%) | Recall (%) |
| --- | --- | --- | --- |
| 1 $\times$ 1 | 46.08 | 27.88 | 50.67 |
| 16 $\times$ 16 | 46.11 | 38.22 | 51.62 |
| 32 $\times$ 32 * | 46.50 | 41.52 | 52.50 |
| 64 $\times$ 64 | 44.72 | 40.65 | 49.14 |

We provide the ablation study on the number of image patches used during the image partition process in Tab. 7. Candidate parameters include 1 $\times$ 1 (without mask partition process), 16 $\times$ 16, 32 $\times$ 32, 64 $\times$ 64. Similar to the results in Tab. 6, Gaga remains insensitive to the choice of this parameter as long as the image partition process is in place. Without the mask partition process, there is a significant drop in precision.

### D.3 Overlap Threshold

**Table 8: Ablation study on the overlap threshold. If the overlap between the current mask and all groups in the memory bank falls below this threshold, we add this mask to the memory bank as a new group. Results indicate that the default setting of 0.1 generally yields better outcomes.**
| Overlap Threshold | IoU (%) | Precision (%) | Recall (%) |
| --- | --- | --- | --- |
| 0.01 | 43.86 | 44.99 | 48.98 |
| 0.1 * | 46.50 | 41.52 | 52.50 |
| 0.2 | 47.57 | 34.77 | 52.40 |

During the group ID assigning process, if none of the existing groups in the memory bank has a larger overlap with the current mask than the threshold, we incorporate this mask into the memory bank as a new group, signifying the discovery of a new 3D object. We present the ablation study on overlap threshold in Tab. 8. When the threshold is set to 0.01, we rarely establish a new group and prefer to associate the mask with an existing group. It provides the best precision but at the expense of inferior IoU performance. Conversely, setting the threshold to 0.2 results in a frequent declaration of new group IDs, yielding the best IoU but a significant decrease in precision. Therefore, we set the threshold to 0.1 to strike a balance in performance across all three metrics.

### D.4 Additional Comparison on Mask Association Methods

Figure: Figure 14: Visual comparison between different mask association methods. Our 3D-aware memory bank offers more detailed associated masks. It accurately tracks identical objects in the scene and assigns them different mask IDs. Conversely, the video object tracker leaves empty regions in positions where it cannot track masks, and it struggles to provide consistent mask ID for the same object across views.
Refer to caption: extracted/5531443/FIG/sup_mask_associ_v3.jpeg

We present visual comparison results for two mask association methods, video object tracker [] utilized by [] and Gaga’s 3D-aware memory bank, in Fig. 14. In the "garden" scene of the MipNeRF 360 dataset, the video object tracker struggles to track objects in the background, whereas the 3D-aware memory bank provides associated results for each mask. For the scene in the ScanNet dataset, the video tracker fails to distinguish between four identical sofas, resulting in multiple masks for the same object. Additionally, it assigns different mask IDs to the table in two views. In contrast, the 3D-aware memory bank precisely locates each object, leading to improved mask association results and better pseudo labels for training segmentation features.

## E Limitations

Though Gaga achieves state-of-the-art performance compared to existing works, there are a few limitations and future works.
First, the optimization process of identity encoding and the rest of the Gaussian parameters are independent, this is because we need to first train 3D Gaussians to acquire their spatial location for mask association.
While this pipeline allows for the utilization of any pre-trained 3D Gaussians as input without the need to re-train the entire scene, it does require additional training steps.
We aim to enable the joint processing of mask association and identity encoding training in future works.

Secondly, artifacts may occur in the segmentation rendered by Gaga due to inherent inconsistency in the 2D segmentation.
For example, an object might be depicted as two separate masks in the initial view but as one entire mask in subsequent views.
This ambiguity introduces challenges to our mask association process.
Preprocessing steps such as dividing, merging, or reshaping the 2D segmentation masks could potentially resolve this issue and improve grouping results.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Can the proposed 3D-aware memory bank be used in other 3D representations such as NeRF, 3D point clouds? Some discussions are expected to be provided. 
In the final version, the author may include a brief discussion in their Future Work or Discussion section about potential adaptations or challenges in applying their 3D-aware memory bank to other 3D representations like NeRF or point clouds.
soundness: 3
strengths: 1. Clear motivation and insightful key idea.
2. Simple but effective, supported by extensive experiments on various datasets.
3. A demo video, more extensive details and limitations are provided in the supplementary material.
4. The paper is well-written.
summary: This paper proposes a new framework called Gaga to segment 3D Gaussians in the open world. The key idea is to assign a multi-view consistent universal mask ID before lifting the 2D result to 3D, which is achieved by searching a memory back for the category ID based on the overlap area with the unprotected mask. Extensive qualitative and quantitative results show the effectiveness of the proposed method.
weaknesses: 1. The whole framework is heavily based on the Identity Encoding proposed in Gaussian Grouping by utilizing 3D Gaussian representations. However, in the abstract, it does not mention the use of 3D GS. This should be addressed in the final version.  
2. Some important works [1, 2, 3, 4] should be discussed in the final version. Especially for both [1, 2, 4], they all propose to merge 3D results in 3D space, [4] also assign universal mask ID derived from prompt ID for better consistency.

*Refs:*.    
[1] SAM3D: Segment Anything in 3D Scenes. ICCV 2023 Workshop.   
[2] SAI3D: Segment Any Instance in 3D with Open Vocabularies. CVPR 2024.   
[3] OpenNeRF: Open-Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views. ICLR 2024.    
[4] SAMPro3D: Locating SAM Prompts in 3D for Zero-Shot Scene Segmentation. Arxiv 2023.   
[5] SAM-guided Graph Cut for 3D Instance Segmentation. ECCV 2024.   
[6] Segment3D: Learning Fine-Grained Class-Agnostic 3D Segmentation without Manual Labels. ECCV 2024.    

**Justification**:   
Since the main framework is heavily based on previous Gaussian Grouping, the idea is incremental. While considering its strong performance and extensive experiments, a Borderline/Weak Accept should be given.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. I am interested in how the performance of Gaga will hold on outdoor datasets and how different hyperparameter settings might impact the results in these scenarios.
2. From the qualitative results in Figures 6 and 7, it appears that Gaga, especially in combination with SAM, consistently produces a black boundary around objects—a phenomenon not seen with other methods. What might be the cause of this effect?
soundness: 3
strengths: 1. The introduction of a 3D-aware memory bank reduces dependence on continuous view changes in training images, enhancing
adaptability in segmentation.
2. Comprehensive experimental comparisons demonstrate the proposed method's effectiveness across multiple tasks, validating its versatility.
3. Gaga shows strong robustness to variations in the amount of training data. maintaining high performance even when training images are significantly reduced.
summary: The paper introduces Gaga, a framework for reconstructing and segmenting open-world 3D scenes using 2D masks from zero-shot segmentation models. Gaga focuses on achieving consistent label assignments across multiple views, crucial for accurate 3D segmentation. Using a 3D-aware memory bank, it proposes a mask association strategy to unify multi-view masks. Evaluations on datasets like LERF-Mask, Replica, ScanNet, and MipNeRF 360 for tasks such as open-vocabulary 3D queries and 3D segmentation demonstrate promising results, highlighting its potential for downstream scene manipulation applications.
weaknesses: 1. The proposed approach, which uses Gaussian overlap and ID assignment by overlap ratio, is straightforward but highly hand-crafted, relying on manually chosen hyperparameters. While results are promising, the method offers limited innovation to the research community.
2. Although effective on indoor datasets, testing on outdoor or mixed environments would better showcase generalizability.
3. Hyperparameter Sensitivity. Key hyperparameters like percentage $x$, patch size, and overlap ratio are crucial, potentially impacting robustness, especially with distribution shifts.
4. Some sections, such as the Gaussian selection process and Fig.4(b), could be more concise and easier to follow, the figure is not clear to show the problem with pixel-wise startegy.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: Besides the following question, please refer to the weaknesses section above for my concerns.
* In Supp Figure 1, it is good that the author provides a qualitative comparison with a method using contrastive learning. However, it lacks quantitative comparisons. Besides, the comparison does not make much sense as the GARField model seems not to have converged, and the rendered RGB from GARField is much worse than that from the proposed method. Furthermore, the clustering is sensitive to parameters, but there is no detailed information provided about these settings.
* I strongly encourage the authors to compare their approach with the additional baselines mentioned above and update them in the main experiments, particularly with OmniSeg3D [3], as it provides a 3D-GS implementation, enabling fairer comparisons. To compare with OmniSeg3D, the authors may consider using the same SAM input to train OmniSeg3D, followed by a clustering algorithm for the final results.
soundness: 2
strengths: * The proposed method is simple yet effective. It introduces a new mask association algorithm that leverages the intrinsic 3D information of reconstructed Gaussians.
* The paper is well-written and easy to follow.
* The experiments have covered datasets with various settings (e.g., different 2D models, and sparse views).
summary: This paper introduces a new Gaussian splatting segmentation method by designing a mask association algorithm and lifting the created view-consistent mask to the reconstructed Gaussians.

* Technique-wise, to address the inconsistency of 2D masks across different views, this paper designs a 3D-aware memory bank that gathers Gaussians belonging to the same group. This memory bank is utilized to align 2D masks across various views.

* The paper conducts extensive experiments to demonstrate the effectiveness of the proposed method compared to the previous method, i.e., Gaussian grouping using video tracking strategies for mask association.
weaknesses: * The paper lacks sufficient comparisons with state-of-the-art methods. For example, existing methods have proposed solutions such as the linear assignment [1] or contrastive learning methods [2,3] combined with clustering algorithms to bypass the need for complex mask association preprocessing. These methods should be included as main baselines to verify the effectiveness of the proposed approach. 

* Although the mask association is designed to group Gaussians from 2D pose images, the algorithm closely resembles SAM3D[4], which groups point clouds from 2D pose images. Are there any significant differences between the proposed algorithm and SAM3D?  A comprehensive discussion highlighting the technical novelties of the proposed algorithm against existing baselines is needed, if any exist.

* Although this paper introduces a new variant of mask association pre-processing with improved performance, there are no significant differences as it still suffers from similar limitations. Similar to video tracking, the proposed pre-processing involves many hyperparameter tuning and is sensitive to its hyperparameters, such as the percentage for finding corresponding Gaussians, overlap threshold, and image partition, as demonstrated in Tables 1, 2, and 3 in Supp. 

* In the proposed algorithm, using a fixed percentage to find corresponding Gaussians is too straightforward, while the optimal percentage should be dependent on the opacity properties. Hence, the preset parameter is likely to be unsuitable for new scenes and may potentially limit the generalization of the proposed method.




[1] Panoptic Lifting for 3D Scene Understanding (CVPR 2023)

[2] Contrastive Lift: 3D Object Instance Segmentation by Slow-Fast Contrastive Fusion. (NeurIPS 2023)

[3] OmniSeg3D: Omniversal 3D Segmentation via Hierarchical Contrastive Learning (CVPR2024).

[4] SAM3D: Segment Anything in 3D Scenes (ICCV'23 Workshop)

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Could you provide visual examples to demonstrate that selecting the nearest 20% of Gaussians is robust across different scenarios? Additionally, is there any evidence that some Gaussians relevant to the segmentation may be excluded?
2. How your method handles cases where the ground truth segmentation is more fine-grained than their mask partitioning approach allows. For example, in the replica ground truth segmentation, the left and right panels in Figure 4b belong to different masks.  However, the mask partition merges them into a single object, which can reduce segmentation accuracy.
soundness: 2
strengths: 1. Gaga proposes a novel 3D-aware memory bank that improves multi-view consistency in 3D segmentation by leveraging spatial information from 3D Gaussians, providing an innovative alternative to video-based mask association methods.
2. Extensive experiments validate Gaga's robustness across challenging scenarios, datasets, and limited data.
summary: The paper introduces Gaga, a framework that segments open-world 3D scenes by addressing label inconsistencies in 2D masks. Using a 3D-aware memory bank, Gaga achieves multi-view consistent segmentation by associating 2D masks across views with a universal ID.
weaknesses: 1. As noted in Fig. 4a (View 2), the author selects the nearest 20% of Gaussians, considering that the background may contribute to foreground segmentation. However, in cases like Fig. 4 (View 1), where the background does not influence the foreground, this approach may lead to Gaussians from the target partition being omitted.  Please provide visual analysis on how different percentages of nearest Gaussians affect performance across various scenarios.  
2.The comparison with existing methods is limited. Including additional comparisons with state-of-the-art approaches, such as Panoptic-Lifting [1] and Contrastive-Lift [2].

[1] Siddiqui Y, Porzi L, Buló S R, et al. Panoptic lifting for 3d scene understanding with neural fields[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 9043-9052.
[2] Bhalgat Y, Laina I, Henriques J F, et al. Contrastive lift: 3d object instance segmentation by slow-fast contrastive fusion[J]. arXiv preprint arXiv:2306.04633, 2023.

Meta-review: The paper proposes an approach to multi-view consistent grouping via gaussian splatting. It builds on the recent Gaussian grouping work and replaces the video-tracker-based mask assignment across frames with an overlap-based assignment and a memory bank to track group assignments. Reviewers found the assignment novel, technically sound, and well explained. Extensive qualitative and quantitative results demonstrate improvements over the baseline.

Reviewers remarked a missing discussion of and comparison to relevant related work. Some reviewers considered the contributions to be incremental and insufficient for acceptance, which was countered by the strong performance delta to the baseline. Reviewers were further concerned about the sensitivity to hyperparameter choices that they expected to be domain-dependent (e.g. outdoor scenes requiring a different threshold to cut off Gaussians than object-centric scenes.).

The AC agrees with reviewers that the technical contribution is rather incremental, but the significantly better results in comparison to the baseline could warrant acceptance. This would however require an experimental evaluation not only against the baseline approach, but other methods representative of the current state of the art to signify that the proposed change is meaningful not only with respect to the baseline but the current state of the art. Experimental results added during the discussion phase together with statements made by the authors raised doubts over the thoroughness of the baseline setup and thus the fairness of evalution. With these doubts not cleared by the authors during the discussion, the AC can only recommend to reject the paper.

Decision: Reject