Gaga: Group Any Gaussians via 3D-aware Memory Bank

Authors: Weijie Lyu, Xueting Li, Abhijit Kundu, Yi-Hsuan Tsai, Ming-Hsuan Yang
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: 3D Open-world Segmentation; Gaussian Splatting; Scene Understanding

Abstract:
We introduce *Gaga*, a framework that reconstructs and segments open-world 3D scenes by leveraging inconsistent 2D masks predicted by zero-shot class-agnostic segmentation models. Contrasted to prior 3D scene segmentation approaches that heavily rely on video object tracking, *Gaga* utilizes spatial information provided by 3D Gaussians and effectively associates object masks across diverse camera poses through a novel 3D-aware memory bank. By eliminating the assumption of continuous view changes in training images, *Gaga* demonstrates robustness to variations in camera poses, particularly beneficial for sparsely sampled images, ensuring precise mask label consistency. Furthermore, *Gaga* accommodates 2D segmentation masks from diverse sources and demonstrates robust performance with different open-world zero-shot class-agnostic segmentation models, significantly enhancing its versatility. Extensive qualitative and quantitative evaluations demonstrate that *Gaga* performs favorably against state-of-the-art methods, emphasizing its potential for real-world applications such as scene understanding and manipulation. The source codes will be made available to the public.

Decision: reject
Reviewer scores: 8, 5, 3, 5

Reviews and discussion:
<thread>
creation date:
1730297351857

thread_type:
official_review

<thread_object>
creator: reviewer_az4n

<content>
summary:
The paper introduces Gaga, a framework that segments open-world 3D scenes by addressing label inconsistencies in 2D masks. Using a 3D-aware memory bank, Gaga achieves multi-view consistent segmentation by associating 2D masks across views with a universal ID.

strengths:
1. Gaga proposes a novel 3D-aware memory bank that improves multi-view consistency in 3D segmentation by leveraging spatial information from 3D Gaussians, providing an innovative alternative to video-based mask association methods.
2. Extensive experiments validate Gaga's robustness across challenging scenarios, datasets, and limited data.

weaknesses:
1. As noted in Fig. 4a (View 2), the author selects the nearest 20% of Gaussians, considering that the background may contribute to foreground segmentation. However, in cases like Fig. 4 (View 1), where the background does not influence the foreground, this approach may lead to Gaussians from the target partition being omitted.  Please provide visual analysis on how different percentages of nearest Gaussians affect performance across various scenarios.  
2.The comparison with existing methods is limited. Including additional comparisons with state-of-the-art approaches, such as Panoptic-Lifting [1] and Contrastive-Lift [2].

[1] Siddiqui Y, Porzi L, Buló S R, et al. Panoptic lifting for 3d scene understanding with neural fields[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 9043-9052.
[2] Bhalgat Y, Laina I, Henriques J F, et al. Contrastive lift: 3d object instance segmentation by slow-fast contrastive fusion[J]. arXiv preprint arXiv:2306.04633, 2023.

questions:
1. Could you provide visual examples to demonstrate that selecting the nearest 20% of Gaussians is robust across different scenarios? Additionally, is there any evidence that some Gaussians relevant to the segmentation may be excluded?
2. How your method handles cases where the ground truth segmentation is more fine-grained than their mask partitioning approach allows. For example, in the replica ground truth segmentation, the left and right panels in Figure 4b belong to different masks.  However, the mask partition merges them into a single object, which can reduce segmentation accuracy.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730387908974

thread_type:
official_review

<thread_object>
creator: reviewer_fmmx

<content>
summary:
This paper introduces a new Gaussian splatting segmentation method by designing a mask association algorithm and lifting the created view-consistent mask to the reconstructed Gaussians.

* Technique-wise, to address the inconsistency of 2D masks across different views, this paper designs a 3D-aware memory bank that gathers Gaussians belonging to the same group. This memory bank is utilized to align 2D masks across various views.

* The paper conducts extensive experiments to demonstrate the effectiveness of the proposed method compared to the previous method, i.e., Gaussian grouping using video tracking strategies for mask association.

strengths:
* The proposed method is simple yet effective. It introduces a new mask association algorithm that leverages the intrinsic 3D information of reconstructed Gaussians.
* The paper is well-written and easy to follow.
* The experiments have covered datasets with various settings (e.g., different 2D models, and sparse views).

weaknesses:
* The paper lacks sufficient comparisons with state-of-the-art methods. For example, existing methods have proposed solutions such as the linear assignment [1] or contrastive learning methods [2,3] combined with clustering algorithms to bypass the need for complex mask association preprocessing. These methods should be included as main baselines to verify the effectiveness of the proposed approach. 

* Although the mask association is designed to group Gaussians from 2D pose images, the algorithm closely resembles SAM3D[4], which groups point clouds from 2D pose images. Are there any significant differences between the proposed algorithm and SAM3D?  A comprehensive discussion highlighting the technical novelties of the proposed algorithm against existing baselines is needed, if any exist.

* Although this paper introduces a new variant of mask association pre-processing with improved performance, there are no significant differences as it still suffers from similar limitations. Similar to video tracking, the proposed pre-processing involves many hyperparameter tuning and is sensitive to its hyperparameters, such as the percentage for finding corresponding Gaussians, overlap threshold, and image partition, as demonstrated in Tables 1, 2, and 3 in Supp. 

* In the proposed algorithm, using a fixed percentage to find corresponding Gaussians is too straightforward, while the optimal percentage should be dependent on the opacity properties. Hence, the preset parameter is likely to be unsuitable for new scenes and may potentially limit the generalization of the proposed method.




[1] Panoptic Lifting for 3D Scene Understanding (CVPR 2023)

[2] Contrastive Lift: 3D Object Instance Segmentation by Slow-Fast Contrastive Fusion. (NeurIPS 2023)

[3] OmniSeg3D: Omniversal 3D Segmentation via Hierarchical Contrastive Learning (CVPR2024).

[4] SAM3D: Segment Anything in 3D Scenes (ICCV'23 Workshop)

questions:
Besides the following question, please refer to the weaknesses section above for my concerns.
* In Supp Figure 1, it is good that the author provides a qualitative comparison with a method using contrastive learning. However, it lacks quantitative comparisons. Besides, the comparison does not make much sense as the GARField model seems not to have converged, and the rendered RGB from GARField is much worse than that from the proposed method. Furthermore, the clustering is sensitive to parameters, but there is no detailed information provided about these settings.
* I strongly encourage the authors to compare their approach with the additional baselines mentioned above and update them in the main experiments, particularly with OmniSeg3D [3], as it provides a 3D-GS implementation, enabling fairer comparisons. To compare with OmniSeg3D, the authors may consider using the same SAM input to train OmniSeg3D, followed by a clustering algorithm for the final results.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730616511285

thread_type:
official_review

<thread_object>
creator: reviewer_vyo9

<content>
summary:
The paper introduces Gaga, a framework for reconstructing and segmenting open-world 3D scenes using 2D masks from zero-shot segmentation models. Gaga focuses on achieving consistent label assignments across multiple views, crucial for accurate 3D segmentation. Using a 3D-aware memory bank, it proposes a mask association strategy to unify multi-view masks. Evaluations on datasets like LERF-Mask, Replica, ScanNet, and MipNeRF 360 for tasks such as open-vocabulary 3D queries and 3D segmentation demonstrate promising results, highlighting its potential for downstream scene manipulation applications.

strengths:
1. The introduction of a 3D-aware memory bank reduces dependence on continuous view changes in training images, enhancing
adaptability in segmentation.
2. Comprehensive experimental comparisons demonstrate the proposed method's effectiveness across multiple tasks, validating its versatility.
3. Gaga shows strong robustness to variations in the amount of training data. maintaining high performance even when training images are significantly reduced.

weaknesses:
1. The proposed approach, which uses Gaussian overlap and ID assignment by overlap ratio, is straightforward but highly hand-crafted, relying on manually chosen hyperparameters. While results are promising, the method offers limited innovation to the research community.
2. Although effective on indoor datasets, testing on outdoor or mixed environments would better showcase generalizability.
3. Hyperparameter Sensitivity. Key hyperparameters like percentage $x$, patch size, and overlap ratio are crucial, potentially impacting robustness, especially with distribution shifts.
4. Some sections, such as the Gaussian selection process and Fig.4(b), could be more concise and easier to follow, the figure is not clear to show the problem with pixel-wise startegy.

questions:
1. I am interested in how the performance of Gaga will hold on outdoor datasets and how different hyperparameter settings might impact the results in these scenarios.
2. From the qualitative results in Figures 6 and 7, it appears that Gaga, especially in combination with SAM, consistently produces a black boundary around objects—a phenomenon not seen with other methods. What might be the cause of this effect?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730642078813

thread_type:
official_review

<thread_object>
creator: reviewer_d6rq

<content>
summary:
This paper proposes a new framework called Gaga to segment 3D Gaussians in the open world. The key idea is to assign a multi-view consistent universal mask ID before lifting the 2D result to 3D, which is achieved by searching a memory back for the category ID based on the overlap area with the unprotected mask. Extensive qualitative and quantitative results show the effectiveness of the proposed method.

strengths:
1. Clear motivation and insightful key idea.
2. Simple but effective, supported by extensive experiments on various datasets.
3. A demo video, more extensive details and limitations are provided in the supplementary material.
4. The paper is well-written.

weaknesses:
1. The whole framework is heavily based on the Identity Encoding proposed in Gaussian Grouping by utilizing 3D Gaussian representations. However, in the abstract, it does not mention the use of 3D GS. This should be addressed in the final version.  
2. Some important works [1, 2, 3, 4] should be discussed in the final version. Especially for both [1, 2, 4], they all propose to merge 3D results in 3D space, [4] also assign universal mask ID derived from prompt ID for better consistency.

*Refs:*.    
[1] SAM3D: Segment Anything in 3D Scenes. ICCV 2023 Workshop.   
[2] SAI3D: Segment Any Instance in 3D with Open Vocabularies. CVPR 2024.   
[3] OpenNeRF: Open-Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views. ICLR 2024.    
[4] SAMPro3D: Locating SAM Prompts in 3D for Zero-Shot Scene Segmentation. Arxiv 2023.   
[5] SAM-guided Graph Cut for 3D Instance Segmentation. ECCV 2024.   
[6] Segment3D: Learning Fine-Grained Class-Agnostic 3D Segmentation without Manual Labels. ECCV 2024.    

**Justification**:   
Since the main framework is heavily based on previous Gaussian Grouping, the idea is incremental. While considering its strong performance and extensive experiments, a Borderline/Weak Accept should be given.

questions:
Can the proposed 3D-aware memory bank be used in other 3D representations such as NeRF, 3D point clouds? Some discussions are expected to be provided. 
In the final version, the author may include a brief discussion in their Future Work or Discussion section about potential adaptations or challenges in applying their 3D-aware memory bank to other 3D representations like NeRF or point clouds.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732312134849

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to All Reviewers

comment:
We sincerely appreciate your positive and constructive feedback. We would like to address common questions first in the following:

**[Question 1] Hyperparameters sensitivity.**

We use three hyperparameters in our experiments: Percentage of front Gaussians, number of image patches during mask partition, and Gaussian overlap threshold. We use the same set of hyperparameters (20%, 32 $\times$ 32, 0.1) for all experiments in the paper to show that *Gaga* does not require scene-specific hyperparameter tuning for optimal results. Additionally, we provide ablation studies in the supplementary material (Sec. D) to show that our method is robust to hyperparameter selection, as long as the key concepts behind these hyperparameters (i.e., selecting front Gaussians to avoid including background objects, the mask partition process, etc.) are in place. To summarize, *Gaga* is not sensitive to hyperparameters and does not need hand-craft tuning.

Meanwhile, language-embedding-based methods, such as LERF, LEGaussians, LangSplat, and contrastive learning-based approaches like OmniSeg3D, all rely on hyperparameters, such as similarity threshold to determine whether two pixels should be grouped into the same segment. Similarly, Gaussian Grouping employs the video tracking method DEVA, which also requires manually set parameters. As reviewer FmMx noted, GARField also depends on hyperparameter selection and is sensitive to these parameters. Although our method involves hyperparameters, this is a common practice in related research, and our approach is not sensitive to these parameters.

**[Question 2] Comparison with existing methods.**

* **Panoptic Lifting** and **Contrastive Lift**: Both of these two methods focus on 3D panoptic segmentation or instance segmentation. They associate inconsistent labels of instance segmentation masks where class labels are known. Instead, *Gaga* focuses on class-agnostic segmentation. We tried to apply Panoptic Lifting's linear assignment mask association method to class-agnostic segmentation masks but could not obtain meaningful results. Fig. 4 row 1 of the Gaussian Grouping paper also shows similar findings.
* **OmniSeg3D**: We compare with OmniSeg3D on LERF-Mask Dataset and the results are shown as follows. Note that OmniSeg3D requires a manually input pixel x, y to select target segmentation with feature similarity, we use GroundingDINO to get a segmentation mask for the test prompt and select the center of the mask as the manually input pixel x, y. We use similarity threshold 0.9 as there's no official setting in OmniSeg3D, and 0.9 gives us the best results.

*mIoU(%)*

| Method | figurines | ramen | teatime| Overall |
| --- | --- | --- | --- | --- |
| OmniSeg3D | 70.21 | 60.66 | 68.39 | 66.42 |
| *Gaga* | 90.66 | 64.15 | 69.34 | 74.71 |

*mBIoU(%)*

| Method | figurines | ramen | teatime| Overall |
| --- | --- | --- | --- | --- |
| OmniSeg3D | 68.99 | 54.91 | 62.68 | 62.20 |
| *Gaga* | 88.99 | 61.59 | 65.99 | 72.19 |

Furthermore, OmniSeg3D (similarly GARField) attaches features learned through contrastive learning to each Gaussian, which is different from our learned identity encoding. Their features cannot be used for rendering multi-view consistent segmentation maps, which is the focus of our approach. This also explains why we cannot compare with their methods on Replica or ScanNet datasets, as their segmentation maps are not view-consistent.

* **SAM3D**: SAM3D focuses on point clouds segmentation. Please refer to the response to reviewer FmMx for a detailed discussion about the difference between SAM3D and *Gaga*.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732458495754

thread_type:
official_comment

<thread_object>
creator: reviewer_fmmx

<content>
title:
Severe issues of this paper

comment:
After reading the initial submission and the rebuttal, I found that the following severe issues have not been resolved in this paper:


* **Comparisons seem spurious to me.**

During the rebuttal, the author provides the results of OmniSeg-3D on the LERF-Masked dataset. However, the provided result (mIoU: 66.4) of OmniSeg-3D is not aligned with the results (mIoU: 79.4) reported in the recent literature [1]. For me, this discrepancy is unusual, and the author should provide more clarification for this.

[1] Click-Gaussian: Interactive Segmentation to Any 3D Gaussians. ECCV 2024.

Besides, the author provides qualitative comparisons with the one NeRF-based baseline (i.e., GARField) in the supplementary materials, but it seems that the NeRF backbone has not converged sufficiently to reconstruct the 3D scene. Consequently, it does not make much sense to compare the segmentation results as the quality of segmentation is highly dependent on the quality of reconstruction. Furthermore, the author should provide more details about the experiment settings, as the results presented do not align with typical experimental outcomes on the widely-used Replica dataset.

* **The originality of the proposed idea is not clear.**

The originality of the proposed idea (mask association algorithm) is not clear. Essentially, this algorithm is similar to point cloud segmentation algorithms like SAM3D. It is important to note that the proposed algorithm only uses the Gaussian mean (3D point) as the algorithm input. However, I **did not** observe any significant differences compared to SAM3D, and **no** quantitative comparisons were provided. **Given the above issue, I have a strong concern about whether the paper merely adapts the SAM3D algorithm to solve the mask association algorithm.** If so, the current version does not meet the standards expected of an ICLR paper.


* **Lack of comparisons with existing baselines.**

For the main baseline for multi-view consistent segmentation, this paper only compares it with Gaussian Grouping, which is far from sufficient. After I highlighted that there are many important baselines including Panoptic Lifting, Contrastive Lift, and OmniSeg-3D, the author provides the results of **only OmniSeg-3D** on a **single** dataset (LERF-Masked). Although the results of OmniSeg-3D are lower than those of the proposed method on this dataset, I do not see **any convincing reasons** for this outcome. Particularly, some discussions are confusing and even inaccurate. These confusions raise further concerns about whether the newly added comparison is conducted in a fair setting.

By the way, I noted that the author provides some comparisons with language-based (i.e., CLIP) methods. However, the results do not make much sense as these methods primarily focus on encoding open-vocabulary features and do not concentrate on providing an accurate segmentation mask output.

* **Hyperparameters sensitivity.**

Since the proposed association algorithm is a **purely hand-crafted** algorithm based on Euclidean space, the hyperparameters are closely related to varying physical sizes of objects across different scenes. Although the proposed algorithm provided various strategies to solve this problem, they resemble engineering tricks more than robust solutions. Given that the single algorithm contains too many hyperparameters (percentage for finding corresponding Gaussians, overlap threshold, and image partition), it is difficult to verify its robustness through experiments on limited datasets. Hence, I have concerns about the sensitivity to hyperparameters.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732524096707

thread_type:
official_comment

<thread_object>
creator: reviewer_d6rq

<content>
title:
I Believe the Practical Value of This Paper Deserves a Borderline Accept or Weak Accept

comment:
Dear All Reviewer,

I have carefully read all of your reviews.

I want to raise one key point is that: this paper indeed shows some practical values compared to previous works. 

Although there may be some minor issues or limited theoretical novelty of this paper, the practical value of it may still contribute a lot to the community. Also, the experiments are very extensive, from my understanding.

As for **the comparison with SAM3D** as mentioned by Reviewer FmMx, from my understanding, the proposed method in this submission solves the multi-view consistency problem, while SAM3D can not as it uses automatic-SAM on each frame (which must cause the view-inconsistency problem). Considering this, **the novelty of the submission is valid**. (However, the authors are still encouraged to provide more discussions with SAM3D, SAMPro3D, etc.)

I think that the standard to judge a paper should not be single or fixed.

I have read lots of accepted papers of recent ICLR papers or other top conferences, and I believe that this paper deserves a Borderline Accept or even Weak Accept.

Currently, to defend my views, I would raise the score to Accept (since there is no Weak Accept option)
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes an approach to multi-view consistent grouping via gaussian splatting. It builds on the recent Gaussian grouping work and replaces the video-tracker-based mask assignment across frames with an overlap-based assignment and a memory bank to track group assignments. Reviewers found the assignment novel, technically sound, and well explained. Extensive qualitative and quantitative results demonstrate improvements over the baseline.

Reviewers remarked a missing discussion of and comparison to relevant related work. Some reviewers considered the contributions to be incremental and insufficient for acceptance, which was countered by the strong performance delta to the baseline. Reviewers were further concerned about the sensitivity to hyperparameter choices that they expected to be domain-dependent (e.g. outdoor scenes requiring a different threshold to cut off Gaussians than object-centric scenes.).

The AC agrees with reviewers that the technical contribution is rather incremental, but the significantly better results in comparison to the baseline could warrant acceptance. This would however require an experimental evaluation not only against the baseline approach, but other methods representative of the current state of the art to signify that the proposed change is meaningful not only with respect to the baseline but the current state of the art. Experimental results added during the discussion phase together with statements made by the authors raised doubts over the thoroughness of the baseline setup and thus the fairness of evalution. With these doubts not cleared by the authors during the discussion, the AC can only recommend to reject the paper.