InfoGS: Efficient Structure-Aware 3D Gaussians via Lightweight Information Shaping

Authors: Yunchao Zhang, Guandao Yang, Leonidas Guibas, Yanchao Yang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Mutual Information Maximization; 3D Reconstruction; 3D Editing

Abstract:
3D Gaussians, as an explicit scene representation, typically involve thousands to millions of elements per scene. This makes it challenging to control the scene in ways that reflect the underlying semantics, where the number of independent entities is typically much smaller. Especially, if one wants to animate or edit objects in the scene, as this requires coordination among the many Gaussians involved in representing each object. To address this issue, we develop a mutual information shaping technique that enforces resonance and coordination between correlated Gaussians via a Gaussian attribute decoding network. Such correlations can be learned from putative 2D object masks in different views. By approximating the mutual information with the gradients concerning the network parameters, our method ensures consistency between scene elements and enables efficient scene editing by operating on network parameters rather than massive Gaussians. In particular, we develop an effective learning pipeline named ***InfoGS*** with lightweight optimization to shape the attribute decoding network ,while ensuring that the shaping (consistency) is maintained during continuous edits,  avoiding re-shaping after parameter changes. Notably,  our training only touches a small fraction of all Gaussians in the scene yet attains the desired correlated behavior according to the underlying scene structure. The proposed technique is evaluated on challenging scenes and demonstrates significant performance improvements in 3D object segmentation and promoting scene interactions, while inducing low computation and memory requirements. Our code is available at: https://github.com/StylesZhang/InfoGS.

Decision: accept (poster)
Reviewer scores: 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730271198124

thread_type:
official_review

<thread_object>
creator: reviewer_fawd

<content>
summary:
This paper introduces a mutual information technique based on 3D Gaussian splatting. Building upon JacobiNeRF, the authors present a more efficient training pipeline that optimizes activations instead of Jacobians. This approach ensures that the correlations between Gaussians are correctly shaped and remain consistent through successive parameter changes. Experiments on open-vocabulary segmentation and scene editing demonstrate the efficiency and significant capabilities of the proposed method.

strengths:
1. The paper is well-written and organized, the authors provide a thorough theoretical analysis of the proposed method.
2. Experiments demonstrate that the proposed method outperforms other baselines while maintaining robustness to incorrect masks.

weaknesses:
1. Compared to methods that edit the scene after obtaining the labels for each Gaussian, such as Gaussian Grouping, the direct manipulation of network parameters appears less flexible. For instance, while Gaussian Grouping can achieve style transfer, the proposed method seems limited to modifying the colors of selected Gaussians to be uniform.

2. The authors conduct open-vocabulary segmentation in only three scenes, evaluating just one object per scene if I understand correctly. Including more experiments would enhance the robustness and credibility of the results.

questions:
1. Aside from JacobiGS in the paper, which utilizes the same mutual information shaping loss as JacobiNeRF, why was the original JacobiNeRF not included as a baseline for comparison?


2. Regarding weakness 2, is it possible for the proposed method to edit the color of an object to a textured color? Additionally, for the object movement task, can the method modify the trajectory of a selected object to follow a position-related trajectory, such as rotation?


3. Besides open-vocabulary segmentation tasks, would it be feasible to conduct experiments on semantic or instance segmentation tasks to further enhance the credibility of the findings?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730526924186

thread_type:
official_review

<thread_object>
creator: reviewer_fqfp

<content>
summary:
This paper proposes information shaping for the 3D Gaussian field, following NeRF-based work JacobiNeRF. By contrastive learning between Gaussian primitives, the resulting Gaussian primitives on the same entities have a strong correlation while primitives on different entities have a weak correlation. The technical part follows the basic idea of JacobiNeRF and is tailored to 3D Gaussian with a attributes decoding network.  The learned Gaussian field is easier to edit, such as object removal and object movement.

strengths:
- This paper adapts the basic idea of JacobiNeRF to 3DGS and makes it work in terms of scene editing.
- The method achieves good performance in the open vocabulary segmentation task.
- The application of object removal seems have good qualitative performance.

weaknesses:
- The technical novelty is limited. The Jacobian-based mutual information learning is the core part of this paper, which mainly follows JacobiNeRF. The authors are suggested to emphasize the unique designs that are different from the JacobiNeRF and demonstrate the effectiveness of the unique designs. The mask guidance and smoothness regularization differ from the DINO adopted in JacobiNeRF. However, these components seem to be incremental techniques, and their effectiveness is not well validated in the experiments.
- The writing should be improved. Many concepts lack a clear definition. For example, what are the exact definitions of perturbation and the sequential of perturbations (e.g., 1st/2nd/3rd perturbation). What are the exact parameters perturbated and how are they perturbated (i.e., perturbation direction and norm).
- The demonstration is weak. The object movement in Figure 6 is not impressive and it seems to be a sample handpicked with effort. It would be better to present more object movement demonstrations, using the main components of a 3D scene, such as the earthmover toy in the kitchen scene. Can you move the earthmover to anywhere else on the desk? Furthermore, the demonstrations have low diversity, mainly adopting the scene of kitchen and bear.
- The method outperforms JacobiNeRF and JacobiGS. However, the authors do not show the reasons via an apple-to-apple comparison with JacobiNeRF and JacobiGS. As I suggested in the first item, the authors are encouraged to show the performance roadmap from the JacobiGS baseline (or other reasonable baseline) and add the unique components one by one, offering a better understanding of the inner workings to readers.

questions:
Does the decoding network reduce the rendering efficiency?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730777699582

thread_type:
official_review

<thread_object>
creator: reviewer_wxpj

<content>
summary:
This paper presents a method for correlating Gaussians that belong to the same semantic group. It leverages a network to decode feature Gaussians into attributes and approximates mutual information via gradients with respect to network parameters. This approach enables editing operations through adjustments to the network parameters rather than direct manipulation of numerous Gaussians. The mutual information between Gaussians is approximated by shaping activations from perturbed network weights.

strengths:
Previous approaches embed semantic features directly within each Gaussian primitive, requiring extensive manipulation of individual Gaussians to edit scenes. This paper instead proposes encoding mutual information within an attribute decoding network, allowing scene edits to be made by directly adjusting the network parameters.

weaknesses:
The paper is generally challenging to follow, with many points lacking clear explanations. For example, Equation 7 and Line 304 state that "we find that ∂h corresponds to repeated activations σ(h(l−1))." However, this lacks sufficient context, as does Line 317, which mentions that "shaping activations ∂h within the attribute decoding network Φa supports a sequence of editing operations." Further explanation on why shaping these activations enables multiple edits would improve clarity.

The proposed method uses an MLP network to model mutual information between Gaussians, where the Gaussians primarily serve to splat features onto images used as inputs for the MLP. The primary contribution appears to be an enhancement to the JacobiNeRF framework, rather than a significant innovation in the context of 3D Gaussian Splatting (3DGS).

The contributions of Section 3.5 on coarse mask guidance and Section 3.6 on smoothness regularization seem minor, primarily focused on refining supervision masks. These sections could potentially be merged, as similar techniques have been explored in prior work.

The method depends on a 2D video mask tracker for grouping Gaussians. Based on the experimental results, it is difficult to determine if performance improvements arise from the mask tracker or the proposed approach. For instance, in Figure 4, the poor mask quality generated by Gaussian Grouping may stem from suboptimal 2D masks.

questions:
Please refer to the weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732122616495

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response addressing all Reviewers (Part 1)

comment:
We would like to thank the reviewers for the insightful feedback! We are glad that reviewers find our study “unique” compared to pervious approaches (Reviewer WXPj), “effective” (Reviewer fQfp, fawD), and “theoretically insightful” (Reviewer faWD). To recap, we propose a mutual information shaping technique that can coordinate correlated Gaussians via an attribute decoding net, and the scene could be edited by adjusting the network parameters. To the best of our knowledge, we are the first to explore the mutual information between Gaussians to boost the efficiency of editing in the 3DGS field. We demonstrate that our shaping technique can be more efficient than previous 3D editing approaches. Our method can outperform several baselines in tasks including removal, inpainting, and segmentation. **We hope our paper inspires innovative designs for 3D representations that effectively leverage correlation information, enabling interactive tasks requiring a deep understanding of the underlying scene structure.**

Below, we address the reviewers’ common questions. 

## **Common Issue #1. Major Contribution** ##
Reviewers WXPj and fQfp expressed confusion about our major contribution. We are happy to clarify our contributions by providing a clear roadmap.

Our core contribution is innovating the existing 3DGS framework by leveraging mutual information, thereby enhancing the performance of various 3D editing tasks. Previously, JacobiNeRF introduced point-wise mutual information shaping based on NeRF for label propagation, which only allows one-time perturbation consistency. Extending JacobiNeRF's method to 3DGS for editing is nontrivial, because:

1. NeRF and 3DGS are fundamentally different types of representations. NeRF is a continuous scene representation modeled by a single MLP, whereas 3DGS represents a scene as a discrete set of millions of independent 3D Gaussians.To address the challenge of associating these independent Gaussians, we introduce a shared attribute decoding network that decodes the attributes for all Gaussians. By applying a similar mutual information shaping technique as in JacobiNeRF to this updated 3DGS framework, **we establish a new baseline for correlation shaping in 3DGS, named JacobiGS**. For a fair comparison, **we maintain consistency in other loss terms and supervision refinement (video tracker and regularization loss) within JacobiGS, as in our proposed method.**

2. To achieve 3D editing beyond label propagation, we modify JacobiNeRF’s shaping loss based on further derivations of the Jacobians. Our detailed theoretical analysis, provided in Appendix C, leads to the final result in Eq. 7. This new shaping loss ensures correlation consistency among Gaussians after parameter adjustments, without the need for the direct calculation of higher-order gradients in JacobiNeRF. **By solely replacing the original shaping loss in the JacobiGS framework with the new shaping loss, our method supports versatile and consecutive 3D editing**. In the task of 3D object movement (as shown in Fig. 6), existing techniques fail after three steps of continuous scene manipulation. In contrast, our method successfully maintains consistency throughout continuous editing, affecting only the target object. We have included videos from different views and scenes in the supplementary materials to persuasively illustrate the differences between JacobiGS and our method in object movement tasks involving successive operations. For tasks like 3D segmentation that do not require consecutive manipulations, as demonstrated in Table 1, JacobiGS performs worse compared to existing 3D segmentation methods. It indicates our proposed loss not only ensures consistency but also performs better in shaping correlations among Gaussians.
3. Furthermore, the original JacobiNeRF method is computationally expensive. As shown in Table 1, JacobiGS takes almost ten times longer for fine-tuning (excluding the time required for reconstruction, which is the same). The batch size for JacobiGS per epoch is only 48 (ours uses 512), and we maintain the same number of fine-tuning iterations for both JacobiGS and our method (refer to “training hyperparameters” in Appendix A). Additionally, the Jacobian calculation becomes extremely computationally intensive when the network layer being optimized is far from the network output. The computation path of Jacobian is longer in this case. **In practice, optimizing the Jacobian of the second layer and sampling only 128 Gaussians per epoch for contrastive learning in JacobiGS exceeds the memory limits of a single A100 GPU and requires hours for fine-tuning.**

In conclusion, our motivation is fundamentally different from JacobiNeRF, which focuses on label propagation. Our contribution and novelty lie in providing a theoretical analysis for consistent mutual information shaping, along with proposing a training pipeline with lightweight optimization that yields superior performance in 3D editing tasks.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a method using mutual information shaping technique for 3D Gaussian splatting. It has the advantages of efficient scene editing through attribute decoding networks, rather than direct manipulation of individual Gaussians. While the method inspired by JacobiNeRF, it offers significant contributions by applying the approach to 3D Gaussian fields, and achieving robust performance in open-vocabulary segmentation and editing tasks. Through concerns about limited technical novelty and weaker demonstrations, after the discussions, reviewers agreed on the innovative application of mutual information shaping, and the qualitative improvements in scene editing. I’d recommend the authors to consider all comments in the revision.