InfoGS: Efficient Structure-Aware 3D Gaussians via Lightweight Information Shaping
Authors: Yunchao Zhang
Guandao Yang
Leonidas Guibas
Yanchao Yang
Venue: ICLR 2025 Poster (2025)
Keywords: Mutual Information Maximization; 3D Reconstruction; 3D Editing
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: 3D Gaussians, as an explicit scene representation, typically involve thousands to millions of elements per scene. This makes it challenging to control the scene in ways that reflect the underlying semantics, where the number of independent entities is typically much smaller. Especially, if one wants to animate or edit objects in the scene, as this requires coordination among the many Gaussians involved in representing each object. To address this issue, we develop a mutual information shaping technique that enforces resonance and coordination between correlated Gaussians via a Gaussian attribute decoding network. Such correlations can be learned from putative 2D object masks in different views. By approximating the mutual information with the gradients concerning the network parameters, our method ensures consistency between scene elements and enables efficient scene editing by operating on network parameters rather than massive Gaussians. In particular, we develop an effective learning pipeline named ***InfoGS*** with lightweight optimization to shape the attribute decoding network ,while ensuring that the shaping (consistency) is maintained during continuous edits,  avoiding re-shaping after parameter changes. Notably,  our training only touches a small fraction of all Gaussians in the scene yet attains the desired correlated behavior according to the underlying scene structure. The proposed technique is evaluated on challenging scenes and demonstrates significant performance improvements in 3D object segmentation and promoting scene interactions, while inducing low computation and memory requirements. Our code is available at: https://github.com/StylesZhang/InfoGS.

Published as a conference paper at ICLR 2025
InfoGS: EFFICIENT STRUCTURE-AWARE 3D GAUSSIANS VIA LIGHTWEIGHT INFORMATION SHAPING
Yunchao Zhang!
Guandao Yang²
Leonidas Guibas
Yanchao Yang'
The University of Hong Kong
2 Stanford University
ABSTRACT
3D Gaussians, as an explicit scene representation, typically involve thousands to millions of elements per scene. This makes it challenging to control the scene in ways that reflect the underlying semantics, where the number of independent enti- ties is typically much smaller. Expecially, if one wants animate or edit objects in the scene, as this requires coordination among the many Gaussians involved in repre- senting each object. To address this issue, we develop mutual information shaping technique that enforces resonance and coordination between correlated Gaussians via Gaussian attribute decoding network. Such correlations can be learned from putative 2D object masks in different views. By approximating the mutual infor- mation with the gradients concerning the network parameters, our method ensures consistency between scene elements and enables efficient scene editing by operat- ing on network parameters rather than massive Gaussians. In particular, we develop effective learning pipeline named InfOGS with lightweight optimization to shape the attribute decoding network, while ensuring that the shaping (consistency) is maintained during continuous edits, avoiding re-shaping after parameter changes. Notably, our training only touches small fraction of all Gaussians in the scene yet attains the desired correlated behavior according to the underlying scene struc- ture. The proposed technique is evaluated on challenging scenes and demonstrates significant performance improvements in 3D object segmentation and promoting scene interactions, while inducing low computation and memory requirements. Our code is available at: https: : //gi thub. wwwm/tt////e ee InffGG
INTRODUCTION
Open-world 3D scene understanding and editing play important roles in gaming, AR/VR, and robotics applications. Recently, 3D Gaussian Splatting (3DGS) (Kerbl et al. 2003), has significantly advanced both the rendering quality and inference efficiency in 3D learning by representing scenes through set of 3D Gaussians. Moreover, follow-up works in scene editing (Cen et al. 2023: Zhou et al.) 2023: Ye et al., 2023) propose augmenting each Gaussian's 's parameters with task-relevant attributes, allowing users edit scenes by selecting Gaussians based these attributes and modifying their colors or positions. Despite the progress with EDGGS-bsee scene editing, the use of point-wise 3D) Gaussians presents limitations, among which, the fact that the number of distinct and interactive entities in a scene is typically far fewer than the number of Gaussians is neglected. Furthermore, augmenting attributes of Gaussians introduce significant storage overhead and do not fundamentally address the lack of correlation between Gaussians.
The lack of intrinsic correlation among scene elements renders clear contrast between the editing of 3D Gaussians and the perturbation in the real world. For example, real-world scenes consist of particles that aggregate into cohesive objects through physical or chemical bonds, forming highly organized, interdependent structures. Interactions with small part of an object can induce changes throughout the entire entity due to these intrinsic correlations. Such phenomenon implies a high degree of mutual information between particles of the same entity, and even across distinct objects. Ideally, 3D scene representations should capture these correlation structures, facilitating efficient learning and interaction for tasks like scene editing. If modifying the state of single element could result in coherent adjustment in the states of all correlated elements, which promotes efficiency by eliminating the need to individually examine and edit each Gaussian.

---

Published as a conference paper at ICLR 2025
+ IR Representation Manifold)
(1) No pa rameter changes
000
Perturb network parameters according to user-selecttd Goussian
(2) Perturb color branch
g(a)
Attribute Decoding ((:
OF 3D Gaussians
(3) Perturb opacity branch
Impur festures
Color
0 pacity
g(z)
Scale
Decode attributes Gaussians from features
Efficient scene editing by parameter perturbation
Our InfoGS is a mutual information shaping technique of the attribute decoding network based on 3D Gaussian Splatting (Kerbl et al. 2023). It can capture the underlying structure of the scene, while maintaining selected Gaussian (specified in Sec. 3.7). It promotes efficient scene editing by perturbing the network parameters, including re-colorization, segmentation, object removal, etc.
Figure 1:
correlations after consecutive parameter changes according to user-
To enable efficient object-level interactive tasks, propose the key is to enforce the inherent correlation structure within the scene representation, with which, the propagation of changes across correlated Gaussians be easily achieved. Instead of assigning additional attributes to each Gaussian for grouping purposes, we introduce a novel learning scheme that encodes the correlations into the representation and effective approach for manipulating scenes. Note that, few 3DGS- based compression methods have exploited similarities between Gaussians to reduce storage costs, such as clustering Gaussians with similar parameter values (Fan et al. 2023), introducing anchors to group nearby Gaussians in different spatial regions (Lu et al. (023), organizing 3D space with feature closeness (Chen et al. 2024). These techniques primarily target scene reconstruction and rendering, however, we propose an explicit modeling of the correlations from the perspective of mutual information maximization to enable efficient scene editing.
We develop upon the common practice of 3DGS compression methods, where Gaussian attributes (used for reconstruction) are decoded from their features, and we introduce a mutual information shaping, named InfoGS, scheme to explicitly enforce the correlations between Gaussians via the fcature-decoding network. This scheme ensures that perturbing the network parameters results in coherent changes among correlated Gaussians, enabling controllable and semantically meaningful interactions with objects in the rendered scene. Our strategy circumvents the need to estimate and encode pairwise feature correlations among millions of Gaussians. Instead, constructs well-organized tangent space that captures the underlying structure of the scene within the network.
Specifically, devise shaping loss that maximizes the mutual information (MJ) between Gaussians belonging to the same entity (as evidenced by putative metrics), which correlates Gaussians associated with the same object and de-correlates otherwise. Furthermore, mutual information between Gaus- sians is approximated by the cosine similarity between the Jacobians with respect to the perturbed weights. This legitimates the tangent space of the feature-decoding network parameters to encode the desired correlation structure. However, simply shaping the tangent space does not guarantee that the perturbed network will preserve this correlation structure when the shaped weights are modified, i.e., the after-modification parameters may not have shaped tangent space, thus, making it incapable of consistent perturbations across multiple editing operations. Another challenge lies in the computational and memory overhead required calculate Jacobians, as this necessitates storing the full computational graph regarding the derivative of the Jacobians a significant cost for 3DGS, where storage and training efficiency critical concerns.
2

---

Published as a conference paper at ICLR 2025
To further overcome these challenges, propose shaping the activations of the perturbed weights instead of the Jacobians. We prove that this technique ensures that the network undergoes a conformal transformation in its tangent space, maintaining the correlation structure and enabling consistent editing across sequential operations. In other words, the proposed is a sufficient condition for mutual information maximization while inducing the benefits not possessed by directly shaping Jacobians. Finally, we develop an efficient pipeline that shapes the feature-decoding network with consistency preserved along a sequence of perturbations, while being computationally efficient. This pipeline is fully automated and weight. We evaluate the proposed MI shaping technique across various sene-ediiing applications, including 3D object removal, inpainting, colorization, and scene recomposition.
In summary, 1) we propose a novel approach that leverages semantic correlations between 3D Gaussians for efficient scene editing; 2) We design an efficient training pipeline that shapes the scene representation once and ensures that the perturbed representation continues to support coherent edits without reshaping along the perturbation trajectory: 3) We demonstrate that our MI shaping technique affects only approximately ~ 7% of all Gaussians during training, while existing scene- editing methods require optimizing the entire set of Gaussians. Despite the lightweightness. method achieves significant performance gains tasks such as 3D segmentation, object removal, and inpainting, while substantially reducing computational and memory overhead.
2 RELATED WORKS
Neural representations for 3D reconstruction Implicit neural representations have recently transformed the field of 3D reconstruction, such as Neural Radiance Fields (NeRF) (Mildenhall et al.
2021) and its extensions (Liu et al. 2020 Karras et al.] 2021 Martin-Bruat Brualla et al. 2021 Barron et al. 2022 Deng et al.) [022] Zhang et al. 2023). which utilize neural networks to parameterize 3D
scenes, enabling detailed and accurate reconstructions. Subsequent advancements have focused on optimizing the NeRF framework for large-scale scenes. Techniques such as multi-resolution hash tables and voxel grids have been proposed to compress scene representations (Garbin al. 202T Reiser et al. [ Müller et al. 2022). Additionally, hybrid approaches have incorporated explicit
modeling strategies, leveraging point clouds to enhance scene representation (Fridovich- Keil et 2022 Xu et al. 1022,Kerhh et al. 2023). Notably, 3D Gaussian Splatting (3DGS) (Kerbl et al. 2023)
employs point-wise Gaussians achieve efficient training and rendering. These 3D representations usefuul for various downstream tasks, such semantic understanding (Zhi et al. 2021 Gao et al.,
2022
Bao et al. [2023] Wang et al. 2023b [i et al. 2024), segmentation (Fu et al. [002 Ye et al.
2023
Cen al. 2023), and 3D content creation (Tang et al. 2023 Chen 2003b). Nevertheless,
current methods predominantly rely on first-order supervision, often neglecting the higher-order
correlations encoded in the scene representation.
Scene representations for 3D editing Despite advancements in reconstruction, interactive 3D scene editing remains challenging. Current neural representation-baat approaches (Schwarz et al. 2020 Wang et al., 2022, Yuan et al. 2022, Wang et al.) 2023a Zhou et al.) 2023 Chen et al. 2023a typically rely on attributes computed by distilling features from 2D foundation models and perform editing in a point-wise manner. In contrast, our work delves into manipulating scenes based on a second-order correlation structure. The idea of enforcing second-order correlation first proposed in JacobiNeRF (Xu et al. 2023), while it mainly focuses on label propagation and one-time perturbation consistency, limiting its potential for diverse scene manipulations. Furthermore, the correlation shaping in Xu et al. (2023) is both memory- and time-intensive during training. Later, InfoNorm (Wang et al. (0224) has shown that enforcing second-order correlation also facilitates geometric reconstruction from few views. Other approaches involve using physies simulators to guide scene editing (Li et al. 2023) Xie et al.) (023). yet incurring a high computational cost in solving partial differential equations at inference time. Our approach offers efficient optimization and enables fast scene editing during inference by learning continuously well-shaped tangent spaces.
3 INFOGS: MODELING MUTUAL INFORMATION BETWEEN 3D GAUSSIANS
Our goal is to develop scene representations that not only capture 3D geometry and appearance with high precision but also encode the interrelationships between individual elements. These enriched
3

---

Published as a conference paper at ICLR 2025
representations enable efficient scene editing by ensuring coherent and realistic updates, where changes propagate in a coordinated and intuitive manner.
Building on 3D Gaussian Splatting (3DGS) (Kerbl et al.] 2023). state-of-the-art technique for 3D reconstruction that represents scenes using independent sets of 3D Gaussians, we aim to introduce meaningful correlations between these Gaussians. To achieve this, incorporate mutual information (MI) to capture their underlying interdependence. However, traditional MI-based methods involve heavy optimization processes and are typically applied point-wise, meaning they are valid only for the current representation. This renders them unsuitable for scene editing, which requires both real-time response and persistent editability.
To overcome this limitation, we propose a novel training approach that efficiently shapes the attribute decoding network to capture and maintain these correlations throughout the decoding process. Notably, these structured correlations persist even after multiple consecutive scene edits. Finally, we demonstrate how these well-formed correlations improve a range of downstream scene editing tasks.
3.1 PRELIMINARIES: 3D GAUSSIAN SPLATTING AND ATTIBUTE DECODING
Given dataset D consisting of multi-view 2D images and corresponding camera poses, 3D Gaussian Splatting (3DGS) reconstructs the 3D scene by learning a set of 3D Gaussians G = {gi, g2, ......... g.v } where N denotes the number of 3D Gaussians in the entire scene. Each 3D Gaussian gi possesses multiple attributes, denoted as {x,, St, qi, O C.). Specifically, x, E R represents the location of the centroid gi, 8; E IR denotes the scale, and q R denotes the rotational quaternion, such that and q, jointly represent the 3D covariance matrix E of g:- Additionally, og E IR represents the opacity, and o denotes the color, which is represented as the spherical harmonics (SH) coefficients. Given camera pose, the color (p) of pixel p can be computed by blending set of depth-ordered Gaussians N that overlap with p:
C(p) Σ cia; [1 LENN
j=1
where c; the color of the i-th Gaussian, and a; is the blending weight, calculated using the opacity, projected 2D covariance of the Gaussian, and the location of the pixel.
While 3DGS offers several advantages, its sparse and unorganized structure leads to significant storage demands and makes scalability challenging. To address this. mutual relationships between Gaussians have been explored in subsequent works to eliminate structural redundancies. Lu et al. (2023): ; Lee et al. (2024) proposed leveraging spatial relationships (i.e., features of Gaussians in local region tend to have similar values) by employing neural networks to decode their features:
a ag
+ (f(g)))
where a(g;) denotes attributes such as the color c; of the Gaussian g, The Gaussian attributes are decoded by attribute decoding network & with parameters 6, taking the feature of the Gaussian (g:) and the viewing direction as inputs. Incorporating the viewing direction into inputs improves the scene representation's 's robustness to substantial viewpoint changes and lighting effects.
Although associating millions of Gaussians via a network has proven effective, we argue that deeper exploration of the correlation structure of Gaussians through mutual information can further enhance scene editing, enabling better coordination among Gaussians for more effective results.
3.2 MUTUAL INFORMATION BETWERN 3D GAUSSIANS
Traditional 3DGS losses predominantly focus on first-order supervision, optimizing for view synthesis rather than scene interaction. In contrast, this work introduces second-order supervision to shape the attribute decoding multilayer perceptron (MLP), facilitating efficient scene editing. Specifically, we optimize the network parameters to induce global structure that encodes correlations among individual Gaussians. This ensures that modifications to the attribute decoding network's 's parameters, denoted by 6, directly correspond to meaningful scene-editing operations. As result, network outputs produce coherent and semantically consistent editing effects in the rendered scene, driven by the learned co-variation correlations.
4

---

Published as a conference paper at ICLR 2025
To formalize this, study the correlations between Gaussians under representation parameter changes in the output attributes. We analyze this behavior by considering two randomly selected Gaussians, 8i and 8j. and their corresponding offsets in output attributes, å(g:) and á(g)). These attribute changes arise from the perturbed network Fa(:00 + n), where E RR represents perturbation applied to the parameter set 0P.
Following Xu al. (023), we show that the mutual information between the attribute variations a(g;) and ((g)) is proportional to the cosine similarity of the Jacobians of o. with respect to the perturbed parameters 0p:
[(((g))) a())) = log
+ const.,
(3)
/1 cos
where denotes the angle between the Jacobians ( and (f(,)(( For clarity, d0p we denote the Jacobians as 0D;; and ; This formulation reveals that the mutual information I(ā(g:), ((),) is positively correlated with the cosine similarity of their gradients with respect to the perturbed parameters.
80p
We apply Eq. to evaluate whether the correla- tions between Gaussians are well-structured by the attribute decoding network 2.. As shown in
owe in
Fig. 2 (a), network weights optimized solely by the reconstruction loss fail to reveal meaningful correlations between Gaussians, primarily due to the unstructured nature of 3DGS. Despite the association of millions of Gaussians via the de- coding network, they remain disorganized limitation of methods optimized solely for novel view synthesis.
200
Perturtation Figure Perturb the attribute decoding network by the Jacobian of Gaussian in the bulldozer without (a) or with (b) MI shaping (Xu et al.) (023), and then move Gaussians according to similarities of Jacobian with selected one.
No Perturbatin
1 Perturaation
JacobiNeRF (Xu et al. 2023) addresses this by enforcing perturbation consistency through max- imizing the quantity in Eq. 3. enabling label propagation tasks. However, this approach is not suitable for scene editing in the 3DGS framework for two main reasons: (1) It does not support consecutive scene editing. As shown in Fig. (b), previous MI shaping fails after multiple editing operations, leading unrealistic scene updates. This is because the shaping in Xu et al. (2023) is point-wise: while 0Φ; and 0Φ; may be similar in the original tangent space, there
guarantee that they remain similar after parameter perturbation. (2) The optimization process is complex, requiring substantial storage to maintain the computational graph and significant time for Jacobian computation. This approach is inefficient for 3DGS and fails to meet the real-time demands of user-driven scene interactions.
Next, will discuss how ensure consistency across consecutive parameter changes while main- taining training efficiency.
3.3 MUTUAL INFORMATION SHAPING WITH CONSISTENCY AND EFFICIENCY
To ensure consistency in correlation shaping after perturbation, we begin by formulating the gradient expression for in-depth analysis. Given that attribute decoding networks in neural fields (c.g.. NeRFs and 3DGS) are MLPs, we focus perturbing the weights W of the I-th linear layer in the network 4. (i.e., A = W()), expressed as (!)) W ((-(h + b(0), Here, denotes the non-linear activation function, 5()) is the bias, and ( is the (hidden) output of the /-th layer.
To reveal the relationship between the gradient of a Gaussian g., 00;; (+((f(..d))))) and the d0
perturbed weight W we consider the gradient of an arbitrary scalar output z in the attribute decoding network with respect to W();
YY(h Y(W +(0))
(4)
5

---

Published as a conference paper at ICLR 2025
Unlabled Gaussians
(e) Coerse Mask Labeling
(b) Contrastive Loamnnn
Original
con'
cont
Space
Lobel
Supervise
3D Labels
MI Sheping
Shaped Space
Mask Association
Mao(pe)
2D Mask
com
Contribution
Rendering
The training pipeline of InfoGS: (a) We use SAM KKrilllv et al. 2023 to generate 2D masks and a pre-trained zero-shot tracker (Cheng et al. 2023) to associate masks from different views. We label the 3D mask of each Gaussian according to the 2D mask of the pixel that owns the Gaussian's 's maximal contribution during rendering across all views. (b) We use the labeled 3D masks as the supervision to conduct contrastive learning for mutual information shaping. After shaping, the Jacobians consistently distributed in the tangent space.
Figure 3:
where Y denotes all transformations after the I-th layer, and z is arbitrary scalar in the vector output of $. The differential dæ can be written as:
8z Oh()
Oz
Oz
dz =
dh
(h (-11
dw (r(o((11-1))
dw
(5)
Ohi
Oh
since dæ
0s Tway
dw()).
derive:
Oz Oz
OW ahe) (h (1 (T
(6)
By expressing c7s(7) in Eq. 3 using the right-hand side of Eq. 6 derive:
cos cos(O (0 (d)) = cas(@h))) (0)
(1)) d deN,
(7)
when ah and @h after the d-th perturbation, and oh represents the Jacobian aw of g: without perturbation. find that Oh corresponds repeated activations (1h(111)). A detailed derivation and the proof of Eq. 7 can be found in the Appendix C Eq. indicates that activation shaping can substitute for Jacobian calculation in Eq. to satisfy object-level mutual relations, i.e.. |[cos(>)]] approaches if g: and g, belong to the same object, or approaches 0 if not.
(0) point in the same or opposite directions. Here, ap represents the Jacobian (A
Upon further derivation,
The key insight of Eq. that by optimizing activations instead of Jacobians, the correlations between Gaussians can be correctly shaped and remain consistent after successive parameter changes. This enables versatile editing operations in the scene and makes training much more efficient, as there is no need to maintain the computational graph for Jacobian calculation.
3.4 SHAPING CORRELATIONS BETWEEN GAUSSIANS FOR SCENE EDITING
According to Eq. B and Eq. 7 ensure that two 3D Gaussians remain highly correlated consistently even after parameter changes thus supporting a sequence of editing operations without reshaping the parameters we shape their activations Oh within the attribute decoding network 4a. To enable meaningful scene editing, define positive and negative pairs based on object-level relationships, where (ge, 9 represents pair of Gaussians belonging to the same entity, and (gt, 9; are indepen- dent Gaussians. For positive pairs, we align their activations in & relative to the perturbed layer W(), while for negative pairs with minimal mutual information, we enforce their Oh to be orthogonal. To achieve this, we minimize the InfoNCE loss (van den Oord et al. 2018):
exp( cos(Oh; (i,,hhh;-1/// 1/T)
Lx log Σe Uti exp(1 cos(Sh, 0h4 -)/)))
(8)

---

Published as a conference paper at ICLR 2025
where T is the temperature parameter. This loss encourages highly correlated Gaussians to exhibit large activation similarity while ensuring that uncorrelated Gaussians exhibit near zero similarity.
3.5 SUPERVISION REFINEMENT
We now address how to sample positive and negative pairs from all 3D Gaussians. Identifying the label of each 3D Gaussian is challenging, 3DGS is trained from set of 2D posed images, complicating the recovery the joint 3D distribution. However, by leveraging the advances of 2D vision foundation models (Kirillov et al. 2023). we label coarse mask by lifting 2D mask 3D:
MaD(g:) M2D (arg max con?),
where conf represents the contribution of gi to pixel p during rendering, and Map and Mnn denote the 3D mask of Gaussians and the 2D mask of pixels. Eq. 9 indicates that the mask of 3D Gaussian is determined by the 2D mask of a pixel where contributes the most across all 2D) images. We compute the contribution con? based on trained 3DGS that contains geometric priors of the scene. Thus the mask of g; is likely to be determined by the 2D mask where its maximal contribution appears. In order to harmonize 2D mask IDs produced from different views, we use pre-trained zero-shot tracker (Cheng et al., (023) to associate 2D masks by treating them as video sequence. We also provide a version without using a tracker the experiments, and it is elaborated in the Appendix A
To further enhance efficiency, we introduce a smoothness regularization in the self-supervised training. Since objects are continuous space, neighboring Gaussians of gi are likely to belong to the same object gi. This observation helps shape the Jacobians of 3D Gaussians that are heavily occluded by others and may not be accurately annotated in the coarse 3D segmentation. Such Gaussians can be supervised by nearby, well-labeled Gaussians. Specifically, La is formulated as:
CR = Ep k
Σ =TEn(a)
1.0 cos( p., px+)], )},
(10)
where P denotes the input space, n²x) represents the neighboring Gaussians of and l: = ||[[)]] is the number of neighbors. practice, the distance between two Gaussians is defined by the Euclidean distance between their centroids x, and h is a fixed hyperparameter during training.
3.6 FULL PIPELINE INFOGS
Fig. 3 illustrates our training pipeline of InfoGS for the correlation shaping, and the loss is as follows:
C = LADGS + Am CMI + X,
(11)
where L3DGS is the photometric reconstruction loss from the original 3DGS pipeline, LMT is the contrastive learning loss (Eq. 8), and CR is the regularization loss, with A, representing its corresponding weight.
3.7 CORRELATION-BASEE SCENE EDITING
After shaping the correlations, scene editing is performed by perturbing the parameters of the attribute decoding network. Specifically, we perturb the attribute decoding network by applying the average Jacobian 0Φ. of user-selected Gaussians. When editing through 2D image rather than directly in 3D space, we select the Gaussians that have the largest influence on the color of the sele-selected pixels, similar to the contribution calculation in Eq.
Different branches of the network produce distinct editing effects. For object removal, we perturb the opacity branch & in the gradient direction that reduces output opacity. This can drive the opacity of the relevant Gaussians to zero, effectively removing the object in the rendered scene. For object re-colorization, perturb the color branch 4. along different gradient directions of output channels to generate diverse color changes. In the case of 3D segmentation and object movement, we move Gaussians according similarities in Jacobian with the selected one. When extending to dynamic scenes (Pumarola et al. 2020), we achieve object movement by shaping and perturbing the motion network. Additional editing tasks discussed in the Appendix
7

---

Published as a conference paper at ICLR 2025
Prompt Toy Chair
P fromp:
Plastic Box
Visuslizatio
Gus sian Grouping
InfoGS
Figure 4: Top: qualitative results of open vocabulary segmentation Ye et al. 2023). Bottom: gallery of relevance maps when perturbing a single Gaussian (highlighted in the purple circle).
4 EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
Training. We shape mutual information between Gaussians starting from pre-trained 3D Gaussian Splatting, applying fine-tuning stage for optimizing attribute decoding network $ We follow the shaping pipeline in Fig. for 1500 iterations. The finetuning stage is conducted 3090 GPU for about minute. Detailed hyper-parameter settings can be found the Appendix A
single RTX
Datasets. We test InfoGS on open-world datasets LERF-Localizainn (Kerr al., 2023 derived LFRF-Mask dataset (Ye et al. (023) with ground-truth segmentation, and Mip-NeRF- 360 (Barron al. 2022). in order evaluate the segmentation and editing quality in complex and compositional scenarios. We also test the effectiveness of our shaping on the real dynamic scenes in the D-NeRF
dataset (Pumarola et al. 2021). Additionally, provide results on the outdoor unbounded scenes in dataset NERDS 360 (Ishad et al. 2023) the Appendix
4.2 3D SEGMENTATION
After shaping the correlations among Gaussians, approach enables 3D segmentation by simply perturbing the network parameters. use Grounding DINO (Liu et al. 2023) to generate the 2D mask prompt. We compare InfoGS with the state-of-the of-the--rttttttt open vocabulary 3D segmentation methods, including both the NeRF-based (Kerr et al. 2023 Cen et al. 2023). and 3DGS based methods (Xu et al. 2023 Ye et al., 2023). It's worth noting that Gaussian Grouping (Ye al. 2023) use the consistent 2D masks generated from the video tracker. To evaluate the effectiveness of our improvement correlation shaping, we utilize the MI shaping loss JacobiNeRR (Xu al. 2023) as a substitute for our MI shaping LMT in Eq. [I] which only permits one-time perturbation consistency. While keeping other loss terms and supervision refinement the same for fair comparison, we name the baseline JaocbiGS. We also include the results of using foundation models.
For quantitative results, Tab. demonstrates that ours significantly outperforms both NeRF-based and GS based methods. Compared with previous state-of-the art results, InfoGS increases the mloU by averagely 11%. Qualitative results are shown Fig. We can see that InfoGS produces sharp boundaries for the segmentation without fewer blurs. We also provide the relevance map when perturbing single Gaussian to select the object. The relevance scores are represented as the cosine similarities of the Jacobians O of the perturbed Gaussian and all other Gaussians. As in Fig. when perturbing single Gaussian, the corresponding object is highlighted, evidencing ours capability of segmentation from another viewpoint. It also shows that the relevance of Jacobian would be higher if a Gaussian is closer to the perturbed one. These phenomena comply with our expectations, indicating the effectiveness of the proposed MI shaping.
8

---

Published as a conference paper at ICLR 2025
Table I: Quantitative results of Open Vocabulary Segmentation LERF-Mask dataset. The time and memory costs mentioned refer to the total consumption of training and finetuning (if have). Comparisons with non-3DGS methods in this table are not feasible.
figurines
leutime
Average Training Cost Memory/M
mloU
mBlol
mloU
mBioU
mloU
mBloU
Time/Minutes
#GS Used
LERF Keer 2023
30.6
42.6
DEVA
neti e a
46.2
45.1
56.8
51.1
54,3
52.2
SA3D
24.9
23.8
7.4
42.5
39.2
JacobiGa XB el 4023
62.2
60.1
64.3
57.7
69.7
70.1
30.2
160.2
17%
Gaussian Grouping To 2023
69.7
67.9
77.0
68.7
71.7
66.1
55.2
757.2
100%
IaloSSS wio DEVA
72.5
70.3
79.1
6979
77.8
77.2
19.7
160.2
15%
InfoGS
80.6
77.2
80.6
70.1
84.5
78.7
16.3
160.2
7%
Rendered Image
Gaussian Grouping
InfoGS
Figure 5: object removal on bear kitchen scenes. Compared Gaussian Grouping, InfoGS removes the object with more fitting curve and less distortion of the irrelevant area.
The superior performance InfoGS can be attributed to the learned awareness the scene structure. Instead of directly assigning scene attributes each Gaussian (Cen al. 2023 Zhou al. 2023 Ye et et al.) 2023), we focus on shaping networks that act on the Gaussians in a way respecting the underlying structures. It helps lot for low-level representation like 3DGS when leveraged for interactive tasks like segmentation and editting. The shaped MLP is aware of the entire scene structure rather than individual Gaussians. Notably, we only sample about 7% of all the Gaussians during finetuning until convergence, boosting the training time while with little memory storage.
4.3
SCENE EDITING
3D Object removal. 3D object removal delete the target object without affecting the background or unrelated area. The space behind the target is unknown from training views and thus it would be noisy after deletion. According to Fig. InfoGS preserves more details of the unrelated area, and the contour of the area caused by the deletion fits the target object shape better. More explicitly, the noisy area resulted from InfoGS after removal in the bear scene accurately matches the rectangle shape of the pedestal of the bear statue. While both are supervised by SAM and DEVA, InfoGS separates the large object more clearly compared to Gaussian Grouping.
3D Object movement. We adopt the task of object movement to evaluate the editing quality under consecutive perturbations. single perturbation is to add the multiplication of the Jacobian 0&, of selected Gaussian fi and scaling factor a, for controlling the impact. As shown in Fig. 6 perturbing scenes with the same Gaussian reveals that InfoGS induces reasonable motion (c.g., the carton is lifted up) while preserving the uncorrelated objects. It exhibits path-consistent motion after consecutive perturbations. In contrast, scenes generated by JacobiGS diverge into unrealistic representations after the second and third perturbation. For example, consecutive perturbations the

---

Published as a conference paper at ICLR 2025
acobiGS
Plain 3DGS
InfoGS
Rendered
Pertrbed Glansian
Perturbation
3 Pertuitation
acobiGS
Dynamic 3DSS
InfeGs
Perturbed Gaussian
Rendered image
14 P erturbation
2r Perturbation
y1 Pertubationn
Figure 6: 3D object movement on Mip-NeRF 360 (Barron al. (022) and D-NeRF (Pumarola et al. 2020). Compared to the correlation shaping in [u et al. (2023), InfoGS enables consecutive editing (no need for reshaping) on both static and dynamic scenes, while maintaining consistency.
Table 2: Ablation study of correlation shaping,
Mip-NeRF 360
D-NeRF
Model
PSNRT
SSIMT
LPIPS]
PSNRT
SSIM
LPIPS
GS Baseline (Kerbl et al. 2023 Yang et al. 2023
28.69
0.870
0.182
41.0
0.995
0.009
InfoGS
28.72
0.842
0.208
39.7
0.993
0,012
carton result in the distortion of the door behind the box (see the red boxes for highlights in Fig. 6 The observations demonstrate that InfoGS supports continuous editing with consistency, thereby verifying the effectiveness the proposed shaping technique.
4.4 THE EFFECT ON THE RECONSTRUCTION QUALITY
We analyze how correlation shaping affects reconstruction quality. From Tab, framework has comparable performance to the baseline method on both static and dynamic scenes. There is small PSNR drop dynamics scenes, original deformation network in Yang et al.] (2023), which aims for 4D reconstruction, making it bit challenging shape all timesteps, and this subject future study.
see that our
simply integrate shaping loss into the
5 DISCUSSION
We attempt the challenge of shaping correlations between independent elements in 3D Gaussian Scene (3DGS) model to enable scene editing through network parameter perturbations. Our derivation demonstrates that enforeing gradient alignment within the network ensures the consistency of mutual information between Gaussians, even after successive parameter perturbations. To achieve this, we introduce an efficient training pipeline, incorporating a contrastive loss and a 2D-to-3ID sampling strategy. With the well-shaped correlation structure, we illustrate how the resonance between Gaussians facilitates 3D segmentation and automatically directs millions Gaussians to respond appropriately to accomplish diverse editing tasks. Despite this efficiency, our current pipeline has limitations when generating arbitrarily complex edits, particularly at fine grained level, such object deformation. In future work, aim to address this limitation by learning intrinsic object features and kinematics from large-scale video datasets, which would enhance its capabilities in more sophisticated scene manipulation tasks.
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGMENT
This work is supported by the Early Career Scheme of the Research Grants Council (grant # 27207224), the HKU-100 Award, donation from the Musketeers Foundation, the HKU Seed Fund for PI Research, and in part by the JC STEM Lab of Robotics for Soft Materials funded by The Hong Kong Jockey Club Charities Trust.
REFERENCES
Chong Bao, Yinda Zhang, Bangbang Yang. Tianxing Fan, Zesong Yang, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Sine: Semantic-driven image-based nerf editing with prior-guided editing field. 2023 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp- 20919- 20929, 2023. UtLhttts:: hetp://///ap...ma...mmmmmmm www.stmaantic..m lesemmnttcccooar...... wtt.oo/////oooooi/oooc 687450)
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman, Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5470-5479, 2022.
Richard Bronson. Matrix methods: An introduction. 1969.
URL https://api
Tim Brooks, Aleksander Holynski, and Alexei Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18392-18402, 2023.
Jiazhong Cen, Zanwei Zhou, Jiemin Fang, Wei Shen, Lingxi Xie, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, et al. Segment anything in 3d with nerfs. Advances in Neural Information Processing Systems, 36:25971-259999 2023.
Yihang Chen, Qianyi Wu, Jianfei Cai, Mehrtash Harandi, and Weiyao Lin. Hac: Hash-grid assisted context for 3d gaussian splatting compression. ArXiv, abs/2403. 14530, 2024. URL https
7/api. 7/api. semant iostmanncccchoo.o.o
Yiwen Chen, Zilong Chen, Chi Zhang, Feng Wang. Xiaofeng Yang, Yikai Wang, Zhongang Cai, Lei Yang. Huaping Liu, and Guosheng Lin. Gaussianeditor: Swift and controllable 3d editing with gaussian splatting. ArXiv, abs/2311 1..222211..4,,,, 14521, 2023a. URL http::ttp::/aat www.::/..ssaaaninco..ooooo
org/Corpus ID: 2664453555
Zilong Chen, Feng Wang, and Huaping Liu. Text-to-3d using gaussian splatting. ArXiv, abs/2309. 16585, 2023b. URL https:///ppt semant wichotccccoolll. aaoolla.oooo wtw.org//orossli/////eeee
Ho Kei Cheng, Scoung Wug Oh, Brian Price, Alexander Schwing, and Joon- Joon-Young Lee. Tracking anything with decoupled video segmentation. In ICCV, 2023.
Kangle Deng, Andrew Liu, Jun-Yan Zhu and Deva Ramanan. Depth-supervise nerf: Fewer views and faster training for free. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12882-1288 2022.
Zhiwen Fan, Kevin Wang, Kairun Wen, Zehao Zhu, Dejia Xu, and Zhangyang Wang. Lightgaussian: Unbounded 3d gaussian compression with 15x reduction and 200+ fps. ArXiv, abs/2311..1244, 2023. httlLttts://// http:://g.....aaaa....ooaoooooor
Sara Fridovich-Keit Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 5501-5510, 2022.
Xiao Fu. Shangzhan Zhang, Tianrum Chen, Yichong Lu, Lanyun Zhu, Xiaowei Zhou, Andreas Geiger, and Yiyi Liao. Panoptic nerf: 3d-to-2d label transfer for panoptic urban scene segmentation. In 2022 International Conference on 31) Vision (3DV), pp. 1-11. IEFF, 2022.
11

---

Published as a conference paper at ICLR 2025
Xuan Gao, Chenglai Zhong, Jun Xiang, Yang Hong. Yudong Guo, and Juyong Zhang. Reconstructing personalized semantic facial nerf models from monocular video. ACM Transactions on Graph- ics (TOG), 41:1 12, 2022. URL httpps://app www.nticscholette htt.://o..a...oooooo.ss......
252846034 6034
Stephan J Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin. Fastnerf: High-fidelity neural rendering at 200fps. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14346-1438 2021.
Muhammad Zubair Irshad, Sergey Zakharov, Katherine Liu, Vitor Guizilini, Thomas Kollar, Adrien Gaidon, Zsolt Kira, and Rares Ambrus. Neo 360: Neural fields for sparse view synthesis of outdoor scenes. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 9187-9198, 2023.
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Advances in neural information processing systems, 34:852 143352-863,,20 2021.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), 2023.
Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. International Conference on Computer Vision (ICCV), 2023.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 141.69,4,,.6980, 2014.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304.026433 2023.
Joo Chan Lee, Daniel Rho, Xiangyu Sun, Jong Hwan Ko, and Eunbyung Park. Compact 3d gaussian representation for radiance field. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 21719-21728, 2024.
Mingrui Li, Shuhong Liu, and Heng Zhou. Sgs-slam: Semantic gaussian splatting for neural dense slam. ArXiv, abs/2402.03246, 2024. URL htttsstts:///pppp ieeemnnticschh... org/
II:
Xuan Li, Yi-Ling Qiao, Peter Yichen Chen, Krishna Murthy Jatavallabhula, Ming Lin, Chenfanfu Jiang, and Chuang Gan. Pac-nerf: Physics augmented continuum neural radiance fields for agometry-agnostic system identification. ArXiv, abs/2303.05512 2023. URL pattps://app
semant wttp://oop/o/SSS022222..0
Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat- Chua, and Christian Theobalt. Neural sparse voxel fields. Advances in Neural Information Processing Systems, 33: 15551-15663, 2020.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.054499 2023.
Tao Lu, Mulin Yu, Linning Xu, Yuanbo Xiangli, Limin Wang, Dahua Lin, and Bo Dai. Scaffold- gs: Structured 3d gaussians for view-adaptive rendering. ArXiv, abs/2312.00109, 2023. URL :2:5555177:25555778888
https: /ppi www.:esssaaaiicchh..ccco www.aaa...........e......
Ricardo Martin-Brualls Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
7210-7219, 2021.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
12

---

Published as a conference paper at ICLR 2025
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG), 41(4): 1-15, 2022.
Onur Özyeşil, Vladislav Voroninski, Ronen Basri, and Amit Singer. A survey of structure from motion*. Acta Numerica, 26:305-364, 2017.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. 202/ IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 10313-10322, 2020. URL stt.:p:://app...eaniiii.
Sttg/CorpusI::2210 :2:
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10318-10327, 2021.
Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. 202/ IEEE/CVF International Conference on Computer Vision (ICCV), pp. 14315-14325, 2021. URL https: /appi.apppp////// semanticscholar. org/
Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. Advances in Neural Information Processing Systems, 33: 20154-20166, 2020.
Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong. Harshith Goka, Kiwoong Park, and Victor Lempitsky. Resolution- robust large mask inpainting with fourier convolutions. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2159-2159, 2022.
Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309. 16653, 2023.
Zachary Teed and Jia Deng. Raft: Recurrett all-pairs field transforms for optical flow. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16. pp. 402-419, Springer, 2020.
Cyrus Vachha and Ayaan Haque. Instruct-gs2gs: Editing 3d gaussian splats with instructions, 2024.
gssgssttttssssss.... tiub... io/
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. ArXiv, abs/ 1001100777.. 2018. URL hhtpp::t///at ieemanticscholart org/
CorpusTD:: ID:
Binglun Wang, Niladri Shekhar Dutt, and Niloy Jyoti Mitra. Proteusnerf: Fast lightweight nerf editing using 3d-aware image context. ArXiv, abs/2310.09965, 2023a. URL helphttts://aat
Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. Clip-nerf: Text-and-image driven manipulation of neural radiance fields. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3835-3844, 2022.
Ruibo Wang, Song Zhang, Ping Huang, Donghai Zhang, and Wei Yan. Semantic is enough:
Only semantic information for nerf reconstruction. 2023 IEEE International Conference Unmanned Systems (ICUS), pp. 906-912, 2023b. URL o1t.
orc//oorpusDD:2655440
Xulong Wang, Siyan Dong, Youyi Zheng, and Yanchao Yang. Infonorm: Mutual information shaping of normals for sparse-view reconstruction. In European Conference on Computer Vision, pp.
242-258. Springer, 2024.
Tianyi Xie, Zeshum Zong, Yuxin Qiu, Xuan Li, Yutao Feng, Yin Yang, and Chenfanfu Jiang. Physgaus- sian: iinteissittegrated 3d gaussians for generative dynamics. arXiv preprint arXiv.2311.12198,
2023.
13

---

Published as a conference paper at ICLR 2025
Qiangeng Xu, Zexiang Xu, Julien Philip. Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann. Point-nerf: Point- based neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5438-5448, 2022.
Xiaomeng Xu, Yanchao Yang, Kaichun Mo, Boxiao Pan, Li Yi, and Leonidas Guibas. Jacobinerf: Nerf shaping with mutual information gradients. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16498-16507, June 2023.
Ziyi Yang. Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. De- formable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. arXiv preprint arXiv:2309. 13101, 2023.
Mingqiao Ye, Martin Danelljan, Fisher Yu, and Lei Ke. Gaussian grouping: Segment and edit anything in 3d scenes. arXiv preprint 2rXiv:2312.0073: 20233
Yu-Jie Yuan, Yang-Tiah Tian Sun, Yu-Kun Lai, Yuewen Ma, Rongfei Jia, and Lin Gao. Nerf-editing: Geom- ctry editing of neural radiance fields. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18332-18343, 2022. URL hhttps://app semanticscholaa
Xiaoshuai Zhang, Abhijit Kundu, Thomas Funkhouser, Leonidas Guibas, Hao Su, and Kyle Genova. Nerflets: Local radiance fields for efficient structure-aware 3d scene representation from 2d supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8274-8275 8284, 2023.
Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J Davison. In-place scene la- belling and understanding with implicit scene representation. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15838-15847 2021.
Shijie Zhou, Haoran Chang, Sicheng Jiang, Zhiwen Fan, Zehao Zhu, Dejia Xu, Pradyumna Chari, Suya You, Zhangyang Wang. and Achuta Kadambi. Feature 3dgs: Supercharging gaussian splatting to enable distilled feature fields. arXiv preprint arXiv:2312.03203, 2023.
14

---

Published as a conference paper at ICLR 2025
ReLU
ReLU
Tanh
+1x128
128x32
32x3
Color:
ReLU
ReLU
Sigmoid
41x128
32x1
Pacitty
Inputs: Nx(32 9)
ReLU
ReLU
Sigmoid
41x128
128:32
32x3
ReLU
ReLU
Normalization
41x128
128x32
32x4
Quatemion: q
Figure 7: The structure of attribute decoding network. For each 3D Gaussian, we use different branches to predict attributes (opacity, color, scale, and quaternion). The inputs are features of Gaussians with view direction.
A IMPLEMENTATION DETAILS
Attribute decoding network. The attribute decoding network is composed of four different branches, including the color branch P... opacity branch Pu, and branches for the covariance & and L. They all implemented by MLPs with RELU activation. The detailed structure is illustrated in Fig. where the output activation of different branches is inspired from Lu et al. (202)). 'hee centroid xr of each Gaussian is initialized by the pre-computed point cloud from STM points (Ozyeşil et al. 2017) and separately optimized. The hidden dimension of each feature is 32 and initialized to be zero.
Training hyperparamters. During training, set the heerrarrameter' AMT 0.1, AR 0.1 and k = 5. We adopt the Adam optimizer Kingma & Ba (2014) when shaping the attribute decoding network, with learning rate of 0.01. The gradient that use to shape the motion network is 8A(0)
Oh aW of each branch by default, where is the linear layer before the final output layer. We train the network for 1500 iterations, sampling random view and batch of 512 3D Gaussians at each iteration to compute the training loss.
When training in the style of previous mutual information shaping in JacobiNeRF (Xu al. 2023) to produce the results of JacobiGS, we modify the sampling batch size from 512 to 48 due to the large memory consumption. This because the shaping method in JacobiNeRI requires the direct calculation of Jacobians 06, which necessitates the preservation of the entire computation graph.
Training without using the tracker for mask association. We use DEVA (Cheng al., 2023 to
associate the masks produced from SAM (Kirillvv et al. 2023) for training. However, the assumption of using the tracker to associate masks from different views that the multi-view training images could be treated as a video. It will not hold when training views sparse and random. In light of this, we introduce another shaping version without mask association, which is supervised by projecting the Jacobians to 2D for sampled view:
1-1
a (p) = Σ aΦ (1-ay).
(12)
iEN
j=1
where 0④;; the Jacobian of the i-th Gaussian and a; is the blending weight that is calculated with the opacity, projected 2D covariance of the Gaussian, and the location of the pixel. By projecting the Jacobians of 3D Gaussians to 2D space, we can now sample Jacobians of pixels instead of 3D Gaussians for contrastive learning. We sample a random view and 512 pixels of it for each iteration, supervised by the mask of this view which generated from SAM. We keep all other hyperparameters the same the version of mask association. Note that we still use Ōh to replace 0Φ for Jacobian shaping according to Eq.
Speed-up module. For the MI shaping version without mask association, it would be time-consuming when projecting Jacobians to the 2D plane as the dimension of the Jacobian is high. We then propose speed-up up module that modifies the rasterization process in the CUDA pipeline. As we only need to sample a small batch Jacobians of pixels at each iteration, we add a mask during the CUDA forward and backward process to avoid the calculation of non-sampled pixels for Jacobians projection.
15

---

Published as a conference paper at ICLR 2025
Figure 8: 3D object Inpainting our method. We use LAMA (Suvorov et al. 2022) inpaint the images after object removal, and then finetune Gaussians based on the inpainted 21 images.
Rendered
h
Figure 9: 3D object style transfer. We use Instruct-Pix2Pe (Brooks et al. 2023) to edit the 2D images (prompt: "Turn the bear into a polar bear"), and then finetune Gaussians based on the edited 2D images. We compare our method with Distuct-GSS(C) (Vachha & Haque 2024), which is also 3DGS-based style transfer method.
B ADDITIONAL EXPERIMENTAL RESULTS
B.1
SCENE EDITING
3D object inpainting. Based on the 3D object removal, we could further accomplish the task of inpainting. Inspired by Gaussian Grouping (Ye al. (2023), we first detect the noisy area after object removal through the mask association by DEVA (Cheng et al. (203), and then use LAMA (Suvoroy al. 2022 inpaint the noisy deletion area for each view. Finally, we finetune Gaussians with similar training process according to the 2D inpainted images for about 5 minutes to get the 3D inpainting results. Fig. & demonstrates the effectiveness of our method for 3D object inpainting.
3D object style transfer. By selecting an object (similar to object removal), we could also achieve 3D object style transfer. First, we select all correlated Gaussians of the object through 3D segmentation. Second, we freeze the parameters of other Gaussians and start finetuning. During finetuning, use the edited 2D ground truth images as supervision which are produced by Instruct-Pix2Pix (Brooks) et al. 2023), updating the original dataset. The pipeline is the same Instruct-GS2GS. During finetuning, We employ L1 loss inside the 2D mask of the selected object and LPIPS loss within the bounding box that encloses the mask. From Fig, we could that method significantly reduced the influence the area that not related to the targeted object. Setruutt-GSGG changes the hue of the whole figure and produce artifacts after editing.
16

---

Published as a conference paper at ICLR 2025
acobiGs
Single Pertursetion
plical Flow
Compositional Perturbe
Optical Flow
Figure 10: Visual comparison of multi-object editing between JacobiGS (Xu et al. 2023) and ours, where slightly perturb three different objects: apple, cup, and bag. We RAF (Teed & Deng 2020) to estimate the optical flow for visualizing the movement.
JacobiGS
Ours
Rendered Image
3 Perturbation
Figure 11: Qualitative results on NeRDS 360 IIshad et al. 2023). Left: Comparison of consecutive editions when moving the car. Right: The gallery of segmentation mask and relevance map from different views(when perturbing single Gaussian), obtained by our method.
Multi-object editing. To achieve edit multiple objects in the same scene, we simply add Jacobians of selected Gaussians of different objects to the network parameters. shown in Fig. TO JacobiGS has difficulty maintaining scene integrity when combining Jacobians of two or more objects for perturbations. For instance, the table nearby is also affected by the perturbation (see red box comparison of the right part in Fig. T0P. Conversely, our method effectively maintains robust correlation shaping, demonstrating superior performance preserving the scene under multi-object editing.
B.2 OUTDOOR SCENARIOS
To test the generalizability of our method, we employ our shaping method in NeRDS 360 (Irshad al. 2023), which an outdoor unbounded dataset comprising 75 unbounded and diverse scenes. From Fig Ia), we can clearly
that our method moves the car without creating any artifacts. JacobiGS fails to make the car move while changing the texture of the surrounding road. Fig. lT]b illustrated the ability to detect objects like cars in the 3D space, showing the potential of our method to be applied in practical scenarios.
B.3 ABLATION STUDIES
Robustness correlation shaping. Though the masks generated by SAM (Kirillov et al. 2023 with DEVA (Cheng et al. 2023) are associated from different views, this strategy still suffers from the multi-granularity across different views and 2D segmentation errors. Fig. 12 demonstrates that method resolves these problems small part of the object as the prompt, our method can produce the segmentation of the entire object, showing that Gaussians inside the object greatly associated.
shaping the correlations between 3D Gaussians. Given
17

---

Published as a conference paper at ICLR 2025
Rendered Image
Figure 12: Robustness of our method the multi- granularity and errors masks. When masks from supervision contain errors (e.g., missing legs of the chair) with multi-granularign (e.g., a small part of the tractor segmented in a certain view), our correlation shaping successfully addresses these problems and produces correct segmentation masks.
Quality of our 3D labels. We visualize the quality of 3D labels Fig 13 which are generated using the coarse mask labeling method from Eq 9 As illustrated, these 3D labels are consistent across multiple views (Fig 13 a)) and exhibit high quality (Fig 13 b)) without requiring any optimization.
Regularization loss. During finetuning the attribute decoding network, we introduce an extra regularization loss help shape the Jacobians. Here Fig. [4] shows the ablation study of this regularization loss. From the area, the area in the red box apparently illustrates the impact of the regularization loss. When perturbing Gaussian the tractor, the one with regularization loss can maintains the minimal affects to other objects, while the without it produces some artifacts the surroundings. demonstrates that the regularization loss can better shape the correlation structure of the tangent space of the motion network.
C PROOF OF DERIVATION
Given that the attribute decoding networks in neural fields (e.g., NeRFs and 3DGS) are MLPs, we focus on perturbing the weights W(O) of the 1-th linear layer the network &, (i.e., On W()). expressed as: (1) (10 (W + (()) Let denote the non-linear activation function, (()) be the bias, and (,(0) be the (hidden) output of the 1-th layer.
To expose the relationship between the gradient of Gaussian gi, 00;; (+( and the 0ep perturbed weight (!),, consider the gradient of arbitrary scalar output in the attribute decoding network with respect to W();
YY(h ((W (()))
(13)
18

---

Published as a conference paper at ICLR 2025
(a) Rendered images and corresponding 3D labels generated by cearse mask labeling
More labels generated by coarse mask labeling
Figure 13: (a) Without any optimization, we produce 3D labels that are consistent across different views. (b) More examples of our 3D labels.done
Randered Image
W/ Regularbation
w/ w/Raguaabbtoos
Figure 14: The impact regularization loss when moving the lego tractor.
where Y denotes all the transformations after 1-th layer, z denotes arbitrary scalar in the vector output of 4. Then the differential dæ could be written as:
Oz dh -(h(-1)) dz
Oz T dw :r(o(((-1))
de @h "w(!),, we could get that:
aw!)),
(14)
@h
@h())
as dz tr( DWW
az
dz 0z
o (h (11)T
(15)
aW
which is 2D matrix that has the same shape with the W Then the cosine similarity of Jacobians O between two Gaussians are:
dz(x))
o (h (1)11((()))
(xx)))
82(x,)
(h (xx)))
( (x.)
cos (8 04,))
|0 ||bb,,
(16)
8z(x;)
0z(x;)
(ar
(1 (xx:) (x)))
ah )h
10
|
19

---

Published as a conference paper at ICLR 2025
aWay could be formulated as a 3D tensor, which could be conceptualized as repeated collection of gradients and each gradient is the vector o (1) If (fh"l1)))) is well-shaped satisfying object-level mutual relations (i.e., the absolute value of cosine similarity of (hh(-1)) approaches if gi and gj belong the same object, or approaches O if not), then Ōh should also
because 0h
satisfy. In this case, if and are affiliated to different objects:
0z(x.)
0=(x;;
(o (1 (xx))
(x.)
(x
cos( (o, ab,)
|00, |1
0z(x.)
(=(x;)
(0)
(17)
(x+))
||00; [
0,
Eq. 17 indicates that if cos, ahj) = when 9i and 9j are affiliated to different objects, then cos O,,, )t)) = O. Meanwhile, o (A (-)) would not change since the input of & is the fixed (J(g), d). Thus cos( 0hg) 0 would be consistent during the perturbation. Therefore, could have the following lemma:
Lemma 1. (Orthgonnalit) After any number of perturbations, the orthogonality of Jacobians between two Gaussians 9i E GA and gj E GB that are affiliated to different objects (G + GB) would remain consistent:
cos(a (d) a ((t) cos (Oh; ,an))) co
(0) , ). N
(18)
In contrast, we have another lemma:
Lemma 2. (Similarity) After any number of perturbations, the similarity of Jacobians between two Gaussians 9i GA and gj E Gg that are affiliated to the same objects (G.A = GB) would remain consistent:
cos(Oe) (d) ( cos( ,on cos (0)
on (0)) ) ddNN
(19)
Proof of Lemma 2: Consider the set of Jacobians JA = (aP E GA} of all Gaussians in an arbitrary object GA. they all would be orthogonal to the Jacobians of Gaussians from other objects, according to Lemma 1 if oh is well-shaped. Thus, the representation space of JA would be very narrow. If the representation space of 0 is compact enough (e.g., the dimension of the representation space of 04 is identical to the number of objects in the scene), then the similarity cos (os() (d) {
1 when GA Gp. In this extreme case, the set of Jacobians from all the objects in the scene is group of canonical basis (Bronson] 1969) of the representation space of 04.
Combining Lemma and Lemma we could have the conclusion Eq.
20


Reviews:
[{"review_id": "OqVlwc1W4v", "reviewer": "Reviewer_WXPj", "summary": "This paper presents a method for correlating Gaussians that belong to the same semantic group. It leverages a network to decode feature Gaussians into attributes and approximates mutual information via gradients with respect to network parameters. This approach enables editing operations through adjustments to the network parameters rather than direct manipulation of numerous Gaussians. The mutual information between Gaussians is approximated by shaping activations from perturbed network weights.", "questions": "Please refer to the weaknesses section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Previous approaches embed semantic features directly within each Gaussian primitive, requiring extensive manipulation of individual Gaussians to edit scenes. This paper instead proposes encoding mutual information within an attribute decoding network, allowing scene edits to be made by directly adjusting the network parameters.", "weaknesses": "The paper is generally challenging to follow, with many points lacking clear explanations. For example, Equation 7 and Line 304 state that \"we find that ∂h corresponds to repeated activations σ(h(l−1)).\" However, this lacks sufficient context, as does Line 317, which mentions that \"shaping activations ∂h within the attribute decoding network Φa supports a sequence of editing operations.\" Further explanation on why shaping these activations enables multiple edits would improve clarity.\n\nThe proposed method uses an MLP network to model mutual information between Gaussians, where the Gaussians primarily serve to splat features onto images used as inputs for the MLP. The primary contribution appears to be an enhancement to the JacobiNeRF framework, rather than a significant innovation in the context of 3D Gaussian Splatting (3DGS).\n\nThe contributions of Section 3.5 on coarse mask guidance and Section 3.6 on smoothness regularization seem minor, primarily focused on refining supervision masks. These sections could potentially be merged, as similar techniques have been explored in prior work.\n\nThe method depends on a 2D video mask tracker for grouping Gaussians. Based on the experimental results, it is difficult to determine if performance improvements arise from the mask tracker or the proposed approach. For instance, in Figure 4, the poor mask quality generated by Gaussian Grouping may stem from suboptimal 2D masks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1ARt6cVQb0", "reviewer": "Reviewer_fQfp", "summary": "This paper proposes information shaping for the 3D Gaussian field, following NeRF-based work JacobiNeRF. By contrastive learning between Gaussian primitives, the resulting Gaussian primitives on the same entities have a strong correlation while primitives on different entities have a weak correlation. The technical part follows the basic idea of JacobiNeRF and is tailored to 3D Gaussian with a attributes decoding network.  The learned Gaussian field is easier to edit, such as object removal and object movement.", "questions": "Does the decoding network reduce the rendering efficiency?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "- This paper adapts the basic idea of JacobiNeRF to 3DGS and makes it work in terms of scene editing.\n- The method achieves good performance in the open vocabulary segmentation task.\n- The application of object removal seems have good qualitative performance.", "weaknesses": "- The technical novelty is limited. The Jacobian-based mutual information learning is the core part of this paper, which mainly follows JacobiNeRF. The authors are suggested to emphasize the unique designs that are different from the JacobiNeRF and demonstrate the effectiveness of the unique designs. The mask guidance and smoothness regularization differ from the DINO adopted in JacobiNeRF. However, these components seem to be incremental techniques, and their effectiveness is not well validated in the experiments.\n- The writing should be improved. Many concepts lack a clear definition. For example, what are the exact definitions of perturbation and the sequential of perturbations (e.g., 1st/2nd/3rd perturbation). What are the exact parameters perturbated and how are they perturbated (i.e., perturbation direction and norm).\n- The demonstration is weak. The object movement in Figure 6 is not impressive and it seems to be a sample handpicked with effort. It would be better to present more object movement demonstrations, using the main components of a 3D scene, such as the earthmover toy in the kitchen scene. Can you move the earthmover to anywhere else on the desk? Furthermore, the demonstrations have low diversity, mainly adopting the scene of kitchen and bear.\n- The method outperforms JacobiNeRF and JacobiGS. However, the authors do not show the reasons via an apple-to-apple comparison with JacobiNeRF and JacobiGS. As I suggested in the first item, the authors are encouraged to show the performance roadmap from the JacobiGS baseline (or other reasonable baseline) and add the unique components one by one, offering a better understanding of the inner workings to readers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z40JZdbiIW", "reviewer": "Reviewer_fawD", "summary": "This paper introduces a mutual information technique based on 3D Gaussian splatting. Building upon JacobiNeRF, the authors present a more efficient training pipeline that optimizes activations instead of Jacobians. This approach ensures that the correlations between Gaussians are correctly shaped and remain consistent through successive parameter changes. Experiments on open-vocabulary segmentation and scene editing demonstrate the efficiency and significant capabilities of the proposed method.", "questions": "1. Aside from JacobiGS in the paper, which utilizes the same mutual information shaping loss as JacobiNeRF, why was the original JacobiNeRF not included as a baseline for comparison?\n\n\n2. Regarding weakness 2, is it possible for the proposed method to edit the color of an object to a textured color? Additionally, for the object movement task, can the method modify the trajectory of a selected object to follow a position-related trajectory, such as rotation?\n\n\n3. Besides open-vocabulary segmentation tasks, would it be feasible to conduct experiments on semantic or instance segmentation tasks to further enhance the credibility of the findings?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and organized, the authors provide a thorough theoretical analysis of the proposed method.\n2. Experiments demonstrate that the proposed method outperforms other baselines while maintaining robustness to incorrect masks.", "weaknesses": "1. Compared to methods that edit the scene after obtaining the labels for each Gaussian, such as Gaussian Grouping, the direct manipulation of network parameters appears less flexible. For instance, while Gaussian Grouping can achieve style transfer, the proposed method seems limited to modifying the colors of selected Gaussians to be uniform.\n\n2. The authors conduct open-vocabulary segmentation in only three scenes, evaluating just one object per scene if I understand correctly. Including more experiments would enhance the robustness and credibility of the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)