GaussianBlock: Building Part-Aware Compositional and Editable 3D Scene by Primitives and Gaussians
Authors: Shuyi Jiang
Qihao Zhao
Hossein Rahmani
De Wen Soh
Jun Liu
Na Zhao
Venue: ICLR 2025 Poster (2025)
Keywords: 3D Decompostion
3D Reconstruction
3D Editing
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Recently, with the development of Neural Radiance Fields and Gaussian Splatting, 3D reconstruction techniques have achieved remarkably high fidelity. However, the latent representations learnt by these methods are highly entangled and lack interpretability. In this paper, we propose a novel part-aware compositional reconstruction method, called GaussianBlock, that enables semantically coherent and disentangled representations, allowing for precise and physical editing akin to building blocks, while simultaneously maintaining high fidelity.
Our GaussianBlock introduces a hybrid representation that leverages the advantages of both primitives, known for their flexible actionability and editability, and 3D Gaussians, which excel in reconstruction quality. Specifically, we achieve semantically coherent primitives through a novel attention-guided centering loss derived from 2D semantic priors, complemented by a dynamic splitting and fusion strategy. 
Furthermore, we utilize 3D Gaussians that hybridize with primitives to refine structural details and enhance fidelity. 
Additionally, a binding inheritance strategy is employed to strengthen and maintain the connection between the two. 
Our reconstructed scenes are evidenced to be disentangled, compositional, and compact across diverse benchmarks, enabling seamless, direct and precise editing while maintaining high quality.

Published as a conference paper at ICLR 2025
GAUSSIANBLOCK: BUILDING PART-AWARE COMPO- SITIONAL AND EDITABLE 3D SCENE BY PRIMITIVES AND GAUSSIANS
Shuyi Jiang', Qihao Zhao', Hossein Rahmani", De Wen Soh', Jun Liu', Na Zhao Singapore Univeristy of Technology and Design, Lancaster University shuyi jiangemymail. sutd. edu.
agihao.zhao, dewen_soke na eautdesuud.eee.. [h. rahmai,,
uc.uk
ABSTRACT
Recently, with the development of Neural Radiance Fields and Gaussian Splat- ting, 3D reconstruction techniques have achieved remarkably high fidelity. How- ever, the latent representations learned by these methods are highly entangled and lack interpretability. In this paper, we propose novel part-aware positional reconstruction method, called GaussianBlock, that enables semanti- cally coherent and disentangled representations, allowing for precise and phys- ical editing akin to building blocks, while simultaneously maintaining high fi- delity. Our GaussianBlock introduces hybrid representation that leverages the ad advantages of both primitives, known for their flexible actionability and editabil- ity, and 3D Gaussians, which excel in reconstruction quality. Specifically, we achieve semantically coherent primitives through novel attention-guider center- ing loss derived from 2D semantic priors, complemented by dynamic splitting and fusion strategy. Furthermore, we utilize 3D Gaussians that hybridize with primitives to refine structural details and enhance fidelity. Additionally, a bind- ing inheritance strategy is employed to strengthen and maintain the connection between the two. Our reconstructed scenes are evidenced to be disentangled. compositional, and compact across diverse benchmarks, enabling seamless, di- rect and precise editing while maintaining high quality. The code is available at
INTRODUCTION
Recently, with the development of Neural Radiance Fields (NeRF) (Barron al. 2021 Mildenhall let al. (02I) and Gaussian Splatting (Jiang et al.) 2023 Kerbl et al. (023) approaches, multi-view 3D reconstruction techniques have substantially progressed, enabling the recovery of high-quality 3D assets from multi-view 2D images. Despite impressive progress, the underlying representations learned by these methods are highly entangled and lack interpretability. This entanglement nature not only discourages the understanding and analysis of the model, but also hinders precise editing of the reconstructed assets, as is difficult to accurately locate the required corresponding rep- resentations and apply precise operations while ensuring other representations remain unaffected. Although advanced 3D editing methods can serve as post-processing tools, e.g., one of the current state-of-te tff-the-art methods, GaussianEditor (Chen et al., 2024), proposes semantic tracing constrain the modification region while keeping other parts unchanged, it remains challenging to achieve pre- cise control shown in Fig. In contrast, compositional reconstruction where disentangled 31D components are reconstructed from 2D image inputs allows for individual analysis, processing, and modification, making it particularly appealing approach. This compositional reconstruction essential for enhancing 3D perception and understanding (Yuan et al. 2023 Ye et al. 2023). as well as for improving robotic interaction and manipulation capabilities (Vezzani et 2017).
A number of prior works have attempted to achieve part-aware compositional reconstruction from 2D inputs; however, several common limitations still exist. 1) Textureless or limited fidelity. Early
"Corresponding Author: zhao@sutd.cdia

---

Published as a conference paper at ICLR 2025
Multi-view Inputs
3D-GS
OmniSeg33
DBW
3D-GS
GS- -dS-dittrr
Our
Wave
Challenges:
"aave the (aft am of
Shake
F Fidelity Editability
X X
Nemmmti-&Coherent Parts
Figure 1: While most related works face at least one of the three common limitations, including fidelity, editability, and semantically coherent part-aware disentanglement Our method addresses all three limitations simultaneously. The underline text and masks are semtanic tracing keywords and results. Best viewed with color and marks.
classical works focus solely on extracting textureless part-aware shapes from images (Niu et al.
2018 Wu et al.
2020 Paschalidou et al. 2020: Michalkiewicz et al. [021 Paschalidou et al. 2021
Petrov et al.]
2023; Alaniz et al., 2023 Liu al. 2024). Although some recent works attempt to
jointly learn appearance through textured UV-mapp (Monnier et al. 2023) or neural features YYu al., 2024 Tertikas et al. 2023), the fidelity of the reconstructed scenes still lags significantly be-
hind current reconstruction methods utilizing Gaussian Splatting. 2) Semantic incoherence. One
line of compositional reconstruction works (Monnier et al. 2023] Alaniz et al.] 2023] Yu et al.) 2024) opts to use superquadrics Paschalidou et al. (2019); Liu et al. (2022). specific type of ge- ometric primitives to build a 3D scene due to their high interpretability and flexible actionability, which can be directly adapted into industrial software for intuitive physical editing. However, these methods typically focus merely on fitting "geometrically-suun suittd"" primitives into 3D scenes, while neglecting the semantic relationships and coherence between compositional parts. This often results in undesirable semantic overlap and unreasonable segmentation among reconstructed primitives. 3) Lack of controllable editablity. Another line of methods achieves compositional reconstruction through generalizable image segmentation models like SAM (Kirillov et al. (023). These methods utilize SAM to segment visual components in images and align the generated 2D masks with im- plicit representations (e.g., NeRF) (Bai et al. 2023 Ying et al. 2024| Cen et al. 2023b Tertikas
al. 2 or advanced explicit representations (e.g., Gaussians) (Ye et al. 2023 Cen et al. 2023a Dou et al. 2024 Ying et al.) (2024). Although these approaches yield satisfactory decomposed re- sults thanks impressive segmentation techniques, the reconstructed scenes only support "holistic" edits, such as translation and deletion, rather than allowing for controllable editablity. For instance, while it is feasible to uniformly remove or translate Gaussians within the same mask, more complex manipulations such waving arms or shaking heads, shown in Fig. [ require each Gaussian to undergo distinct transformations, a capability that these methods do not support.
These challenges prompt to propose novel reconstruction pipeline, named as GaussianBlock, where the semantically coherent and disentangled representation can be precisely and physically edited like building blocks, while simultaneously maintaining competitive high fidelity. Specifically, leveraging the flexible actionability of superquadric primitives and the capacity of 3D Gaussians in reconstructing high-quality scenes, we propose hybrid representation that integrates superquadric primitives and Gaussians, where superquadric serve as coarse but disentangled building blocks, while Gaussians act as "skin" refine the structure and ensure high fidelity.
Our GaussianBlock reconstructs semantically coherent primitives by introducing a novel Attention- guided Centering (AC) loss to regularize the generated superquadrics. Specifically, by using 2D differentiable prompts derived from superquadrics through dual ratteriattizza,on,, the AC loss applies a clustering algorithm to supervise the grouping of queries with the same semantics, encouraging the superquadrics to disentangle accordingly. Additionally, our GaussianBlock enhances the com- pactness and semantic coherence of superquadrics through the introduction of dynamic fusion and splitting strategies. Specifically, single superquadric that encompasses various distinguishable mantics considered for splitting, while multiple superquadrics that share the same semantics or exhibit significant overlap are fused.
The hybrid representation in our GaussianBlock is established by attaching and connecting Gaus- sians to the superquadries. Specifically, inspired by GaussianAvater (Qian et al.) 22224), initialize 3D Gaussians at the triangles of each reconstructed superquadric and perform primitive-base lo- calized transformations during rasterization reinforce their connection. Additionally, binding
2

---

Published as a conference paper at ICLR 2025
inheritance strategy is employed to maintain this connection, assigning each Gaussian new identity parameters that indicate the primitive and triangle to which it belongs. During optimization and den- sity control, our GaussianBlock utilizes regularization loss to encourage the Gaussians to remain closely connected to their corresponding primitives while ensuring that these identity parameters are inherited throughout the process. Based these designs, primitives are manipulated, each 31) Gaussian can be deleted, translated, and rotated according to its associated binding vertex.
Thanks to these innovative designs, our method demonstrates state-of-the-art part-level decompo- sition and controllable, precise editability, with competitive fidelity across various benchmarks. in- cluding DTU (Jensen et al. 20144) Nerfstudio (Tancik al. 2023), BlendedMVS Yao al., 2020). Mip-360- 360--Grden (Barron et al. 2021) and Tank&Templl Truck (Knapitsch et al. 2017).
2 RELATED WORK
Compositional Reconstruction. Compositional reconstruction refers to reconstructing disentan- gled 3D components from 2D image inputs, which can be individually analyzed, processed, and modified. Early classical works often focus on abstracting textureless shape parts from images, mostly through hierarchical unsupervised learning (Paschalidou et al. 20220) deep neural net-
work (Niu et al., 2018 Paschalidou al. 202T, Michalkiewicz al. 2021] Petrov et al.] 2023)
generative models (Wu et al., (020). contrastive learning (Liu et al. 2024) or primitives (Alaniz al.] 2023). Recently, there have been two major lines of work in compositional reconstruction:
one is primitive-based, and the other is segment-based with advanced representation.
In general, primitive-based methods typically fit "geometty-ssuiee"" primitives into the scene. For instance, Monnier et al. (Monnier et al., (223)) optimizes the superquadrics and corresponding tex- ture map concurrently, while Alaniz et al. (Alaniz et al.] 2023) iteratively feeds new superquadrics to an area with high reconstruction error. Recently, DPA-Net Yu et al. 2024) fuses convex quadrics assembly into NeRF framework to generate an occupancy field. However, existing primitive-based works neglect the semantic relationships and coherence between compositional parts, leading to undesirable semantic overlap and unreasonable segmentation among the reconstructed primitives.
With the ad vevent of large generalizable image segmentation models like SAM (Kirillov al. 2023), ),
new line of segmentation-be based methods that integrate with advanced 3D scene representations such as NeRF and Gaussian Splatting has gained popularity. PartNeRF (Tertikas al., 2023) learns inde- pendent NeRF for each pre-segmented part. CompoNeRF (Bai et al. (223)) segments complex text prompts and positions sub-texts within different bounding boxes, followed by the learning of a dis- tinct NeRF for each box. In contrast, SA3D (Cen et al. (023b) utilizes 3D NeRF as input instead of 2D images. It projects 2D pre-segmented masks onto 3D mask grids via density-guided inverse rendering, thereby achieving certain level of 3D disentanglement. GaussianGrouping (Ye et al.) 2023) aligns rendered Gaussians with their corresponding pre-segmented 2D masks and groups Gaussians with similar meanings to achieve disentanglement. Similarly,
[Ce cet al.) (2023a) achieves Gaussian decomposition and segmentation by enforcing the correspondence between Gaussian features and 2D pre-segmented masks. Omniseg3d (Ying et al. (202)) addresses the labor intensity required to ensure that 2D masks are multi view and fine- ame-ggained consistent by introducing 3D hierarchical contrastive learning. Despite the impressive segmentation and decom- position results achieved by the above methods, which rely on pre-segmented masks and advanced 3DD representations, the reconstructed 3D assets still lack flexible editability and actionability.
3D Editing. Precise editing is a challenging task, especially when dealing with the entangled 3D underlying representations of reconstructed scenes. Recently, numerous advanced 3D editing methods have been proposed, typically taking 3D scenes as input. These methods can function as post-processing techniques achieve visual editing effects relevant to our primary focus, i.e., compositional reconstruction. Therefore, we discuss several of these works here. Focaldreamer (Li al., 2024) takes 3D meshes as input and introduces geometry union and focal loss to achieve text-prompied precise editing. SKED (Mikaeili et al. 2023) requires 3D NeRF, sketches, and text prompt, where sketches guide the regions of the NeRF that should adhere to the specified modifica- tions. LENeRF (Hyung al., (023) also employs NeRF as input and incorporates several add-on modules, such as the latent residual mapper and attention field generation, to exccute text prompts that contain the desired local edits. GaussianEditoor (Chen et al. 2024), a leading method in the field, takes well-trained Gaussian scenes and text prompts as inputs, and enhances precise editing through Gaussian semantic tracing, which traces the editing target throughout the training process. Unlike
3

---

Published as a conference paper at ICLR 2025
these methods, approach enables intuitive "drag-mode physical and precise editing, similar to building blocks, by reconstructing actionable and semantic compositional representations.
3 METHOD
Given a set of multi-view images I and their corresponding silhouettes I", our goal is to recon- struct the compositional 3D scene using semantic coherent compositional primitives combined with Gaussians. The overall pipeline is illustrated in 2
In the first stage, we adopt parameterized superquadries primitives θg for their remarkable expres- siveness, which could be achieved with relatively small number of continuous parameters (Barr) 1981 Paschalidou et al. 2019p. In order to query from 2D pretrained segmentaion model and obtain derive the differential point Ps: and box prompts By from parameterized O through soft dual cuul-raaterization, as derived in Sec. 3.1 Subsequently, with the reference image and prompts, can extract the corresponding attention maps A, for each P& through pretrained prompt encoder PE, image encoder E and deconder D, and then apply clustering to Aş to derive a novel AC loss, described in Sec. Sec.B.2] where outlier vertices of the superquadrics are encouraged move toward the centroid (indicated as star in Fig. 2 Additionally, splitting and fusion strategy, explained in Sec. 3.3 is proposed to further strengthen the compactness and disentanglereent, when applying the AC loss and reconstruction loss between reconstructed image I' and silhouette I' during the supervised optimization. Through the first stage, semantic coherent part-aware superquadrics are obtained. In the second stage, to leverage the capacity of 3D Gaussians, Gaussians are initialized and bound to superquadric triangles with localized parameterization, and will globally mapped to world space during rasterization. Additionally, a regularization term pop will be applied together to enhance the connection between them.
semantic prior,
with Legh, as introduced in Sec. 3.4 4
3.1 DIFFERENTIAL RENDERING
Superquadric Parameterization. Following the classic work (Barr 1981 be parameterized as Øx (ax: TK, K, <k) where K, E R. E TR. IR3 s {$i, $2, $3} E 12, and e = 62] E g² denote the number superquadrics, transparency, 6D rotation vector, displacement vector, scaling factor, and continuous shape vector, respectively. Specifically, superquadrics can be formulated as the following explicit surface function (Barr 1981)
superquadrics can
I1
T
81 cos r cos w 82 sin" n)
T (2; O₂)
T2
T3
83 cas 17 ssin w
where x = =rot(Tk) )x tk.
Here 1) E [- 3l and w E [- are the spherical coordinates of the initial unit icosphere vertices, and rot(.) is rigid transformation (Zhou et al. 2019) mapping the 6-D rotation vector a rotation matrix.
Soft Dual Rasterization. In order to query the 2D pretrained segmentation model and obtain mantic priors, we propose soft dual rasterization, which means we rasterize both rendered images I' and 2D prompts Ps derived from the superquadries simultaneously.
To render the superqudaric, we adopt the occupancy function of (Chen et al. 2019 and attach the superquadric surface (x;; Q₂) with transparency information as (Monnier al. (023), which can be summarized as:
As)(m) (A,()) (Alw)))) 0))
Oş(u) a exp min
Given the pixel u and projected face j. (y( < indicates that pixel u is outside face j. and vice versa; serves as a scalar modeling the extent of the soft mask. Based on the occupancy function, we can associate L faces per pixel and obtain the rendered image I' through the classic alpha compositing (Porter & Duff 1984);
I'OL.)) = Σ
II (1 (1-0pp) pcl
O0.
(4)
1=1
4

---

Published as a conference paper at ICLR 2025
1. The first stage
Spliting
Fuilon
Ensader
(Mgg- Ou. ex?
Image
II. The second stage
Global Mapping
Eoyae
Dassity Cuntrol
The framework of our pipeline. In the first stage, the superquadries blocks are optimized guided by reconstruction loss Free and attention-guided centering loss LAC- With the point Px and bounding box B& prompt obtained from soft dual rasterization, the attention maps As from the last layer of pretrained decoder D are clustered, where outliers being encouraged to move towards the centroid. Meanwhile, superquadric splitting and fusion strategy is proposed to further enhance the semantic coherence and compactness during optimization. the second stage, Gaussians are bound to the triangles of superquadrics using localized parameterization and inheritance strategy duing optimization and densification with global mapping transform.
Figure 2:
Additionally, to obtain the reference points prompts as well, all superquadric vertices V can be projected to screen coordinate using the following transformation:
Mrj
Myi
(VV) = U MIn k=1
(5)
where Mr, and My; are projection and view matrices derived from camera's position and orien- tation (Ravi et al.] 2020), w is the resulting component from projection transformation, and M. accounts for the conversion from normalized device coordinates (NDC) space to screen coordinates. Unfortunately, such an aggressive transformation incorporates undesirable vertices, whose positions are occluded due to superquadric overlapping from various perspectives. Consequently, these invis- ible vertices do not "contribute to" the final rendered image, causing them too ambiguous to serve effectively as point prompts. To obtain valid point prompts, we randomly assign each superquadric unique representation color (C&lk E (1, K]], and write the vertices rasterization equation
Pk = (VV)) V& E
Er (
where VE denotes the k-th superquadric's vertices and {Cill E L} represents the color attribute associated with the (-th intersecting face. Vertices are considered as point prompts only when they meaningful, i.e., when the representative color of vertex is visible in the final outcome.
3.2 ATTENTION-GUIDEI CENTERING
Aiming for semantic coherence, we expect the superquadrics to be disentangled in alignment with the semantic parts the image In this work, we utilize SAM (Kirillov et al. (0233) to provide the semantic prior, given its advanced performance and generalizability. Based this semantic prior, we propose a novel Attention -uiided Centering (AC) loss to ensure alignment. The overall procedure for calculating the AC loss is summarized in Algorithm []
Firstly, to trigger the pre-trained SAM model, bounding boxes and point prompts are typically re- quired along with the input images for localization and indication. To be specific, for each

---

Published as a conference paper at ICLR 2025
Algorithm 1 Attention-guided Centering Loss
Input: set of bounding boxes B5 and 2D point prompts Pk. image embedding h(1) Output: the AC loss &g for superquadric 0%.
1: AA = Ø
2: for Pk.m E Pk: do
3:
kkmm - f(r) (ppp. Bx])
>> encode point and box prompt Dr get cross attention map
4: Ak.m = CrossAttention (1(), 1 (kmm)
5: Au f As U [Arm}
6: end for
7: S t HDBSCAN(AA)
cluster A
> find the major cluster St S 8: St arg maxx, ESa, Skk,a and M2 = {mm St}
9: Ag + M &.mmm (Pr(ñx,, mlm.ml Ab.m)
b weighted average among Sp localize the centroid of Sj define outliers >> calculate the AC loss
m arg mEM d( d(A Ai)
P Cp. (Prmml Ak.m E {t
12: l + Epaner d(pk,s: Pk.mi)
perquadric 0z. its rendering can be acquired through Eq. 5 as k.. and corresponding box prompt B₃ can be inferred from the top-left and top-right vertices of Ik- Thus, the prompt set of 0g be denoted as [[kk.. Bx]lPk.m E Pa.k E K}, and the embedding of each prompt (.()) is attained through the pre-trained prompt encoder f" in SAM. Similarly, the embeddings of the input image denoted as h()) are obtained through the image encoder E.
Subsequently, we retrieve the attention map Ag. prompted by (x) from the last cross attention layer of pretrained decoder D. The overall attention map set of Os. denoted as Ar. is assembled from all the prompts within {[pk, Bx]]- Since aim for each superquadric to be disentan- gled and independent, we expect only one semantic cluster to be identified each 0g- Hence, we leverage HDBSCAN (Campello et al., 2015) algorithm to cluster the sets of attention map Ag. Let S In = J.....N; N denote the cluster set associated with 0₁. and let Pr[n|As.m] denote the probability of assigning As,m to specific cluster Sk... The corresponding cluster la- bel for As,m, denoted as f..m. is determined by selecting the cluster with the highest probability: 7k.m = arg max,, Prin|As,m) The major cluster, denoted as S is defined as the largest cluster among Sk, where S arg ma, |S and its counterpart denoted the Cs. St Likewise, the centroid Pk,my represents the coordinate whose As.m is nearest the weighted average attention map A among St. while the outliers P\ those belonging Cs. St.
Finally, to promote disentanglement, outliers are motivated to move towards the centroid, which can be expressed mathematically as:
K
LAC (h, In
(PA
lui 4 = Σ
Ph.mm)
1=1
PE.EE
where d stands for the L2 Euclidean distance function.
3.3 DYNAMIC FUSION AND SPLITTING
To further enhance semantic coherence and compactness while preventing local minima, this section introduces dynamic superquadric splitting and fusion, taking semantic and compact constraints into account.
Splitting. We periodically check whether a single primitive contains multiple distinguishable se- manties. and for such primitives, we apply splitting operation. Moreover, due to the centering action motivated by Lac, superquadrics tend to cover toward containing single disentangled se- mantie, occasionally resulting in local minima. Inspired by (Kerbl al. (023), propose dynamic splitting algorithm tailored for superquadrics, informed by the attention- based centroids.
Based the cluster Sn, centroids maj can be localized following Algorithm and the distances ( i E [1, If the furthest distance between each pair of centroids can be denoted (D)

---

Published as a conference paper at ICLR 2025
D among Di, exceeds the threshold Tæ. which is set according diagonal of B₁ as BBk, then this superquadric 0g shall be split, anchored at 3D vertices of and ef.. In other words, two new superqudries {0_|2 [K + 1, K + 2}} will be initialized to replace 8k. whose rs, az, and ez are inherited from Oz. and 8z = 0.48k. Notably, to anchor the new superquadrics towards { l E [") l2 is initialized as Eq. such that their centers are positioned at e where Ve is the average position of 0x:
1- - ((V o 82)) ? rot((:)).
Finally, the opacity og is set to 0, indicating complete transparency.
Fusion. In addition to splitting, we enable dynamic superquadric fusion enhance the compactness. Assuming that superquadric 9g and Bg share the same semantic information, it is supposed that they should be fused together. Particularly, when only one cluster appears in the attention map union Ae U Ag across all camera views, a new 0 will be initialized to represent the fused one.
Similar to the splitting process, rK+1; tK+1, CK+1, 0K+1, and 8K+1 determined as the mean corresponding values of 0, and 0g. Since the initial fused outcomes are quite coarse, this leads to sudden increase in reconstruction error and associated gradients, which heightens the instability of the optimization and impacts convergence. Considering this, before proceeding with global opti- mization, we temporarily freeze the gradients of k and independently optimize 0k+1 1 for & steps to achieve a more precise fused geometry. Likewise, a and ag are ultimately set to zero
3.4 3D GAUSSIANS BINDING
To leverage the advantages of both the actionability of superquadric and the ability of 3D Gaussians to reproduce high-quality scenes, this section establishes, enhances, and maintains connections be- tween them.
Gaussian Binding and Inheritance. Our compositional superquadrics primarily focus on ob- jects within the silhouette. To represent the remaining space, following the approach outlined in (Monnier et al., 2023), we incorporate a centered icosphere primitive to model the background dome and a plane primitive represent the ground outside the reconstructed superquadries, as il- lustrated in Fig. 2 Subsequently, inspired by GaussianAvaters (Qian et al.) 2024). we initialize Gaussians G at each triangle face centers (T jk E K, m E M) of the superquadrics, defined as GREN = (j, q, 8, a. idg, ide). Here, N. M, and K represent the number of 3D Gaussians in the scene, the primitives, and the faces within the primitives, respectively. The parameters μ. q, a, and denote the Gaussian's position, quaternion, scaling factor, opacity, and spherical har- monic (SH) coefficient, respectively. Additional parameters id, and id, are introduced to specify the corresponding superquadric and face index to which each Gaussian is bound.
During rasterization, localized transformations are applied, where each Gaussian is transformed according the parameters of its bound triangle and superquadrics before rendering. Specifically, we map these properties into the global space by:
n' μ Pe: y'=r- Q(rr),
where μ4 is the world position of each triangle center, is concatenated by the direction vector of one the edges, the normal vector of the triangle, and their cross product, describing the orienta- tion of the triangle in global space as (Qian et al. 2024]; Q(-) transform the rotation vector to quaternions following the (Zhou et al. 2019).
Meanwhile, following the density control strategy in Gaussian Splatting (Kerbl et al. 2023), Gaus- sians shall also be split or cloned within the primitive-based local space, while inheriting their iden- tity parameters, idæ and ide, to the new Gaussians. Based on the aforementioned designs, as prim- itives undergo manipulation, each 3D Gaussian can be deleted, translated, and rotated according to its corresponding binding triangle.
Gaussian Positional Regularization. Furthermore, a fundamental assumption behind binding is that the Gaussian should roughly match the underlying primitives. For example, Gaussian repre- senting the body should not be rigged to a triangle on the cheek, as it may hinder disentanglement and subsequent editing work. Therefore, tives, we implement a simple yet effective regularization loss to encourage a compact relationship between the two throughout the optimization process,
enhance the connection between Gaussians and primi-
Lpos maz (1]m1|2, (pos) Cpos ||2-
(10)
7

---

Published as a conference paper at ICLR 2025
Here Cpos, set 0.5, is threshold that allows Gaussians certain degree of freedom in their position, enabling them to better refine the primitive-basee 3D structure.
3.5 OPTIMIZATION DETAILS
The first stage. The overall objective function for superquadric optimization is summarized as:
ffrret Cree CCAC,
(11)
where Free d(I,, F') is the reconstruction loss between rendering outcome I' and imput silhouette I; y is the weight of Cac. which equals to 0 during first 2k warm-up iterations. Additionally, splitting and fusion operations are performed every 1 iterations. Meanwhile, we automatically filter out nearly transparent superquadrics, whose a is less than 0.1 following (Monnier et al. 2023).
The second stage. The overall objective function for 3D Gaussians binding and optimization is summarized as:
Esecond = Lrgb + Eposr
(12)
where
Lrgb A)L1 + -LDM
(13)
with À = 0.2 following (erbl et al. 2023).
4 EXPERIMENT
The experimental setup is detailed in Sec 7.1 of the Appendix.
4.1 MAIN RESULTS
Primitives Reconstruction. Experiments are conducted on DTU scenes to compare the reconstruc- tion performance of our learnt primitives with SOTA 3D decomposition methods (Liu et al. 2022 Monnier et al. 2023). Notably, EMS (Liu et al.) 2022 is decomposition method that utilizes the ground truth point cloud during optimization, while DBW (Monnier et al., 2023) is the only pub- lished primitive-based reconstruction method from 2D images that considers both decomposition and texture.
Tab. D shows the CD comparison results for each scene of the DTU dataset, where reconstructed superquadric achieves the best performance among most scenes with the smallest Chamfer Distance.
Mothod
Input
S24
S31
$45
$59
563
S83
5105
CD
EMS eiu el
2022
3D GT
6.77
5.93
6.91
3.50
4.72
7.25
6.10
5.67
DBW Monnier et al 2023
Image
4.90
3.13
3,86
4.52
5.02
4.12
6.18
4.78
Image
4.21
3.35
3.82
Table 1: Performance Comparison of Primitives Reconstruction Quality on DTU Scenes. Our prim- itives achieve the best reconstruction quality in all scenes, despite relying only on 2D images.
Part-aware Semantic Coherent Decomposition. Fig. shows the results of our method, Om- niSeg3D (Ying et al. 2024) and DBW on the DTU (first 2 rows) and BlendedMVV (last 2 rows) datasets. OmniSeg3D is a current SOTA method for part segmentation, capable of generating 3D masks containing different semantic information (indicated by colors) from 2D inputs.
From the results, we observe that DBW exhibits unnatural decomposition and fails segment the scene into semantically coherent parts due to the lack of semantic guidance. For example, it incor- rectly represents both the feet and hands of the Smurf using the same superquadric. Additionally, without self- self-adaptive splitting strategy like ours, it tends to fall into local minima, as seen with the toy, where one ear not represented. For the same reason, DBW is sensitive to the initial blocks number K; in complex scenes like Gundam, it requires up to 50 initial blocks to achieve accept- able result. Compared to OmniSeg3D, our method achieves finer- grainer semantic disentanglement. For example, in the Smurf and Gundam scene, OmniSeg3D tends to assign single semantic mask to the entire instance, whereas our approach decomposes the scene into finer- fine-grained parts.
Precise Editablity. In Fig. 4 demonstrate the precise part-level editability of our method. For each data, we provide visualized results in three columns: part-aware decomposition, primitives editing (through MeshLab), and scene editing.
Reconstruction Fidelity. To evaluate the fidelity of our GaussianBlock, we compare it with one of the current of-th--rr reconstruction methods, Gaussian Splatting (Kerbl al. 2023). and
8

---

Published as a conference paper at ICLR 2025
Daraser
DTU PSNR?
BlendedMI'S
Truck
Garden PSNR
Method
SSIM
LPIPS.
SSIM
PSNR1
LPIPS]
SSIM
PSNRT
L.IPS.
SSIM
LPIPS.
3DGS
x
98.3
35.7
8.8
94,4
33.8
9.2
84.7
25.0
17.3
86.2
27.0
10.1
OnunSog
V
97.7
33.6
9.4
93.2
29.4
10.4
82.6
23A
18.0
85.8
25.9
13.0
DBW
x
71.3
19.6
26.5
49.2
16.4
41.0
Ours
/
94.3
30.7
10.3
90.8
28.8
11.2
81.3
22.0
20.7
84.8
23.1
16.9
Table 2: Quantitative Comparison on reconstruction fidelity.
Data
OmniSeg3D
DBW
Ours
Figure 3: Qualitative results. Our method demonstrates fine-grained, semantically coherent part- aware decomposition.
the Gaussian-based OmniSeg3D, primtive-based DBW in the Tab. 2 Although our method does achieve the same better fidelity as purely Gaussian-based approaches, has made significant improvements over the SOTA primitive-based DBW method, and has produced competitive results compared Gaussian- based approaches. However, the primary contribution of our method is not to propose a fidelity-oriented reconstruction algorithm. Instead, our method enables semantically coherent part-level decomposition and controllable actionability, with high but not the best fidelity.
4.2 ABLATION STUDY
Attention-guided Centering Loss. In Tab. we assess the importance of each key component of our model by sequentially removing them and evaluating the average performance across the DTU dataset. Additionally, the total number (#K) of effective superquadrics presented for reference. The #K can reflect the compactness of the decomposition performance; a common goal is to achieve the highest compactness (the smallest #K) while maintaining the primitives performance.
Overall, LAC and splitting strategy consistently improve the primitives quality through semantic guidance, while the fusion strategy successfully enhances the compactness decomposition with a minor quality trade-off. An intuitive visual comparison is shown in Fig. 5 (a). While the model without splitting aggressively uses a single superquadric to represent both the roof and the building, the model without fusion redundantly uses multiple superquadries represent the same part. This lack of compactness makes subsequent skeleton-basee editing less intuitive and meaningful.
Loss Weight. In Tab. and Fig. (b). assess the impact of the weight of LAc, denoted 7, on the DTU dataset. It is evident that with a lower 7, LAC is too weak to ensure that the superquadrics
Data
Editing
Data
Editing
Figure 4: Editing Results. Our reconstructed scenes can be edited seamlessly and precisely.

---

Published as a conference paper at ICLR 2025
Settings
Full
w/o splitting
w/o fusion
0.02
0.2
2
(b)
0.9
0.7
0.5
pos
1.5
1.0
0.5
0.2
(c) Figure 5: Ablation studies on (a): Experiment settings, (b) AC-Loss weight y, (c): Splitting thresh- old 8: (d): Regularization syper-parametel Cpos
(d)
can disentangle the semantic structural information in the input images. Conversely, when y is too high, the superquadrics tend to be small to 'hack' LAC, failing to meet reconstruction expectations. Thus, in this work, 7 is set to 0.2 by default.
Splitting Threshold. Additionally, given that the splitting threshold is 3B,, where B& is the cor- responding diagonal of superquadric 02 reconstruction and dynamically updated during the opti- mization process, the impact of the weight 33 needs to be evaluated. As shown in Fig. 5 (C), with higher 3. the splitting mechanism is less likely to be triggered. This results in the toy body and ears in (c) not being effectively separated into individual superquadries. Conversely, with lower threshold, the scene divided more fine-grained, as seen with the arms and paws. In conclusion, is used control the granularity of the deompositin and should be selected based on the desired level of granularity for subsequent editing operations, which is set to 0.7 by default.
Method
#K
CDL
SSIM
PSNR
LPIPSI
Method
#K
CD.
SSIMF
PSNR1
LPIPS_
0,02
6.2 4.12
84.5
27.2
12.4
Full model
5.8
3.69
94.3
30.7
10.3
7
0.2 (default)
5.8 3.69
94.3
30.7
10.3
wlo Lac
5.4 4.75
84.1
30.3
11.4
2
4,9 6,97
80.4
20.8
13.7
Spliting
4.24
88.6
25.7
16.5
0.5
9.1 3.56
90.4
31.7
10.1
0.7 (default)
5.8
3.69
94.3
30.7
10.3
w/o Pusion
9.2
3,46
92.5
31,0
10.1
0.9
4.4 4.85
84.3
32.9
14.8
Table 3: Ablation studies on key components.
Table 4: Ablation studies on 3s and ys.
Positional Regularization Threshold. We also examined the impact of different thresholds es in regularization in Fig. (d). Given c = 1.5, 1.0, 0.5, and 0.2, the PSNR of the reconstructed scenes are 35.4, 33.7, 32.8, 27.3, respectively. However, during the "delete left ear" editing operation, we observe that higher tends to perform better. This is because, with larger threshold, the connection between the primitives and Gaussians becomes weaker, leading to greater intertwinement between Gaussians. For instance, the Gaussian points on the toy's ear semantic component may become entangled with those in the head region, causing the deletion of the car to affect some undesired parts. On the other hand, when is too low, the fidelity suffers, stronger regularization constraints prevent the Gaussians from fully refining the primitives' structures.
5 CONCLUSION, LIMITATIONS AND FUTURE WORK
In this paper, we propose a novel part-aware compositional reconstruction method, called Gaussian- Block, which enables semantically coherent and disentangled representations, allowing for precise and physical editing akin to building blocks, while simultaneously maintaining high fidelity. How- ever, our work has certain limitations. Currently, we can only perform decomposition for object- centric entities based on their silhouette and are not yet able to handle backgrounds effectively. This is due to the fact that backgrounds typically have more complex and detailed structures with richer semantic information. In the future, beyond mid-level superquadrics, plan to introduce more diverse types of primitives, such as cuboids and curved surfaces, in combination with Gaussians.
6 ACKNOWLEDGMENT
This research is supported by the Ministry of Education, Singapore, under its MOE Academic Re- search Fund Tier SMU-SUTD Internal Research Grant (SMU-SUTD 2023 02 09), it is also supported by the Agency for Science, Technology and Research (A*STAR) under its MTC Pro- grammatic Funds (Grant No. M23L760021).
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Stephan Alaniz, Massimiliano Mancini, and Zeynep Akata. Iterative superquadric recomposition of 3d objects from multiple views. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 18013-18023, 2023.
Haotian Bai, Yuanhuiyi Lyu, Lutao Jiang, Sijia Li, Haonan Lu, Xiaodong Lin, and Lin Wang. Com- ponerf: Text-guided multi-object compositional nerf with editable 3d scene layout. arXiv preprint arXiv:2303. 13843, 2023.
Alan H Barr. Superquadries and angle-preserving transformations. IEEE Computer graphics and Applications, 1(01): 11-23, 1981.
Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Portin-Bruuall,, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5855-5864 2021.
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 5470-5479, 2022.
Ricardo JGB Campello, Davoud Moulavi, Arthur Zimek, and Jörg Sander. Hierarchical density es- timates for data clustering, visualization, and outlier detection, ACM Transactions on Knowledge Discovery from Data (TKDD), 10(1):1-51, 2015.
Jiazhong Cen, Jiemin Fang, Chen Yang, Lingxi Xie, Xiaopeng Zhang, Wei Shen, and Qi Tian. Segment any 3d gaussians. arXiv preprint arXiv: 12.0060, 2023a.
Jiazhong Cen, Zanwei Zhou, Jiemin Fang, Chen Yang. Wei Shen, Lingxi Xie, Dongsheng Jiang, Xiaopeng Zhang, and Qi Tian. Segment anything in 3d with nerfs. In NeurlPS, 2023b.
Wenzheng Chen, Huan Ling, Jun Gao, Edward Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler. Learning to predict 3d objects with interpolation- based differentiable renderer. Ad- vances in neural information processing systems, 32, 2019.
Yiwen Chen, Zilong Chen, Chi Zhang, Feng Wang. Xiaofeng Yang, Yikai Wang, Zhongang Cai, Lei Yang, Huaping Liu, and Guosheng Lin. Gaussianeditor: Swift and controllable 3d editing with gaussian splatting. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 21476-21485, 2024.
Bin Dou, Tianyu Zhang, Yongjia Ma. Zhaohui Wang, and Zejian Yuan. Cosseggaussians: Compact and swift scene segmenting 3d gaussians. arXiv preprint 1.Xii:2401..5922. 2024.
Junha Hyung, Sungwon Hwang, Daejin Kim, Hyunji Lee, and Jaegul Choo. Local 3d editing via 3d distillation of clip knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2023.
Rasmus Jensen, Anders Dahl, George Vogiatzis, Engil Tola, and Henrik Aanæs. Large scale multi- view stereopsis evaluation. In 2014 IEEE Conference on Computer Vision and Pattern Recogni- tion, pp. 406-413. IEEE, 2014.
Yingwenqi Jiang, Jiadong Tu, Yuan Liu, Xifeng Gao, Xiaoxiao Long, Wenping Wang, and Yuexin Ma. Gaussianshader: 3d gaussian splatting with shading functions for reflective surfaces. arXiv preprint arXiv:2311.17977 2023.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), 2023.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Yen Lo, et al. Segment anything. In Proceed- ings of the IEEF/CVF International Conference Computer Vision, pp. 4015-4026, 2023.
11

---

Published as a conference paper at ICLR 2025
Arno Knapitsch, Jaesik Park, Qian- Yi Zhou, and Vladlen Koltun. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Transactions on Graphics, 36(4), 2017.
Yuhan Li, Yishun Dou, Yue Shi, Yu Lei, Xuanhong Chen, Yi Zhang, Peng Zhou, and Bingbing Ni. Focaldreamer: Text-driven 3d editing via focal-fusion assembly. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 3279-3287, 2024.
Anran Liu, Cheng Lin, Yuan Liu, Xiaoxiao Long, Zhiyang Dou, Hao-Xiang Guo, Ping Luo, and Wenping Wang. Part123: Part-aware 3d reconstruction from a single-view image. ACM SIG- GRAPH 2024 Conference Papers, pp. 1-12, 2024.
Weixiao Liu, Yuwei Wu, Sipu Ruan, and Gregory S Chirikjian. Robust and accurate superquadric recovery: a probabilistic approach. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2676-2685, 2022.
Mateusz Michalkiewicz, Stavros Tsogkas, Sarah Parisot, Mahsa Baktashmotlagh, Anders Eriksson, and Eugene Belilovsky. Learning compositional shape priors for few-shot 3d reconstruction. arXiv preprint arXiv:2106.06440, 2021.
Aryan Mikaeili, Or Perel, Mehdi Safaee, Daniel Cohen-Or, and Ali Mahdavi-Amiri. Sked: Sketch- guided text-based 3d editing. In Proceedings of the IEEE/CVF International Conference on Com- puter Vision, pp. 14607-14619, 2023.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
Tom Monnier, Jake Austin, Angjoo Kanazawa, Alexei Efros, and Mathieu Aubry. Differentiable blocks world: Qualitative 3d decomposition by rendering primitives. Advances in Neural Infor- mation Processing Systems, 36:5791-5807,, 2023.
Chengjie Niu, Jun Li, and Kai Xu. Im2struct: Recovering 3d shape structure from single rgb image. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4521-4529, 2018
Despoina Paschalidou, Ali Osman Ulusoy, and Andreas Geiger. Superquadrics revisited: Learning 3d shape parsing beyond cuboids. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, pp. 10344-10353, 2019.
Despoina Paschalidou, Luc Van Gool, and Andreas Geiger. Learning unsupervised hierarchical part decomposition of 3d objects from a single rgb image. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 1060-1070, 2020.
Despoina Paschalidou, Angelos Katharopoulos, Andreas Geiger, and Sanja Fidler. Neural parts: Learning expressive 3d shape abstractions with invertible neural networks. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3204-3215, 2021.
Dmitry Petrov, Matheus Gadelha, Radomír Méch, and Evangelos Kalogerakis. Anise: Assembly- based neural implicit surface reconstruction. IEEE Transactions Visualization and Computer Graphics, 2023.
Thomas Porter and Tom Duff.
Compositing digital images. In Proceedings of the 11th annual conference on Computer graphics and interactive techniques, pp. 253-259, 1984.
Shenhan Qian, Tobias Kirschstein, Liam Schoneveld, Davide Davoli, Simon Giebenhain, and Matthias Nießner. Gaussianavatars: Photorealistic head avatars with rigged 3d gaussians. In Pro- ceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 20299- 20309, 2024.
Xuebin Qin, Hang Dai, Xiaobin Hu, Deng-Ping Fan, Ling Shao, and Luc Van Gool. Highly accurate dichotomous image segmentation. In ECCV, 2022.
12

---

Published as a conference paper at ICLR 2025
Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin John- son, and Georgia Gkioxari. Accelerating 3d deep learning with pytorchšd. arXiv preprint arXiv:2007.08501, 2020.
Matthew Tancik, Ethan Weber, Evonne Ng. Ruilong Li, Brent Yi, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, et al. Nerfstudio: A modular frame- work for neural radiance field development. In ACM SIGGRAPH 2023 Conference Proceedings, pp. 1-12, 2023.
Konstantinos
Tertikas, Despoina Paschalidou, Boxiao Pan, Jeong Joon Park, Mikaela Angelina Uy, Ioannis Emiris, Yannis Avrithis, and Leonidas Guibas. Generating part-aware editable 3d shapes without 3d supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4466-4478, 2023.
Giulia Vezzani, Ugo Pattacini, and Lorenzo Natale. A grasping approach based on superquadric models. In 2017 IEEE International Conference Robotics and Automation (ICRA). pp. 1579- 1586. IEEE, 2017.
Rundi Wu, Yixin Zhuang, Kai Xu, Hao Zhang, and Baoquan Chen. Pq-net: A generative part seq2seq network for 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 829-838, 2020.
Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan. Blendedmvs: A large-scale dataset for generalized multi-view stereo networks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 1790-1799, 2020.
Mingqiao Ye, Martin Danelljan, Fisher Yu, and Lei Ke. Gaussian grouping: Segment and edit anything in 3d scenes. arXiv preprint arXiv:2312.0077 2023.
Haiyang Ying, Yixuan Yin, Jinzhi Zhang, Fan Wang, Tao Yu, Ruqi Huang, and Lu Fang. Om- niseg3d: Omniversal 3d segmentation via hierarchical contrastive learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 20612-20622, 2024.
Fenggen Yu, Yimin Qian, Xu Zhang, Francisca Gil-Ureta, Brian Jackson, Eric Bennetr, and Hao
Zhang. Dpa-net: Structured 3d abstraction from sparse views via differentiable primitive assem- bly. arXiv preprint arXiv:2404.000 00875, 2024.
Jinyang Yuan, Tonglin Chen, Bin Li, and Xiangyang Xue. Compositional scene representation learning via reconstruction: A survey. IEEE Transactions on Pattern Analysis and Machine Intel- ligence, 45(10): 1540-1105,- 2023.
Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. p 5745-5753, 2019.
13

---

Published as a conference paper at ICLR 2025
Splitting
Input
Iter=0
Iter=4k
Iter-
Figure 6: Superquadric Splitting Visualization.
APPENDIX
7.1 EXPERIMENTAL SETUP
Datasets. To explore the reconstruction quality as well as illustrate the compositional ability, several
benchmark datasets (Jensen et al., 2014, Tancik et al., 2023; Yao et al. 2020 Knapitsch et al,
2017 Barron et al., 2022) employed in evaluation. The DTU dataset (Jensen et al. (2014) and BlendedMVS (Yao et al. 2020) are MVS benchmarks, encompasses diverse calibrated indoor and outdoor scenes with annotated 3D ground truth. Nerfstudio (Tancik et al.) 2023), Mip-360 (Barron et al. (017)) offers diverse collection of multi-view real image data that spans dramatically different points in the image space.
ct al. 2021) and T Temple (Knapitsch
Evaluation Metrics. Since the main challenges we focus on include part-level internal semantic coherence, editability, and fidelity, will primarily evaluate the performance in these aspects.
Firstly, due to the limited availability of part-annotated multi-view real datasets, we use Chamfer Distance (CD) (Monnier et al., 2023 Jensen al. 2014) as an auxiliary metric quantitatively evaluate the quality of the reconstructed primitives, answering the question of how well the primi- tives fit the scenes. CD computes the Chamfer distance between the ground-truth points and points sampled from the reconstructed primitives on selected DTU scenes, our baseline DBW (Monnier jet al. 2023). Additionally, since semantic coherence and precise editablity are mainly subjective aspects, we will present qualitative results from all five datasets for subjective evaluation. Finally, to prove the competitive fidelity of our reconstructed scene, we will also use common metrics for quantitative evaluation, including SSIM, PSNR and LPIPS.
Implementation Details. In the primitive optimization step, the initial K set to 10, while the input image is resized to 400 x 300 (Monnier al. 2023). Approximately 50k iterations performed for each scene. Besides, the splitting threshold B, the fusion warm up steps & and the weight y of the Ac are set 0.7, 100 and 0.2, respectively. In terms of the silhouette settings, the offline pre-trained model (Qin et al. 2022) is used to provide the instance masks.
For the Gaussian Splatting, we Adam for parameter optimization (the same hyperparameter values are used across all subjects). We set the learning rate to 5e-3 for the position and 1.7e-2 for the scaling 3D Gaussians and keep the same learning rates 3D Gaussian Splatting (Kerbl et al. 2023) for the rest of the parameters. We train for 20k iterations, and exponentially decay the learning rate for the splat positions until the final iteration, where it reaches 0.01 the initial value. We enable adaptive density control with binding inheritance every 2,000 iterations. The overall training time is around hours on single 4090Ti.
7.2
SPLITTING AND FUSION ALGORITHM
Algorithm is the specific algorithm we use for superquadric splitting, can be visualized as Fig. Additionally, Algorithm 3 is the specific algorithm we use for superquadric fusion.
7.3
FAILURE CASES
We also include some failure cases in Fig. [[a). As discussed in the Limitations section, our method may encounter failure cases when fitting detailed structures, such as woods and leaves, due to the relatively coarse and mid-level nature of the superquadric representation. In future work, we aim to enhance our pipeline by augmenting superquadrics with a more diverse set of hybrid primitives, such cuboids and curved surfaces, to better address these limitations.
14

---

Published as a conference paper at ICLR 2025
Algorithm 2 Split
Input: set of clusters St. set of vertices V2, threshold for split max number of iterations T,
a norm factor Y
1: if N > 2 then
2:
D t ONXN
3:
for i = to N - 1 do
4:
MR = Apm E Ek,ss
5:
AR, 5 M Ememe, (Pr(ñx,m Ap,ml Ap,m)
weighted average among S & find the centroid of Sk
6:
ma. arg mE d(Akims Ak,i)
7:
for j = i to N do
8:
MR.j = [mm Sk,s
9:
Ma.s (Pr(ñx. AARml Apm) maj arg min mim d(A Ake
AR.j - MI Eme EmeMa..
weighted average among Sk find the centroid of Sk.t b Calculate the distance between Sk and Sk.
11:
Di.j + d(pk,me.. pk.m..))
12:
end for end for
13:
14:
""
arg max;,; Di.j
find the two clusters with largest distance D split if distance is larger than the threshold
15:
if
Di- > r then
16:
01 ark and 0²2 t O#k
17:
aj = 1
5 t YEk and 6/2 + -Ek
19:
& Tk and r22 + T
S t 0.4s% and 8/2 0.4sk
21:
. + Vk.my.s- and 07.2 + m
22:
I "h [81 (V (r))
23:
t/ 22 1652 ( (o)()))
24:
freeze the parameters of other blocks
end
26: end if
Algorithm Fuse
Input:
all the vertices [Vilk: = I,.... embedding (1)
......KK. KJ.
the bounding boxes (Balk 1,..., j. image
1: k + arg min (Emen Vxl/ Enweey oe v/VelVlll) find the nearest
neighbor
2: B max (B, B
merge bounding boxes
3: A 2
4: for pi [Pk. Pkr] do
5:
n(")) + f(m) (lp;, BI)
b encode point and box prompt
A, = CrossAttention( ( (()))
6:
D get cross attention map
7: Au +- Au U (
8: end for
9: S₁ t HDBSCAN(AL)
cluster A
10: if [ = 1 then
D fuse if only one cluster found
11:
0/ 0.5cm 0.50z
12:
and ap = 1
0.5g + 0.5Ek
14:
r' = 0.5r + 0.5r
s' 0.5&; + 0.5sg
16:
t' = 0.5eg + 0.5e
17:
freeze the parameters other blocks
18:
for t= 1 to 7' do
optimize the new fused block only 20: end for
19:
21:
end if
15

---

Published as a conference paper at ICLR 2025
Level
Level
(+) Failure case: Unnatural decomposition under detailed structures
(b) Superquadric Resolution Comparison
Figure 7: Failure case and Superquadric Resolution Comparison.
Figure 8: Multi-view reconstrucion results.
More multi-view reconstruction and editing results are shown in Fig. 8 and Fig.
7.4 SUPERQUADRIC RESOLUTION COMPARISON
As shown Fig. 7 using a higher-resolution superquadric would enhance reconstruction quality. However, higher resolution also implies unfair comparisons with the baseline and highly increases the computational and memory overhead. In the future, we plan to further explore the integration of higher-resolution and more detailed primitives into pipeline to achieve better results while maintaining efficiency.
7.5 MULTI-VIEW RESULTS
16

---

Published as a conference paper at ICLR 2025
Figure 9: Multi-view editing results.
17


Reviews:
[{"review_id": "oIvlTLZOkA", "reviewer": "Reviewer_CY7v", "summary": "The paper is on 3D part aware semantic editing of scenes using Gaussian Splatting . Similar to\nprevious work like GaussianAvatar the paper uses a prior for initializing the gaussians in the\nform of super-quadratics . The paper proposes a 2 stage training process in order to obtain\nsemantically coherent and disentangled gaussians that can obtain high fidelity edited images\n.The first stage optimizes the super-quadratics and the second stage uses these to initialize the\ngaussians and rasterize images. The underlying super-quadratics can be used to edit the parts\nof the object and reflect the changes subsequently in the gaussians the rasterized images .", "questions": "- Since performance of the method seems to be reliant on the super-quadratic primitives this places a limit on the number of parts that can be edited. Is there any way to control this while not losing fidelity ?\n- What is the effect on editing multiple parts in the same object in a single pass?\n- Can you show some results on more complex objects with more than 4-5 parts ?\n- Show more results on some forward facing scenes (Shiny, LLFF etc) ?\n- Why are the results for DTU and BlendedMVS shown for the whole dataset but for TnT and Mip-Nerf360 on a few scenes ? Why not the entire dataset ? or at least a few more scenes (2 scenes are not enough)\n- What is the required time for training since it is a 2 stage process ? The mention of 6 hours for 1 scene seems high and which dataset task does that scene belong to? Training time for splatting varies across different datasets and scenes so it is important\nto clarify that .", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Using super-quadratics as a prior for part aware editing using gaussian splatting is a novel approach .\n- Soft Dual rasterization for rasterizing the vertices and bounding boxes is novel, though this needs to be explained better in the paper.", "weaknesses": "- The number of parts seems to be decided by the super-quadratics which implies there is\nno control over the granularity of the parts?\n- All results in the paper edit a single part of an object in the input image. \n- All results are shown for 360 multi-view scenes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8tKuG9vW1K", "reviewer": "Reviewer_mShQ", "summary": "This paper proposes a pipeline for part-aware compositional reconstruction with 3D Gaussians, enabling precise and physically realistic editing similar to building with blocks. The method involves two training stages: In the first stage, superquadric blocks are optimized using a reconstruction loss and an attention-guided centering loss, guided by the SAM model. In the second stage, Gaussians are bound to the triangles of primitives using localized parameterization and are further optimized with an RGB loss and a local position regularization. Experiments on various datasets demonstrate state-of-the-art part-level decomposition and controllable, precise editability while maintaining high rendering fidelity.", "questions": "1. Does the method have any failure cases on these datasets in the paper, aside from challenges with complex backgrounds?\n2. The paper reports in the supplementary materials that the initial K is set to 10. How was this number determined, and how robust is the method to different initial values?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The block-based, part-aware compositional reconstruction enables intuitive local editing compared to SAM-based decomposition methods, which I find particularly interesting.\n2. To decompose a 3D scene into semantically coherent compositional primitives combined with Gaussians, the method proposes an effective two-stage optimization approach to tackle this challenging problem. It’s not straightforward to prevent sub-optimal decomposition, yet the results show compact, well-defined parts.\n3. The paper demonstrates several types of local editing with the decomposed primitives, such as moving, duplicating, and rigging parts.\n4. The paper is well-written, and the figures are beautiful and clear.", "weaknesses": "The reconstruction quality is not comparable to the original 3DGS and other baselines. On the DTU, Truck, and Garden datasets, the PSNR of this method is 5 points lower than that of the original 3DGS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "doFaW1haZz", "reviewer": "Reviewer_ovQt", "summary": "This paper proposes GaussianBlock, a part-aware compositional 3D scene representation that combines Gaussian splatting with superquadric primitives. Leveraging the strengths of both, the authors introduce Attention-guided Centering (AC) Loss and dynamic fusion/splitting modules to enhance semantic coherence and editability.", "questions": "Could you provide details on FPS and training times for both stages to clarify the overall running time? Real-time performance and faster training are also advantages of incorporating 3D Gaussians.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is technically well-written, presenting ideas clearly and effectively.\n2. Detailed experiments and visualizations demonstrate the method’s effectiveness.\n3. The controllable editability feature is highly practical, enabling applications in diverse 3D scene settings.", "weaknesses": "1. Adding more multi-view visualizations in Figure 4 would provide clearer insights into the coherence of reconstructed scenes from various perspectives.\n2. The reconstruction quality is lower than standard 3D Gaussian methods, potentially limiting fidelity in highly detailed scenes. Improvements here could enhance the method’s overall competitiveness.\n3. Background handling, also a known limitation of DBW, is not fully addressed in this work, leaving room for further improvement in complex scenes where background elements are significant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "afJhhEeEW9", "reviewer": "Reviewer_5Nb1", "summary": "The paper proposes a new 3D reconstruction pipeline based on semantic primitives that facilitates 3D scene editing and animation. At the core of the proposed method is the 3D representation based on superquadrics that is derived from the pixel-aligned SAM features. By necessary attention-guided clustering and splitting&fusion strategy, the centroids are fused into part-wise primitives to represent the 3D object. In the second stage, 3D Gaussians are bound to the surface of primitives for photorealistic rendering while maintaining the ability for animation. Although the reconstruction quality of the proposed method cannot surpass previous non-editable methods for 3D reconstruction like 3DGS, it improves the performance against editable and primitive-based methods for 3D reconstruction like DBW by a large margin.", "questions": "- The proposed method requires bounding box and point prompts along with input posed images. This difference should be highlighted as a vanilla 3D reconstruction method does not require such information. Additional information introduced into the pipeline could lead to unfair comparisons. An interesting baseline would be using 3DGS to reconstruct the semantic scenes where the SAM segmented images are used as training images. The semantic correspondence could be further introduced into the original 3DGS by finding minimum distance based on world coordinates.\n- Is the proposed method (especially for the first stage) sensitive to segmentation failure? Some scenes like forward-facing scenes in LLFF have complicated scene geometry (e.g., leaves and flowers), which could be difficult for accurate segmentation.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- To enable intuitive drag-based 3D editing and animation, the paper proposes a new hybrid representation based on superquadrics followed by 3DGS. It works well in terms of decomposing object-centric scenes into semantic primitives with a quality boost compared with previous SOTA DBW.\n- The algorithm designed for semantic alignment of superquadrics from the semantic prior of SAM looks neat to me.\n- The paper is well-structured and easy to follow.", "weaknesses": "- Lack of necessary qualitative results to support the paper’s claim: As a method for 3D editing and animation, I personally hope to see qualitative results in multiple viewpoints and timestamps, especially dynamic results which could be better demonstrated by a demo video. Otherwise, there is no clue to support that the proposed method is a good fit for editing and animating 3D scenes.\n- Is Superquadrics + 3DGS a good design? Basically, the superquadrics used in the paper have two roles: 1) offering a good initialization for the latter 3D Gaussians and 2) providing group-wise semantic correspondence of each Gaussian centroid which facilitates animation and editing. However, this two-stage pipeline inherits the downside of 3D Gaussians when generalizing to unseen “poses” of objects. As shown in Figure 4, the animated results contain severe artifacts when the animated part is moved.\n- Worthy discussion against another primitive-based representation: It is worth mentioning Mixture-of-Volumetric-Primitives as an alternative representation for the target task in this paper. It naturally has the properties for both stages in the proposed method: 1) semantic correspondence alignment and 2) photorealistic differentiable rendering. It would be great to see authors’ discussions and even experiments for this representation. Ideally, the only thing to do is to apply semantic alignment for all primitives without involving the second stage 3DGS training.\n- There is prior work in deforming and animating a well-trained 3D scene representation, which could be treated as a top-down approach (the proposed method could be treated as a bottom-up approach) to solve similar tasks:\n    - **Deforming Radiance Fields with Cages. ECCV 2022**\n    - **CageNeRF: Cage-based Neural Radiance Fields for Generalized 3D Deformation and Animation. NeurIPS 2022.**", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dOSmSzUn51", "reviewer": "Reviewer_k5SA", "summary": "This paper presents a 3D scene reconstruction approach that achieves high fidelity, editability, and part-awareness by combining superquadric primitives and 3D Gaussians.", "questions": "1. In Line 078-083, this paper discuss about the problem of \"lacking controllable editability\". However, multi-grained decomposition has already been achieved in lots of previous works for both GS-based or NeRF-based, such as [1, 2]. Besides, \"waving arms or shaking heads\" as mentioned are common editing demonstration in the field of dynamic gaussian works based on my knowledge, such as [3]. As an evidence, for the editing results in Fig.4, I believe they can be achieved by [1] or [2]. Therefore, a straightforward method for \"controllable editability\" defined in this paper might be combining existing works. I suggest the authors make the claims clearer for the motivation of the design.\n\n\n[1] Garfield: Group anything with radiance fields, CVPR 2024\n\n[2] Total-Decom: Decomposed 3D Scene Reconstruction with Minimal Interaction, CVPR 2024\n\n[3] Sc-gs: Sparse-controlled gaussian splatting for editable dynamic scenes, CVPR 2024", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Novel Hybrid Representation: The paper proposes a novel hybrid model combining superquadric primitives for part-awareness and 3D Gaussians. This hybrid design achieves high-quality 3D reconstructions while supporting precise part-level editing.\n\n2. Semantic Coherence Through Attention-Guided Centering Loss: It ensures that each superquadric primitive aligns semantically with different parts of the object. By clustering attention maps, this loss encourages disentanglement, making each part more coherent and interpretable.", "weaknesses": "1. The datasets used in the experiments are limited to DTU, BlendedMVS, Truck, and Garden, which makes it challenging to assess the generalizability of the proposed method. A broader range of data would better demonstrate its robustness across diverse scenarios.\n\n2. As shown in Table 2, the method exhibits a noticeable drop in rendering quality compared to the 3DGS baseline and does not demonstrate a clear advantage over baseline methods. The authors do not provide a detailed analysis to explain this performance gap. While editability is an attractive feature, it should not come at the cost of compromising fundamental rendering quality.\n\n3. High Computational Cost: This approach takes around 6 hours for the training time, which is time-consuming. This paper lacks the rendering frame rate and the information related to the time cost during the editing process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)