GaussianBlock: Building Part-Aware Compositional and Editable 3D Scene by Primitives and Gaussians

## Abstract

Abstract Recently, with the development of Neural Radiance Fields and Gaussian Splatting, 3D reconstruction techniques have achieved remarkably high fidelity. However, the latent representations learnt by these methods are highly entangled and lack interpretability. In this paper, we propose a novel part-aware compositional reconstruction method, called GaussianBlock, that enables semantically coherent and disentangled representations, allowing for precise and physical editing akin to building blocks, while simultaneously maintaining high fidelity.
Our GaussianBlock introduces a hybrid representation that leverages the advantages of both primitives, known for their flexible actionability and editability, and 3D Gaussians, which excel in reconstruction quality. Specifically, we achieve semantically coherent primitives through a novel attention-guided centering loss derived from 2D semantic priors, complemented by a dynamic splitting and fusion strategy.
Furthermore, we utilize 3D Gaussians that hybridize with primitives to refine structural details and enhance fidelity.
Additionally, a binding inheritance strategy is employed to strengthen and maintain the connection between the two.
Our reconstructed scenes are evidenced to be disentangled, compositional, and compact across diverse benchmarks, enabling seamless, direct and precise editing while maintaining high quality.

## 1 Introduction

Recently, with the development of Neural Radiance Fields (NeRF) (Barron et al., ; Mildenhall et al., ) and Gaussian Splatting (Jiang et al., ; Kerbl et al., ) approaches, multi-view 3D reconstruction techniques have substantially progressed, enabling the recovery of high-quality 3D assets from multi-view 2D images. Despite the impressive progress, the underlying representations learnt by these methods are highly entangled and lack interpretability.
This entanglement nature not only discourages the understanding and analysis of the model, but also hinders precise controllable editing of the reconstructed assets, as it is difficult to accurately locate the required corresponding representations and apply precise operations while ensuring other representations remain unaffected. Although advanced 3D editing methods can serve as post-processing tools, for example, one of the current state-of-the-art methods, GaussianEditor (Chen et al., ), proposes semantic tracing to constrain the modification region while keeping other parts unchanged, it remains challenging to achieve precise control as shown in Fig 1.
In contrast, compositional reconstruction - where disentangled 3D components are reconstructed from 2D image inputs - allows for individual analysis, processing, and modification, making it a particularly appealing approach. This compositional reconstruction is essential for enhancing 3D perception and understanding (Yuan et al., ; Ye et al., ), as well as for improving robotic interaction and manipulation capabilities (Vezzani et al., ).

Figure: Figure 1: While most related works face at least one of the three common limitations, including fidelity, editability, and semantically coherent part-aware disentanglement. Our method addresses all three limitations simultaneously. The underline text and masks are semtanic tracing keywords and results. Best viewed with color and marks.
Refer to caption: x1.png

A number of prior works have attempted to achieve part-aware compositional reconstruction from 2D inputs; however, several common limitations still exist.
1) Textureless or limited fidelity. Early classical works focus solely on extracting textureless part-aware shapes from images (Niu et al., ; Wu et al., ; Paschalidou et al., ; Michalkiewicz et al., ; Paschalidou et al., ; Petrov et al., ; Alaniz et al., ; Liu et al., ). Although some recent works attempt to jointly learn appearance through textured UV-maps (Monnier et al., ) or neural features (Yu et al., ; Tertikas et al., ), the fidelity of the reconstructed scenes still lags significantly behind current reconstruction methods utilizing Gaussian Splatting.
2) Semantic incoherence. One line of compositional reconstruction works  (Monnier et al., ; Alaniz et al., ; Yu et al., ) opt to use superquadrics Paschalidou et al. (); Liu et al. (), a specific type of geometric primitives
to build a 3D scene due to their high interpretability and flexible actionability, which can be directly adapted into industrial software for intuitive physical editing.
However, these methods typically focus solely on fitting “geometrically-suited” primitives into 3D scenes, while neglecting the semantic relationships and coherence between compositional parts. This often results in undesirable semantic overlap and unreasonable segmentation among reconstructed primitives.
3) Lack of controllable editablity.
Another line of methods achieves compositional reconstruction through generalizable image segmentation models like SAM (Kirillov et al., ). These methods utilize SAM to segment visual components in images and align the generated 2D masks with either implicit representations (e.g., NeRF) (Bai et al., ; Ying et al., ; Cen et al., ; Tertikas et al., ) or advanced explicit representations (e.g., Gaussians) (Ye et al., ; Cen et al., ; Dou et al., ; Ying et al., ). While these approaches yield satisfactory decomposed results thanks to impressive segmentation techniques, the reconstructed scenes only support “holistic” edits, such as translation and deletion, rather than allowing for controllable editablity. For instance, while it is feasible to uniformly remove or translate Gaussians within the same mask, more complex manipulations - such as waving arms or shaking heads, as shown in 1 - require each Gaussian to undergo distinct transformations, a capability that these methods do not support.

These challenges prompt us to
propose a novel reconstruction pipeline, named as GaussianBlock, where the semantically coherent and disentangled representation can be precisely and physically edited like building blocks, while simultaneously maintaining competitive high fidelity.
Specifically, leveraging the flexible actionability of superquadric primitives and the capacity of 3D Gaussians to reconstruct high-quality scenes, we propose a hybrid representation that integrates superquadric primitives and Gaussians, where superquadric serve as coarse but disentangled building blocks, while Gaussians act as “skin” to refine the structure and ensure high fidelity.

Our GaussianBlock reconstructs semantically coherent primitives by introducing a novel Attention-guided Centering (AC) loss to regularize the generated superquadrics.
Specifically, by using 2D differentiable prompts derived from superquadrics through dual-rasterization, the AC loss applies a clustering algorithm to supervise the grouping of queries with the same semantics, encouraging the superquadrics to disentangle accordingly.
Additionally, our GaussianBlock enhances the compactness and semantic coherence of superquadrics through the introduction of dynamic fusion and splitting strategies.
Specifically, a single superquadric that encompasses various distinguishable semantics is considered for splitting, while multiple superquadrics that share the same semantics or exhibit significant overlap are fused.

The hybrid representation in our GaussianBlock is established by attaching and connecting Gaussians to the superquadrics. Specifically, inspired by GaussianAvater (Qian et al., ), we initialize 3D Gaussians at the triangles of each reconstructed superquadric and perform primitive-based localized transformations during rasterization to reinforce their connection. Additionally, a binding inheritance strategy is employed to maintain this connection, assigning each Gaussian new identity parameters that indicate the primitive and triangle to which it belongs. During optimization and density control, our GaussianBlock utilizes a regularization loss to encourage the Gaussians to remain closely connected to their corresponding primitives while ensuring that these identity parameters are inherited throughout the process. Based on these designs, as primitives are manipulated, each 3D Gaussian can be deleted, translated, and rotated according to its associated binding vertex.

Thanks to these innovative designs, our method demonstrates state-of-the-art part-level decomposition and controllable, precise editability, with competitive fidelity across various benchmarks, including DTU Jensen et al. (), Nerfstudio Tancik et al. (), BlendedMVS Yao et al. (), Mip-360-Garden (Barron et al., ) and Tank&Temple-Truck (Knapitsch et al., ).

## 2 Related Work

Compositional Reconstruction.
Compositional reconstruction refers to reconstructing disentangled 3D components from 2D image inputs, which can be individually analyzed, processed, and modified. Early classical works often focus on abstracting textureless shape parts from images, mostly through hierarchical unsupervised learning (Paschalidou et al., ), deep neural network (Niu et al., ; Paschalidou et al., ; Michalkiewicz et al., ; Petrov et al., ), generative models (Wu et al., ), contrastive learning (Liu et al., ) or primitives (Alaniz et al., ). Recently, there have been two major lines of work in compositional reconstruction: one is primitive-based, and the other is segment-based with advanced representation.

In general, primitive-based methods typically fit “geometry-suited” primitives into the scene. For instance, Monnier et al. (Monnier et al., ) optimize the superquadrics and corresponding texture map concurrently, while Alaniz et al. (Alaniz et al., ) iteratively feed new superquadrics to an area with high reconstruction error. Recently, DPA-Net (Yu et al., ) fuses convex quadrics assembly into NeRF framework to generate an occupancy field. However, existing primitive-based works neglect the semantic relationships and coherence between compositional parts, , leading to undesirable semantic overlap and unreasonable segmentation among the reconstructed primitives.

With the advent of large generalizable image segmentation models like SAM (Kirillov et al., ), a new line of segmentation-based methods that integrate with advanced 3D scene representations such as NeRF and Gaussian Splatting has gained popularity. PartNeRF Tertikas et al. () learns independent NeRF for each pre-segmented part. CompoNeRF (Bai et al., ) segments complex text prompts and positions sub-texts within different bounding boxes, followed by the learning of a distinct NeRF for each box. In contrast, SA3D Cen et al. () utilizes a 3D NeRF as input instead of 2D images. It projects 2D pre-segmented masks onto 3D mask grids via density-guided inverse rendering, thereby achieving a certain level of 3d disentanglement.
GaussianGrouping (Ye et al., ) aligns rendered Gaussians with their corresponding pre-segmented 2D masks and groups Gaussians with similar meanings to achieve disentanglement.
Similarly, SegAnyGaussians Cen et al. () achieves Gaussian decomposition and segmentation by enforcing the correspondence between Gaussian features and 2D pre-segmented masks.
Omniseg3d (Ying et al., ) addresses the labor intensity required to ensure that 2D masks are multi-view and fine-grained consistent by introducing 3D hierarchical contrastive learning.
Despite the impressive segmentation and decomposition results achieved by the above methods, which rely on pre-segmented masks and advanced 3D representations, the reconstructed 3D assets still lack flexible editability and actionability.

3D Editing. Precise editing is a challenging task, especially when dealing with the entangled 3D underlying representations of reconstructed scenes.
Recently, numerous advanced 3D editing methods have been proposed, typically taking 3D scenes as input. These methods can function as post-processing techniques to achieve visual editing effects relevant to our primary focus, i.e., compositional reconstruction. Therefore, we discuss several of these works here.
Focaldreamer (Li et al., ) takes 3D meshes as input and introduces geometry union and focal loss to achieve text-prompted precise editing. SKED (Mikaeili et al., ) requires 3D NeRF, sketches, and text prompt, where sketches guide the regions of the NeRF that should adhere to the specified modifications. LENeRF (Hyung et al., ) also employs NeRF as input and incorporates several add-on modules, such as the latent residual mapper and attention field generation, to execute text prompts that contain desired local edits. GaussianEditor (Chen et al., ), a leading method in the field, takes well-trained Gaussian scenes and text prompts as inputs, and enhances precise editing through Gaussian semantic tracing, which traces the editing target throughout the training process.
Unlike these methods, our approach enables intuitive “drag-mode” physical and precise editing, similar to building blocks, by reconstructing actionable and semantic compositional representations.

## 3 Method

Given a set of multi-view images $\mathbf{I}$ and their corresponding silhouettes $\mathbf{I^{s}}$, our goal is to reconstruct the compositional 3D scene using semantic coherent compositional primitives combined with Gaussians. The overall pipeline is illustrated in Fig 2.

In the first stage,we adopt parameterized superquadrics primitives $\Theta_{K}$, for their remarkable expressiveness, which could be achieved with a relatively small number of continuous parameters (Barr, ; Paschalidou et al., ). In order to query from 2d pretrained segmentaion model and obtain seamtic prior, we can derive the differential point $P_{k}$ and box prompts $B_{k}$ from parameterized $\Theta_{K}$ through soft dual-rasterization, as derived in Sec. 2. Subsequently, with the reference image $\mathbf{I}$ and prompts, we can extract the corresponding attention maps $A_{k}$ for each $P_{k}$ through pretrained prompt encoder $PE$, image encoder $E$ and deconder $D$, and then apply clustering to $A_{k}$ to derive a novel AC loss in Sec. 3.2, where outlier vertices of the superquadrics are encouraged to move toward the centroid (indicated as star in Fig 2). Besides, splitting and fusion strategy, explained in Sec. 3.3, is proposed to further strengthen the compactness and disentanglement, during the AC loss and reconstruction loss between reconstructed image $I^{r}$ and silhouette $I^{s}$ supervised optimization. Through the first stage, semantic coherent part-aware superquadrics are obtained.
In the second stage, to leverage the capacity of 3D Gaussians, Gaussians are initialized and bound to superquadric triangles with localized parameterization, and will be globally mapped to world space during rasterization. Additionally, to enhance the connection between them, a regularization term $L_{pos}$ will be applied together with $L_{rgb}$, as introduced in Sec. 3.4.

### 3.1 Differential Rendering

Superquadric Parameterization.
Following the classic work (Barr, ), superquadrics can be parameterized as $\Theta_{K}=\{\alpha_{K},r_{K},t_{K},s_{K},\epsilon_{K}\}$, where $K$, $\alpha\in\mathbb{R}$, $r\in\mathbb{R}^{6}$, $t\in\mathbb{R}^{3}$, $s=\{s_{1},s_{2},s_{3}\}\in\mathbb{R}^{3}$, and $\epsilon=\{\epsilon_{1},\epsilon_{2}\}\in\mathbb{R}^{2}$ denote the number of superquadrics, transparency, 6D rotation vector, displacement vector, scaling factor and continuous shape vector, respectively.
Specifically, superquadrics can be formulated as the following explicit surface function  (Barr, ):

$$ $\displaystyle\Phi(x;\Theta_{k})$ $\displaystyle=\left[\begin{array}[]{cc}x_{1}\\ x_{2}\\ x_{3}\end{array}\right]^{T}\left[\begin{array}[]{cc}s_{1}\cos^{\epsilon_{1}} \eta\cos^{\epsilon_{2}}\omega\\ s_{2}\sin^{\epsilon_{1}}\eta\\ s_{3}\cos^{\epsilon_{1}}\eta\sin^{\epsilon_{2}}\omega\end{array}\right],$ (7) $\displaystyle\text{where}~{}~{}x$ $\displaystyle=\mathbf{rot}(r_{k})x+t_{k}.$ (8) $$

Here $\eta\in[-\frac{\pi}{2},\frac{\pi}{2}]$ and $\omega\in[-\pi,\pi]$ are the spherical coordinates of the initial unit icosphere vertices, and $\mathbf{rot}(\cdot)$ is a rigid transformation (Zhou et al., ) mapping the 6-D rotation vector to a rotation matrix.

Figure: Figure 2: The framework of our pipeline. In the first stage, the superquadrics blocks are optimized guided by reconstruction loss $\mathcal{L}_{rec}$ and attention-guided centering loss $\mathcal{L}_{AC}$. With the point $P_{k}$ and bounding box $B_{k}$ prompt obtained from soft dual rasterization, the attention maps $A_{k}$ from the last layer of pretrained decoder D are clustered, where outliers being encouraged to move towards the centroid. Meanwhile, superquadric splitting and fusion strategy is proposed to further enhance the semantic coherence and compactness during optimization. In the second stage, Gaussians are bound to the triangles of superquadrics using localized parameterization and inheritance strategy duing optimization and densification with global mapping transform.
Refer to caption: x2.png

Soft Dual Rasterization.

In order to query the 2D pretrained segmentation model and obtain semantic priors, we propose soft dual rasterization, which means we rasterize both rendered images $I^{r}$ and 2d prompts $P_{k}$ derived from the superquadrics simultaneously.

To render the superqudaric, we adopt the occupancy function of (Chen et al., ) and attach the superquadric surface with transparency information $\alpha_{k}$ Monnier et al. (), which can be summarized as:

$$ $O_{j}(u)=\alpha_{k_{j}}\exp\left(\min\left(\frac{\Delta_{j}(u)}{\sigma},0 \right)\right).$ (9) $$

Given the pixel $u$ and projected face $j$, $\Delta_{j}(u)<0$ indicates that pixel $u$ is outside face $j$, and vice versa; $\sigma$ serves as a scalar modeling the extent of the soft mask. Based on the occupancy function, we can associate $L$ faces per pixel and obtain the rendered image $\mathbf{I}^{r}$ through the classic alpha compositing (Porter & Duff, ):

$$ $\mathbf{I}^{r}(O_{1:L})=\sum_{l=1}^{L}\left(\prod_{p<l}^{L}\left(1-O_{p}\right )\right)\odot O_{l};$ (10) $$

Additionally, to obtain the reference points prompts as well, all superquadric vertices $V$ can be projected to screen coordinate using the following transformation:

$$ $\Psi(V)=\bigcup_{k=1}^{K}\mathbb{M}_{Ns}\left(\frac{\mathbb{M}_{Pj}\cdot \mathbb{M}_{Vi}\cdot\left[\begin{array}[]{cc}V\\ 1\end{array}\right]}{\tilde{\omega}}\right);$ (11) $$

where $\mathbb{M}_{Pj}$ and $\mathbb{M}_{Vi}$ are projection and view matrices derived from camera’s position and orientation (Ravi et al., ), $\tilde{\omega}$ is the resulting component from projection transformation, and $\mathbb{M}_{Ns}$ accounts for the conversion from normalized device coordinates (NDC) space to screen coordinates. Unfortunately, such an aggressive transformation incorporates undesirable vertices, whose positions are occluded due to superquadric overlapping from various perspectives. Consequently, these invisible vertices do not “contribute to” the final rendered image, causing them too ambiguous to serve effectively as point prompts. To obtain valid point prompts, we randomly assign each superquadric a unique representation color $\{C_{k}|k\in[1,K]\}$, and write the vertices rasterization equation as:

$$ $\displaystyle P_{k}=\left\{\Psi(V)\;|\;V\in\left[\sum_{l=1}^{L}\mathbf{I}^{r} \left(\Psi(V_{k})\right)\odot C_{l}\bigcap C_{k}\right]\right\},$ (12) $$

where $\{C_{l}|l\in L\}$ represents the color attribute associated with the $l$-th intersecting face. Vertices are considered as point prompts only when they are meaningful, i.e., when the representative color of a vertex is visible in the final outcome.

### 3.2 Attention-guided Centering

Aiming for semantic coherence, we expect the superquadrics to be disentangled in alignment with the semantic parts of the image $\mathbf{I}$. In this work, we utilize SAM (Kirillov et al., ) to provide the semantic prior, given its advanced performance and generalizability. Based on this semantic prior, we propose a novel Attention-guided Centering (AC) loss to ensure alignment.
The overall procedure for calculating the AC loss is summarized in Algorithm 1.

Firstly, to trigger the pre-trained SAM model, bounding boxes and point prompts are typically required along with the input images for localization and indication.
To be specific, for each superquadric $\theta_{k}$, its rendering can be acquired through Eq. 11 as $\mathbf{I_{k}}$, and corresponding box prompt $B_{k}$ can be inferred from the top-left and top-right vertices of $\mathbf{I_{k}}$. Thus, the prompt set of $\theta_{k}$ can be denoted as $\{[p_{k,m},B_{k}]|p_{k,m}\in P_{k},k\in K\}$, and the embedding of each prompt $h_{k,m}^{(p)}$ is attained through the pre-trained prompt encoder $f^{p}$ in SAM. Similarly, the embeddings of the input image denoted as $h^{(\mathbf{I})}$ are obtained through the image encoder $\mathbf{E}$.

Then, we retrieve the attention map $A_{k,m}$ prompted by $h_{k,m}^{(p)}$ from the last cross attention layer of pretrained decoder $\mathbf{D}$. Then, the overall attention map set of $\theta_{k}$, denoted as $\mathcal{A}_{k}$, is assembled from all the prompts within $\{[p_{k,m},B_{k}]\}$. Since we aim for each superquadric to be disentangled and independent, we expect only one semantic cluster to be identified in each $\theta_{k}$. Hence, we leverage HDBSCAN (Campello et al., ) algorithm to cluster the sets of attention map $\mathcal{A}_{k}$. Assume the cluster set of the $\theta_{k}$ is represented as $\mathcal{S}_{k}=\{S_{k,n}|n=1,\dots,N\}$, and $\mathrm{Pr}[n|A_{k,m}]$ is the probability that $A_{k,m}$ is assigned to a cluster $S_{k,n}$, where its corresponding cluster label is $\tilde{n}_{k,m}=\operatorname*{arg\,max}_{n}\mathrm{Pr}[n|A_{k,m}]$.
The major cluster, denoted as $S_{k}^{*}$, is defined as the largest cluster among $\mathcal{S}_{k}$, where $S_{k}^{*}=\operatorname*{arg\,max}_{S_{k,n}}|S_{k,n}|$, and its counterpart is defined as the $\complement_{S_{k}}S_{k}^{*}$.
Likewise, the centroid $p_{k,m_{k}^{*}}$ is defined as the coordinate whose $A_{k,m}$ is nearest to the weighted average attention map $\bar{A}_{k}^{*}$ among $S_{k}^{*}$, while the outliers $P_{k}^{\dagger}$ belong to $\complement_{S_{k}}S_{k}^{*}$.

Finally, to promote disentanglement, outliers are motivated to move towards the centroid, which can be expressed mathematically as:

$$ $\displaystyle\mathcal{L}_{\mathrm{AC}}\left(h,\{B_{k}\}_{k=1}^{K},\{P_{k}\}_{k =1}^{K}\right)=\sum_{k=1}^{K}\ell_{k},\quad\ell_{k}=\sum_{p_{k,i}\in P_{k}^{ \dagger}}\mathbf{d}(p_{k,i},p_{k,m_{k}^{*}});$ (13) $$

where $\mathbf{d}$ stands for the L2 Euclidean distance function.

Figure: Algorithm 1 Attention-guided Centering Loss

### 3.3 Dynamic Fusion and Splitting

To further enhance semantic coherence and compactness while preventing local minima, this section introduces dynamic superquadric splitting and fusion, taking semantic and compact constraints into account.

Splitting.
We periodically check whether a single primitive contains multiple distinguishable semantics, and for such primitives, we apply a splitting operation. Moreover, due to the centering action motivated by $L_{AC}$, superquadrics tend to cover toward containing single disentangled semantic, occasionally resulting in local minima. Inspired by  (Kerbl et al., ), we propose a dynamic splitting algorithm tailored for superquadrics, informed by the attention-based centroids.

Based on the cluster set $\mathcal{S}_{k}$, centroids $m_{k,j}$ can be localized following Algorithm. 1, and the distances between each pair of centroids can be denoted as $\{D^{k}_{i,j}|i,j\in[1,N]\}$. If the furthest distance $D^{k}_{i^{*},j^{*}}$ among $D^{k}_{i,j}$ exceeds the threshold $\tau_{s}$, which is set according to diagonal of $B_{k}$ as $\beta\check{B}_{k}$, then this superquadric $\theta_{k}$ shall be split, anchored at 3D vertices $v^{k}_{i^{*}}$ and $v^{k}_{j^{*}}$. In other words, two new superqudrics $\{\theta_{z}|z\in[K+1,K+2]\}$ will be initialized to replace $\theta_{k}$, whose $r_{z}$, $\alpha_{z}$, and $\epsilon_{z}$ are inherited from $\theta_{k}$, and $s_{z}=0.4s_{k}$. Notably, to anchor the new superquadrics towards $\{v^{k}_{x}|x\in[i^{*},j^{*}]\}$,
$t_{z}$ is initialized as Eq. 14 such that their centers are positioned at $v^{k}_{x}$, where $\bar{V}_{k}$ is the average position of $\theta_{k}$. Finally, the opacity $\alpha_{k}$ is set to 0, indicating complete transparency.

$$ $\displaystyle t_{z}=v_{x}^{k}-\left(\left(\bar{V}_{k}\circ s_{z}\right)\odot \mathbf{rot}(r_{z})\right).$ (14) $$

Fusion.
In addition to splitting, we enable dynamic superquadric fusing to enhance the compactness.
Assuming that superquadric $\theta_{e}$ and $\theta_{g}$ share the same semantic information, it is supposed that they should be fused together. Particularly, when only one cluster appears in the attention map union $\mathcal{A}_{e}\cup\mathcal{A}_{g}$ across all camera views, a new $\theta_{K+1}$ will be initialized to represent the fused one.

Similar to the splitting process, $r_{K+1}$, $t_{K+1}$, $\epsilon_{K+1}$, $\alpha_{K+1}$, and $s_{K+1}$ are determined as the mean corresponding values of $\theta_{e}$ and $\theta_{g}$. Since the initial fused outcomes are quite coarse, this leads to a sudden increase in reconstruction error and associated gradients, which heightens the instability of the optimization and impact convergence. Therefore, before proceeding with global optimization, we temporarily freeze the gradients of $\Theta_{K}$ and independently optimize $\theta_{K+1}$ for $\xi$ steps to achieve a more precise fused geometry. Likewise, $\alpha_{e}$ and $\alpha_{g}$ are ultimately set to zero.

### 3.4 3D Gaussians Binding

To leverage the advantages of both the actionability of superquadricc and the ability of 3D Gaussians to reproduce high-quality scenes, this section establishes, enhances, and maintains connections between them.

Gaussian Binding and Inheritance.
Our compositional superquadrics primarily focus on objects within the silhouette. To represent the remaining space, following the approach outlined in (Monnier et al., ), we incorporate a centered icosphere primitive to model the background dome and a plane primitive to represent the ground outside the reconstructed superquadrics, as illustrated in Fig 2.
Subsequently, inspired from GaussianAvaters (Qian et al., )
we initialize Gaussians $\mathcal{G}$ at each triangle face centers $\{T_{k}^{m}|k\in K,m\in M\}$ of the superquadrics, defined as $\mathcal{G}_{n\in N}=\{\mu,q,s,\alpha,c,id_{k},id_{t}\}$. Here, $N$, $M$, and $K$ represent the number of 3D Gaussians in the scene, the primitives, and the faces within the primitives, respectively. The parameters $\mu$, $q$, $s$, $\alpha$, and $c$ denote the Gaussian’s position, quaternion, scaling factor, opacity, and spherical harmonic (SH) coefficient, respectively. Additional parameters $id_{k}$ and $id_{t}$ are introduced to specify the corresponding superquadric and face index to which each Gaussian is bound.

During rasterization, localized transformations are applied, where each Gaussian is transformed according to the parameters of its bound triangle and superquadrics before rendering. Specifically, we map these properties into the global space by:

$$ $\displaystyle{\mu^{\prime}}={\mu}+\mu_{t},\quad r^{\prime}=r\cdot\text{Q}(r_{t }),$ (15) $$

where $\mu_{t}$ is the world position of each triangle center, $r_{t}$ is combined from the direction vector of one of the edges, the normal vector of the triangle, and their cross product; Q($\cdot$) transform the rotation vector $r_{t}$ to quaternions following  (Zhou et al., ).

Meanwhile, following the density control strategy in Gaussian Splatting (Kerbl et al., ), Gaussians shall also be split or cloned within the primitive-based local space, while inheriting their identity parameters, $id_{k}$ and $id_{t}$, to the new Gaussians. Based on the aforementioned designs, as primitives undergo manipulation, each 3D Gaussian can be deleted, translated, and rotated according to its corresponding binding triangle.

Gaussian Positional Regularization.
Furthermore, a fundamental assumption behind binding is that the Gaussian should roughly match the underlying primitives. For example, a Gaussian representing the body should not be rigged to a triangle on the cheek, as it may hinder disentanglement and subsequent editing work. Therefore, to enhance the connection between Gaussians and primitives, we implement a simple yet effective regularization loss to encourage a compact relationship between the two throughout the optimization process, as

$$ $\displaystyle\mathcal{L}_{pos}=\big{\|}max\big{(}\|\mu\|_{2},\epsilon_{pos} \big{)}-\epsilon_{pos}\big{\|}_{2}.$ (16) $$

Here $\epsilon_{pos}$, set to 0.5, is a threshold that allows Gaussians a certain degree of freedom in their position, enabling them to better refine the primitive-based 3D structure.

### 3.5 Optimization Details

The first stage. The overall objective function for superquadric optimization is summarized as:

$$ $\displaystyle\mathcal{L}_{first}=\mathcal{L}_{rec}+\gamma\mathcal{L}_{\mathrm{ AC}}$ (17) $$

where $\mathcal{L}_{rec}=\mathbf{d}(\mathbf{I}^{s},\mathbf{I}^{r})$ is the reconstruction loss between rendering outcome $\mathbf{I}^{r}$ and input silhouette $\mathbf{I}^{s}$; $\gamma$ is the weight of $\mathcal{L}_{\mathrm{AC}}$, which equals to 0 during first 2k warm-up iterations. Additionally, splitting and fusion operations are performed every 1k iterations. Meanwhile, we automatically filter out nearly transparent superquadrics, whose $\alpha$ is less than 0.1 following (Monnier et al., ).

The second stage. The overall objective function for 3D Gaussians binding and optimization is summarized as:

$$ $\displaystyle\mathcal{L}_{second}$ $\displaystyle=\mathcal{L}_{rgb}+\mathcal{L}_{pos},$ (18) $\displaystyle where\quad\mathcal{L}_{rgb}$ $\displaystyle=(1-\lambda)\mathcal{L}_{1}+\lambda\mathcal{L}_{\mathrm{D-SSIM}},$ (19) $$

with $\lambda$ = 0.2 following (Kerbl et al., ).

## 4 Experiment

The experimental setting is shown in the Appendix.

**Table 1: Performance Comparison of Primitives Reconstruction Quality on DTU Scenes. Our primitives achieve the best reconstruction quality in all scenes, despite relying only on 2D images.**
| Method | Input | S24 | S31 | S45 | S59 | S63 | S83 | S105 | $\overline{\text{CD}}\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| EMS (Liu et al., ) | 3D GT | 6.77 | 5.93 | 6.91 | 3.50 | 4.72 | 7.25 | 6.10 | 5.67 |
| DBW (Monnier et al., ) | Image | 4.90 | 3.13 | 3.86 | 4.52 | 5.02 | 4.12 | 6.48 | 4.78 |
| Ours | Image | 4.21 | 2.25 | 3.15 | 3.35 | 3.12 | 3.82 | 3.95 | 3.69 |

### 4.1 Main Results

Primitives Reconstruction.
Experiments are conducted on DTU scenes to compare the reconstruction performance of our learnt primitives with SOTA 3D decomposition methods (Liu et al., ; Monnier et al., ). Notably, EMS (Liu et al., ) is a decomposition method that utilizes the ground truth point cloud during optimization, while DBW (Monnier et al., ) is the only published primitive-based reconstruction method from 2D images that considers both decomposition and texture.

Table 1 shows the CD comparison results for each scene of the DTU dataset, where our reconstructed superquadric achieves the best performance among most scenes with the smallest Chamfer Distance.

Figure: Figure 3: Qualitative results. Our method demonstrates fine-grained, semantically coherent part-aware decomposition.
Refer to caption: x3.png

Figure: Figure 4: Editing Results. Our reconstructed scenes can be edited seamlessly and precisely.
Refer to caption: x4.png

Part-aware Semantic Coherent Decomposition.
Fig. 3, shows the results of our method, OmniSeg3D (Ying et al., ) and DBW on the DTU (first 2 rows) and BlendedMVS (last 2 rows) datasets. OmniSeg3D is a current SOTA method for part segmentation, capable of generating 3D masks containing different semantic information (indicated by colors) from 2D inputs.

From the results, we observe that DBW exhibits unnatural decomposition and fails to segment the scene into semantically coherent parts due to the lack of semantic guidance. For example, it incorrectly represents both the feet and hands of the Smurf using the same superquadric. Additionally, without a self-adaptive splitting strategy like ours, it tends to fall into local minima, as seen with the toy, where one ear is not represented. For the same reason, DBW is sensitive to the initial blocks number $K$; in complex scenes like Gundam, it requires up to 50 initial blocks to achieve an acceptable result.
Compared to OmniSeg3D, our method achieves finer-grained semantic disentanglement. For example, in the Smurf and Gundam scene, OmniSeg3D tends to assign a single semantic mask to the entire instance, whereas our approach decomposes the scene into finer-grained parts.

Precise Editablity.
In Fig. 4, we demonstrate the precise part-level editability of our method. For each data, we provide visualized results in three columns: part-aware decomposition, primitives editing (through MeshLab), and scene editing results.

**Table 2: Quantitative Comparison of reconstruction fidelity.**
| Dataset | Se. | Edit | DTU | BlendedMVS | Truck | Garden |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | SSIM$\uparrow$ | PSNR$\uparrow$ | LPIPS$\downarrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | LPIPS$\downarrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | LPIPS$\downarrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | LPIPS$\downarrow$ |  |  |
| 3DGS | ✗ | ✗ | 98.3 | 35.7 | 8.8 | 94.4 | 33.8 | 9.2 | 84.7 | 25.0 | 17.3 | 86.2 | 27.0 | 10.1 |
| OmniSeg | ✓ | ✗ | 97.7 | 33.6 | 9.4 | 93.2 | 29.4 | 10.4 | 82.6 | 23.4 | 18.0 | 85.8 | 25.9 | 13.0 |
| DBW | ✗ | ✓ | 71.3 | 19.6 | 26.5 | 49.2 | 16.4 | 41.0 | - | - | - | - | - | - |
| Ours | ✓ | ✓ | 94.3 | 30.7 | 10.3 | 90.8 | 28.8 | 11.2 | 81.3 | 22.0 | 20.7 | 84.8 | 23.1 | 16.9 |

Reconstruction Fidelity.
To evaluate the fidelity of our GaussianBlock, we compare it with one of the current state-of-the-art reconstruction methods, Gaussian Splatting (Kerbl et al., ), and the Gaussian-based OmniSeg3D, primtive-based DBW in the Table 2.
Although our method does not achieve the same or better fidelity as purely Gaussian-based approaches, it has made significant improvements over the SOTA primitive-based DBW method, and has produced competitive results compared to Gaussian-based approaches. However, the primary contribution of our method is not to propose a fidelity-oriented reconstruction algorithm. Instead, our method enables semantically coherent part-level decomposition and controllable actionability, with high but not the best fidelity.

### 4.2 Ablation Study

| Method | #K | CD$\downarrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | LPIPS$\downarrow$ |
| --- | --- | --- | --- | --- | --- |
| Full model | 5.8 | 3.69 | 94.3 | 30.7 | 10.3 |
| w/o $\mathcal{L}_{AC}$ | 5.4 | 4.75 | 84.1 | 30.3 | 11.4 |
| w/o Splitting | 4.3 | 4.24 | 88.6 | 25.7 | 16.5 |
| w/o Fusion | 9.2 | 3.46 | 92.5 | 31.0 | 10.1 |

Table: Table 3: Ablation studies on key components.

| Method | #K | CD$\downarrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | LPIPS$\downarrow$ |  |
| --- | --- | --- | --- | --- | --- | --- |
| $\gamma$ | 0.02 | 6.2 | 4.12 | 84.5 | 27.2 | 12.4 |
| 0.2 (default) | 5.8 | 3.69 | 94.3 | 30.7 | 10.3 |  |
| 2 | 4.9 | 6.97 | 80.4 | 20.8 | 13.7 |  |
| $\beta$ | 0.5 | 9.1 | 3.56 | 90.4 | 31.7 | 10.1 |
| 0.7 (default) | 5.8 | 3.69 | 94.3 | 30.7 | 10.3 |  |
| 0.9 | 4.4 | 4.85 | 84.3 | 32.9 | 14.8 |  |

Table: Table 4: Ablation studies on $\beta$s and $\gamma$s.

Figure: Figure 5: Ablation studies on (a): Experiment settings, (b) AC-Loss weight $\gamma$, (c): Splitting threshold $\beta$; (d): Regularization hyper-parameter $\epsilon_{pos}$
Refer to caption: x5.png

Attention-guided Centering Loss. In Table 3, we assess the importance of each key component of our model by sequentially removing them and evaluating the average performance across the DTU dataset. Additionally, the total number (#K) of effective superquadrics also be presented for reference.
The #K can reflect the compactness of the decomposition performance; a common goal is to achieve the highest compactness (the least #K) while maintaining the primitives performance.

Overall, $\mathcal{L}_{AC}$ and splitting strategy consistently improve the primitives quality through semantic guidance, while the fusion strategy successfully enhances the compactness decomposition with a minor quality trade-off.
An intuitive visual comparison is shown in Fig. 5 (a). While the model without splitting aggressively uses a single superquadric to represent both the roof and the building, the model without fusion redundantly uses multiple superquadrics to represent the same part. This lack of compactness makes subsequent skeleton-based editing less intuitive and meaningful.

Loss Weight.
In Table 4 and fig. 5 (b), we assess the impact of the weight of $\mathcal{L}_{AC}$, denoted as $\gamma$, on the DTU dataset. It is evident that with a lower $\gamma$, $\mathcal{L}_{AC}$ is too weak to ensure that the superquadrics can disentangle the semantic structural information in the input images. Conversely, when $\gamma$ is too high, the superquadrics tend to be small to ‘hack’ $\mathcal{L}_{AC}$, failing to meet reconstruction expectations. Thus, in this work, $\gamma$ is set to be 0.2 by default.

Splitting Threshold.
Additionally, given that the splitting threshold is $\beta\check{B}_{k}$, where $\check{B}_{k}$ is the corresponding diagonal of superquadric $\theta_{k}$ reconstruction and is dynamically updated during the optimization process, the impact of the weight $\beta$ needs to be evaluated.
As shown in Fig. 5 (c), with a higher $\beta$, the splitting mechanism is less likely to be triggered. This results in the toy’s body and ears in (c) not being effectively separated into individual superquadrics. Conversely, with a lower threshold, the scene is divided more fine-grained, as seen with the arms and paws. In conclusion, $\beta$ is used to control the granularity of the deompositin and should be selected based on the desired level of granularity for subsequent editing operations, which is set to 0.7 by default.

Positional Regularization Threshold.
We also examined the impact of different thresholds $\epsilon_{p}os$ in regularization in Fig. 5 (d). Given $\epsilon=1.5$, $1.0$, $0.5$, and $0.2$, the PSNR of the reconstructed scenes are 35.4, 33.7, 32.8, 27.3, respectively. However, during the “delete left ear” editing operation, we observe that higher $\epsilon$ tends to perform better. This is because, with a larger threshold, the connection between the primitives and Gaussians becomes weaker, leading to greater intertwinement between Gaussians. For instance, the Gaussian points on the toy’s ear semantic component may become entangled with those in the head region, causing the deletion of the ear to affect some undesired parts. On the other hand, when $\epsilon$ is too low, the fidelity suffers, as stronger regularization constraints prevent the Gaussians from fully refining the primitives’ structures.

## 5 Conclusion, Limitations and Future work

In this paper, we propose a novel part-aware compositional reconstruction method, called GaussianBlock, which enables semantically coherent and disentangled representations, allowing for precise and physical editing akin to building blocks, while simultaneously maintaining high fidelity. However, our work has certain limitations. Currently, we can only perform decomposition for object-centric entities based on their silhouette and are not yet able to handle backgrounds effectively. This is due to the fact that backgrounds typically have more complex and detailed structures with richer semantic information. In the future, beyond mid-level superquadrics, we plan to introduce more diverse types of primitives, such as cuboids and curved surfaces, in combination with Gaussians.

## 6 Appendix

### 6.1 Experimental Settings

Datasets.
To explore the reconstruction quality as well as illustrate the compositional ability, several benchmark datasets (Jensen et al., ; Tancik et al., ; Yao et al., ; Knapitsch et al., ; Barron et al., ) are employed in evaluation.
The DTU dataset (Jensen et al., ) and BlendedMVS (Yao et al., ) are MVS benchmarks, encompasses diverse calibrated indoor and outdoor scenes with annotated 3D ground truth.
Nerfstudio (Tancik et al., ), Mip-360 (Barron et al., ) and Tank&Temple (Knapitsch et al., ) offers a diverse collection of multi-view real image data that spans dramatically different points in the image space.

Evaluation Metrics.
Since the main challenges we focus on include part-level internal semantic coherence, editability, and fidelity, we will primarily evaluate the performance in these aspects.

Firstly, due to the limited availability of part-annotated multi-view real datasets, we use Chamfer Distance (CD) (Monnier et al., ; Jensen et al., ) as an auxiliary metric to quantitatively evaluate the quality of the reconstructed primitives, answering the question of how well the primitives fit the scenes. CD computes the Chamfer distance between the ground-truth points and points sampled from the reconstructed primitives on selected DTU scenes, as our baseline DBW (Monnier et al., ). Additionally, since semantic coherence and precise editablity are mainly subjective aspects, we will present qualitative results from all five datasets for subjective evaluation. Finally, to prove the competitive fidelity of our reconstructed scene, we will also use common metrics for quantitative evaluation, including SSIM, PSNR and LPIPS.

Implementation Details.
In the primitive optimization step, the initial $K$ is set to 10, while the input image is resized to $400\times 300$ as (Monnier et al., ). Approximately 50k iterations are performed for each scene. Besides, the splitting threshold $\beta$, the fusion warm up steps $\xi$ and the weight $\gamma$ of the $\mathcal{L}_{AC}$ are set to 0.7, 100 and 0.2, respectively.
In terms of the silhouette settings, the offline pre-trained model (Qin et al., ) is used to provide the instance masks.

For the Gaussian Splatting, we use Adam for parameter optimization (the same hyperparameter values are used across all subjects). We set the learning rate to 5e-3 for the position and 1.7e-2 for the scaling of 3D Gaussians and keep the same learning rates as 3D Gaussian Splatting (Kerbl et al., ) for the rest of the parameters. We train for 20k iterations, and exponentially decay the learning rate for the splat positions until the final iteration, where it reaches 0.01 $\times$ the initial value. We enable adaptive density control with binding inheritance every 2,000 iterations.
The overall training time is around 6 hours on a single 4090Ti.

### 6.2 Split Algorithm

Alg 2 is the specific algorithm we use for superquadric splitting, can be visualized as Fig 6

Figure: Algorithm 2 Split

Figure: Figure 6: Superquadric Splitting Visualization.
Refer to caption: x6.png

### 6.3 Fuse Algorithm

Alg. 3 is the specific algorithm we use for superquadric Fusing.

Figure: Algorithm 3 Fuse

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: - Since performance of the method seems to be reliant on the super-quadratic primitives this places a limit on the number of parts that can be edited. Is there any way to control this while not losing fidelity ?
- What is the effect on editing multiple parts in the same object in a single pass?
- Can you show some results on more complex objects with more than 4-5 parts ?
- Show more results on some forward facing scenes (Shiny, LLFF etc) ?
- Why are the results for DTU and BlendedMVS shown for the whole dataset but for TnT and Mip-Nerf360 on a few scenes ? Why not the entire dataset ? or at least a few more scenes (2 scenes are not enough)
- What is the required time for training since it is a 2 stage process ? The mention of 6 hours for 1 scene seems high and which dataset task does that scene belong to? Training time for splatting varies across different datasets and scenes so it is important
to clarify that .
soundness: 3
strengths: - Using super-quadratics as a prior for part aware editing using gaussian splatting is a novel approach .
- Soft Dual rasterization for rasterizing the vertices and bounding boxes is novel, though this needs to be explained better in the paper.
summary: The paper is on 3D part aware semantic editing of scenes using Gaussian Splatting . Similar to
previous work like GaussianAvatar the paper uses a prior for initializing the gaussians in the
form of super-quadratics . The paper proposes a 2 stage training process in order to obtain
semantically coherent and disentangled gaussians that can obtain high fidelity edited images
.The first stage optimizes the super-quadratics and the second stage uses these to initialize the
gaussians and rasterize images. The underlying super-quadratics can be used to edit the parts
of the object and reflect the changes subsequently in the gaussians the rasterized images .
weaknesses: - The number of parts seems to be decided by the super-quadratics which implies there is
no control over the granularity of the parts?
- All results in the paper edit a single part of an object in the input image. 
- All results are shown for 360 multi-view scenes.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Does the method have any failure cases on these datasets in the paper, aside from challenges with complex backgrounds?
2. The paper reports in the supplementary materials that the initial K is set to 10. How was this number determined, and how robust is the method to different initial values?
soundness: 3
strengths: 1. The block-based, part-aware compositional reconstruction enables intuitive local editing compared to SAM-based decomposition methods, which I find particularly interesting.
2. To decompose a 3D scene into semantically coherent compositional primitives combined with Gaussians, the method proposes an effective two-stage optimization approach to tackle this challenging problem. It’s not straightforward to prevent sub-optimal decomposition, yet the results show compact, well-defined parts.
3. The paper demonstrates several types of local editing with the decomposed primitives, such as moving, duplicating, and rigging parts.
4. The paper is well-written, and the figures are beautiful and clear.
summary: This paper proposes a pipeline for part-aware compositional reconstruction with 3D Gaussians, enabling precise and physically realistic editing similar to building with blocks. The method involves two training stages: In the first stage, superquadric blocks are optimized using a reconstruction loss and an attention-guided centering loss, guided by the SAM model. In the second stage, Gaussians are bound to the triangles of primitives using localized parameterization and are further optimized with an RGB loss and a local position regularization. Experiments on various datasets demonstrate state-of-the-art part-level decomposition and controllable, precise editability while maintaining high rendering fidelity.
weaknesses: The reconstruction quality is not comparable to the original 3DGS and other baselines. On the DTU, Truck, and Garden datasets, the PSNR of this method is 5 points lower than that of the original 3DGS.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Could you provide details on FPS and training times for both stages to clarify the overall running time? Real-time performance and faster training are also advantages of incorporating 3D Gaussians.
soundness: 3
strengths: 1. The paper is technically well-written, presenting ideas clearly and effectively.
2. Detailed experiments and visualizations demonstrate the method’s effectiveness.
3. The controllable editability feature is highly practical, enabling applications in diverse 3D scene settings.
summary: This paper proposes GaussianBlock, a part-aware compositional 3D scene representation that combines Gaussian splatting with superquadric primitives. Leveraging the strengths of both, the authors introduce Attention-guided Centering (AC) Loss and dynamic fusion/splitting modules to enhance semantic coherence and editability.
weaknesses: 1. Adding more multi-view visualizations in Figure 4 would provide clearer insights into the coherence of reconstructed scenes from various perspectives.
2. The reconstruction quality is lower than standard 3D Gaussian methods, potentially limiting fidelity in highly detailed scenes. Improvements here could enhance the method’s overall competitiveness.
3. Background handling, also a known limitation of DBW, is not fully addressed in this work, leaving room for further improvement in complex scenes where background elements are significant.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: - The proposed method requires bounding box and point prompts along with input posed images. This difference should be highlighted as a vanilla 3D reconstruction method does not require such information. Additional information introduced into the pipeline could lead to unfair comparisons. An interesting baseline would be using 3DGS to reconstruct the semantic scenes where the SAM segmented images are used as training images. The semantic correspondence could be further introduced into the original 3DGS by finding minimum distance based on world coordinates.
- Is the proposed method (especially for the first stage) sensitive to segmentation failure? Some scenes like forward-facing scenes in LLFF have complicated scene geometry (e.g., leaves and flowers), which could be difficult for accurate segmentation.
soundness: 2
strengths: - To enable intuitive drag-based 3D editing and animation, the paper proposes a new hybrid representation based on superquadrics followed by 3DGS. It works well in terms of decomposing object-centric scenes into semantic primitives with a quality boost compared with previous SOTA DBW.
- The algorithm designed for semantic alignment of superquadrics from the semantic prior of SAM looks neat to me.
- The paper is well-structured and easy to follow.
summary: The paper proposes a new 3D reconstruction pipeline based on semantic primitives that facilitates 3D scene editing and animation. At the core of the proposed method is the 3D representation based on superquadrics that is derived from the pixel-aligned SAM features. By necessary attention-guided clustering and splitting&fusion strategy, the centroids are fused into part-wise primitives to represent the 3D object. In the second stage, 3D Gaussians are bound to the surface of primitives for photorealistic rendering while maintaining the ability for animation. Although the reconstruction quality of the proposed method cannot surpass previous non-editable methods for 3D reconstruction like 3DGS, it improves the performance against editable and primitive-based methods for 3D reconstruction like DBW by a large margin.
weaknesses: - Lack of necessary qualitative results to support the paper’s claim: As a method for 3D editing and animation, I personally hope to see qualitative results in multiple viewpoints and timestamps, especially dynamic results which could be better demonstrated by a demo video. Otherwise, there is no clue to support that the proposed method is a good fit for editing and animating 3D scenes.
- Is Superquadrics + 3DGS a good design? Basically, the superquadrics used in the paper have two roles: 1) offering a good initialization for the latter 3D Gaussians and 2) providing group-wise semantic correspondence of each Gaussian centroid which facilitates animation and editing. However, this two-stage pipeline inherits the downside of 3D Gaussians when generalizing to unseen “poses” of objects. As shown in Figure 4, the animated results contain severe artifacts when the animated part is moved.
- Worthy discussion against another primitive-based representation: It is worth mentioning Mixture-of-Volumetric-Primitives as an alternative representation for the target task in this paper. It naturally has the properties for both stages in the proposed method: 1) semantic correspondence alignment and 2) photorealistic differentiable rendering. It would be great to see authors’ discussions and even experiments for this representation. Ideally, the only thing to do is to apply semantic alignment for all primitives without involving the second stage 3DGS training.
- There is prior work in deforming and animating a well-trained 3D scene representation, which could be treated as a top-down approach (the proposed method could be treated as a bottom-up approach) to solve similar tasks:
    - **Deforming Radiance Fields with Cages. ECCV 2022**
    - **CageNeRF: Cage-based Neural Radiance Fields for Generalized 3D Deformation and Animation. NeurIPS 2022.**

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. In Line 078-083, this paper discuss about the problem of "lacking controllable editability". However, multi-grained decomposition has already been achieved in lots of previous works for both GS-based or NeRF-based, such as [1, 2]. Besides, "waving arms or shaking heads" as mentioned are common editing demonstration in the field of dynamic gaussian works based on my knowledge, such as [3]. As an evidence, for the editing results in Fig.4, I believe they can be achieved by [1] or [2]. Therefore, a straightforward method for "controllable editability" defined in this paper might be combining existing works. I suggest the authors make the claims clearer for the motivation of the design.


[1] Garfield: Group anything with radiance fields, CVPR 2024

[2] Total-Decom: Decomposed 3D Scene Reconstruction with Minimal Interaction, CVPR 2024

[3] Sc-gs: Sparse-controlled gaussian splatting for editable dynamic scenes, CVPR 2024
soundness: 2
strengths: 1. Novel Hybrid Representation: The paper proposes a novel hybrid model combining superquadric primitives for part-awareness and 3D Gaussians. This hybrid design achieves high-quality 3D reconstructions while supporting precise part-level editing.

2. Semantic Coherence Through Attention-Guided Centering Loss: It ensures that each superquadric primitive aligns semantically with different parts of the object. By clustering attention maps, this loss encourages disentanglement, making each part more coherent and interpretable.
summary: This paper presents a 3D scene reconstruction approach that achieves high fidelity, editability, and part-awareness by combining superquadric primitives and 3D Gaussians.
weaknesses: 1. The datasets used in the experiments are limited to DTU, BlendedMVS, Truck, and Garden, which makes it challenging to assess the generalizability of the proposed method. A broader range of data would better demonstrate its robustness across diverse scenarios.

2. As shown in Table 2, the method exhibits a noticeable drop in rendering quality compared to the 3DGS baseline and does not demonstrate a clear advantage over baseline methods. The authors do not provide a detailed analysis to explain this performance gap. While editability is an attractive feature, it should not come at the cost of compromising fundamental rendering quality.

3. High Computational Cost: This approach takes around 6 hours for the training time, which is time-consuming. This paper lacks the rendering frame rate and the information related to the time cost during the editing process.

Meta-review: This paper introduced GaussianBlock, a hybrid 3D representation combining primitives and 3D Gaussians for unsupervised, part-aware scene decomposition guided by 2D semantic priors. It claims to achieve disentangled, editable, high-fidelity reconstructions without requiring ground truth segmentation.

Paper's strengths lie in several aspects: (a) an effective two-stage optimization strategy that ensures compact and well-defined parts, (b) intuitive local editing capabilities such as moving and rigging, and (c) seamless semantic alignment through attention-guided centering. On the performance side, the method demonstrates advantages over previous approaches in producing high-quality rendering and editable results. The paper is well-structured and easy to understand.

The major weaknesses include limited qualitative results, concerns about the superquadrics + 3DGS design, dataset diversity, and a drop in reconstruction quality. The authors addressed these by adding multi-view results with more datasets, clarifying the pipeline for additional inputs, and explaining trade-offs in quality and design. They also clarify the difference between their method and the deformation-based approaches and Mixture-of-Volumetric Primes, emphasizing their method's unique goals and capabilities. No quantitative benchmarks to validate these claims. 

The paper provides a reasonable 2-stage approach making the 3DGS editable. The idea is novel, and the performance is appealing in general. Despite remaining concerns, such as the degraded quality, limited dataset diversity, and limitations of the superquadrics + 3DGS design, the method’s innovative hybrid representation and practical controllable editability make it a valuable contribution to the field.

Decision: Accept (Poster)