Mani-GS: Gaussian Splatting Manipulation with Triangular Mesh

## Abstract

Abstract Neural 3D representations such as Neural Radiance Fields (NeRF), excel at producing photo-realistic rendering results but lack the flexibility for manipulation and editing which is crucial for content creation. Previous works have attempted to address this issue by deforming a NeRF in canonical space or manipulating the radiance field based on an explicit mesh. However, manipulating NeRF is not highly controllable and requires a long training and inference time. With the emergence of 3D Gaussian Splatting (3DGS), extremely high-fidelity novel view synthesis can be achieved using an explicit point-based 3D representation with much faster training and rendering speed. However, there is still a lack of effective means to manipulate 3DGS freely while maintaining rendering quality. In this work, we aim to tackle the challenge of achieving manipulable photo-realistic rendering. We propose to utilize a triangular mesh to manipulate 3DGS directly with self-adaptation. This approach reduces the need to design various algorithms for different types of Gaussian manipulation. By utilizing a triangle shape-aware Gaussian binding and adapting method, we can achieve 3DGS manipulation and preserve high-fidelity rendering after manipulation. Our approach is capable of handling large deformations, local manipulations, and soft body simulations while keeping high-quality rendering. Furthermore, we demonstrate that our method is also effective with inaccurate meshes extracted from 3DGS. Experiments conducted demonstrate the effectiveness of our method and its superiority over baseline approaches.

###### Keywords:

## 1 Introduction

Figure: Figure 1: Our proposed approach allows for 3DGS manipulation, including large deformation, local manipulation, and even physical simulation (such as soft body simulation), while maintaining high-quality rendering. Please zoom in for more details.
Refer to caption: x1.png

Manipulating and editing 3D content is essential for content creation and has various applications in movies, gaming, and virtual/augmented reality. 3D model editing enables users to create and modify models flexibly, thereby enhancing production efficiency. The traditional pipeline for modeling and editing a 3D asset with photo-realistic rendering involves a process with geometry modeling, texturing, UV mapping, lighting, and rendering, which is a tedious and time-consuming flow requiring lots of manual work.

Over the past few years, the neural radiance field (NeRF)[] has been widely studied due to its high capability and simple reconstruction process in 3D representation. However, the implicit representation poses challenges for editing.
To address this, some methods are proposed to edit this implicit neural radiance field[]. NeRF-Editing[] is the first to utilize the triangular mesh to help edit the implicit radiance field. They train a canonical NeuS [] and extract the triangular mesh from NeuS[]. A tetrahedra grid is then constructed to contain the object mesh. To render the deformed object by editing the triangular mesh, a volume rendering is conducted in the deformed space. The sampling points in deformed space are mapped to canonical space based on the constructed tetrahedra grid, where the points in deformed space have the same tetrahedron barycentric coordinate in the same tetrahedron with their corresponding points in canonical space. Moreover, [, ] demonstrate similar ideas by employing tetrahedra to deform sampling points and achieve editable nerf-based scenes. Instead, Neu-Mesh[] and SERF[] define the neural radiance field by associating each mesh vertex with radiance and geometry features. Then they can conduct volume rendering like Point-NeRF[] in deformed space without backward mapping. However, these editing methods based on implicit neural radiance fields still suffer from inconvenient manipulation, suboptimal rendering results, and long training and rendering times.

Meanwhile, 3DGS[] has gained significant attention in differential rendering due to its high-fidelity and fast rendering proficiency. However, despite being an explicit 3D representation, it still lacks an effective method for manipulating 3DGS while maintaining high-quality rendering. SuGaR[], the work most closely related to our objective, develops a novel algorithm to extract a triangular mesh from 3DGS. Although their main goal is not to facilitate editable photorealistic rendering, they bind the 3DGS to the extracted mesh, enabling model animation as demonstrated in their demo.

This work proposes a method that enables 3DGS manipulation, achieving high-quality and photo-realistic rendering. Our key insight is to manipulate 3DGS using a triangular mesh as the proxy, which allows for direct transfer of mesh manipulation to 3DGS with 3DGS self-adaptation. With our methods, we can achieve large deformation, local manipulation, and soft body simulaton with high-quality results as shown in Figure 1, which also avoid the need to design various algorithms for different types of manipulation.

To achieve controllable 3DGS manipulation through the mesh, an intuitive approach is to bind the GS to lay perfectly on the triangle and enforce the GS to be thin enough. After mesh manipulation, the GS will automatically adapt its rotation and position with the attached triangle, as employed in SuGaR[]. However, SuGaR heavily relies on the accuracy of mesh geometry, inheriting the defects of mesh rendering. Specifically, for inaccurate parts, SuGaR cannot inpaint the missing parts or remove the redundant parts in the final rendering.

Adding an offset to the position of attached Gaussians during the reconstruction may seem like a reasonable solution to compensate for mesh inaccuracy. However, this fixed offset cannot be generalized well to the deformed space after novel manipulation.
Our proposed solution is to define a local coordinate system for each triangle, which we refer to as local triangle space. We then bind Gaussians to each triangle and optimize the Gaussian attributes, including rotation, position, and scaling, in the attached local triangle space.

During mesh manipulation, the attributes in the local triangle space remain unchanged, while the global Gaussian position, scaling, and rotation will be self-adaptively adjusted according to our proposed formula. As a result, our proposed approach enables us to manipulate 3DGS using a triangular mesh while maintaining rendering quality. Since our Gaussians are set to be free outside the triangle, we can also support high-fidelity manipulation even when the Gaussians are bound to an inaccurate mesh, exhibiting a high tolerance for mesh accuracy.

GaMeS[] and Mesh-GS [] are two concurrent works that also employ triangular meshes for Gaussian Splatting manipulation. In particular, GaMeS[] constrains the Gaussians on the surface exactly; Mesh-GS [] permits an offset along the normal direction without adapting the Gaussian scale when the triangle shape changes.
In contrast, our model allows for Gaussian move in the triangle local space which means we can achieve high-quality rendering without the need for accurate mesh. Gaussian scaling also adapts in response to changes in triangle shape when large deformations are applied to the mesh.

In summary, the contributions of our paper are listed as follows:

- •
We propose a 3DGS manipulation method that can effectively transfer the triangular mesh manipulation to 3DGS and maintain high-quality rendering.
- •
We introduce a triangle shape aware Gaussian binding strategy with self-adaption, which has a high tolerance for mesh accuracy and supports various 3DGS manipulations.
- •
We evaluate our method and achieve state-of-the-art results, demonstrating various 3DGS manipulations, including large deformation, local manipulation, and soft body simulation.

## 2 Related Work

### 2.1 NeRF Editing

Recently, neural radiance field [] (NeRF) has garnered significant attention due to its high-quality and photo-realistic rendering results for novel view synthesis. NeRF represents the scene as a continuous function that maps a spatial location and viewing direction to a volume density and color, which is parameterized by a multilayer perceptron (MLP). Owing to the implicit representation that encodes the scene within the network parameters, editing and deforming the geometry of the NeRF scene explicitly like mesh can be challenging. To enable user editing of NeRF,
[] introduce editing conditional radiance fields trained on a shape category. However, it only supports basic editing operations, such as removing/adding object parts or shape transfer. CLIP-NeRF [] achieves NeRF editing with text or images by leveraging CLIP model [] but still can not edit the geometry locally. Some other work[, , , ] edit the NeRF in texture level which is not the focus of this paper.

To edit and deform NeRF locally, [, , ] construct a tetrahedra grid based on the underlying 3D shape. After explicitly deforming the tetrahedra into the posed space for editing, the sampled 3D points are mapped from the posed space to the canonical space through the unaltered tetrahedron, which means the canonical position can be calculated from the shared barycentric coordinate for both deformed and canonical tetrahedron. The density and radiance in the posed space can be calculated for the mapped sampling points in canonical space. On the other hand, [, , ] employ mesh as the guidance for deformation. NeuMesh [] presents a novel representation to encode neural implicit field on a mesh-based scaffold for geometry and texture editing. [] achieves the manipulation of both the geometry and color of neural implicit fields through differentiable colored meshes. Furthermore, there are several methods [, , , , ] that particularly focus on editing NeRF for avatar. In this work, we introduce an editing method based on 3DGS for general objects.

### 2.2 Mesh-based NeRF Rendering

NeRFs have shown impressive rendering results, however, rendering one pixel using NeRF representation necessitates a volumetric rendering algorithm that involves inferring MLP hundreds of times to estimate their radiance and density. This process is significantly slower than traditional mesh rendering. To accelerate NeRF rendering, several methods [, , , , , ] have been proposed to combine NeRF representation with mesh reconstruction. By converting this implicit representation to an explicit mesh, these methods may also facilitate applications like editing. In particular, MobileNeRF [] represents NeRF as a collection of polygons with deep feature textures, which can be rendered using the classic polygon rasterization pipeline, generating a feature vector for each pixel and passing it to an MLP to decode the color. MobileNeRF is capable of achieving rendering even on standard mobile devices. Additionally, NeRFMeshing [] distills the reconstructed NeRF into a signed surface approximation network to extract 3D mesh and shows the simulation results for editing. BakedSDF [] proposes a neural surface-volume representation to extract the mesh and supports editing like material decomposition, appearance editing, and physics simulation. However, the editing results of these methods heavily rely on the accuracy of the extracted mesh. Artifacts present in the mesh will directly influence the editing results. In contrast, our method demonstrates robustness to the reconstructed mesh and can still yield promising results even with the inaccurate mesh.

### 2.3 Gaussian Splatting Editing, Simulation and Animation

3D Gaussian Splatting [] presents an innovative 3D Gaussian scene representation, accompanied by a differentiable renderer that attains real-time rendering of radiance fields while maintaining high quality. Initially, 3D Gaussian Splatting focuses solely on static scenes, which has been extended to model dynamic scenes [, , , , ], human avatars [, , , , , ], Gaussian Splatting simulation and animation [, , , ]. Specifically, SuGaR [] proposes a method to extract meshes from 3D Gaussian Splatting with additional regularization, in which they bind 3DGS on extracted mesh and animated with Gaussian Splatting rendering. GSP [] incorporates physically-based fluid dynamics in 3DGS and PhysGaussian [] introduces a unified simulation-rendering pipeline that generates physics-based dynamics with photorealistic renderings. VR-GS [] achieves interactive physics-based editing in Virtual Reality. In this work, we also introduce a Gaussian Splatting manipulation method that binds Gaussian Splatting to the mesh, achieving state-of-the-art results.

## 3 Method

Recently, due to its exceptional high-fidelity rendering capabilities and fast rendering speed, Gaussian Splatting [] has emerged as a popular 3D representation in the differential rendering field. However, despite being an explicit 3D representation, it still lacks a methodology for manipulating this 3D representation for editing while maintaining high-quality rendering after the manipulation. In this work, giving multi-view RGB images of an object as input, we introduce a method for object manipulation that can achieve photorealistic editable rendering by employing Gaussian Splatting.

The pipeline of our method is illustrated in Figure 2 and consists of three main stages. First, we extract a mesh from 3D Gaussian Splatting (3DGS) or a neural surface field for subsequent 3D Gaussian binding (Sec. 3.2). Next, we devise a novel Mesh-Gaussian binding method dedicated to manipulating Gaussian Splatting while maintaining photo-realistic rendering quality (Sec. 3.3). Finally, we describe the types of Gaussian manipulation we support and how to render the manipulated results (Sec. 3.4).

### 3.1 Preliminary

Thanks to its superior capability for high-fidelity and rapid rendering, 3D Gaussian Splatting (3DGS) [] has recently emerged as a popular 3D representation in differential rendering. 3DGS utilizes explicit 3D Gaussians as its primary rendering primitives. A 3D Gaussian point is mathematically defined as:

$$ $G(\bm{x})=exp(-\frac{1}{2}(\bm{x}-\bm{\mu})^{\top}\Sigma^{-1}(\bm{x}-\bm{\mu})).$ (1) $$

Each 3D Gaussian point is characterized by a 3D mean position coordinate $\bm{\mu}$ and a covariance matrix $\Sigma$. Additionally, each Gaussian has an opacity $\bm{o}$ and a view-dependent color $\bm{c}$ represented by a set of spherical harmonics (SH). To ensure that the covariance matrix $\Sigma$ retains its meaningful interpretation, it is parameterized as a unit quaternion $\bm{q}$ and a 3D scaling vector $\bm{s}$, defined as ${\Sigma=\bm{RS}\bm{S}^{\top}\bm{R}^{\top}}$.

To render an image from a specific viewpoint, 3D Gaussians are projected onto the image plane, resulting in 2D Gaussians. The 2D covariance matrix is approximated as:

$$ $\Sigma^{\prime}=\bm{J}\bm{W}\Sigma\bm{W}^{\top}\bm{J}^{\top},$ (2) $$

where $\bm{W}$ and $\bm{J}$ denote the viewing transformation and the Jacobian of the affine approximation of perspective projection transformation [], respectively. The 2D means are calculated through the projection matrix. After this, the pixel color is composited through the alpha blending of $N$ ordered 2D Gaussians:

$$ $\mathcal{C}=\sum_{i\in{N}}T_{i}\alpha_{i}\bm{c}_{i}\text{ with }\hskip 5.0ptT_ {i}=\prod_{j=1}^{i-1}(1-\alpha_{i}).$ (3) $$

Here, $\alpha$ is obtained by multiplying the opacity $\bm{o}$ with the 2D covariance’s probability computed from $\Sigma^{\prime}$ and pixel coordinate on the image space.

### 3.2 Mesh Extraction

Our method is capable of achieving high-quality reconstruction and editing using guided meshes obtained from various methods. In this section, we investigate different mesh extraction and reconstruction techniques with different mesh accuracy and processing time to guide the 3D Gaussians in our approach.

Screened Poisson Reconstruction. 3D Gaussian Splatting could be considered a type of point cloud, making it intuitive to extract the mesh using the Poisson-reconstruction algorithm. However, the 3D Gaussians do not have normal vectors for reconstruction. Inspired by recent 3DGS inverse rendering methods [, ], we allocate an additional gaussian attribute, normal $\bm{n}$, for 3D Gaussians:

$$ $\{\mathcal{D},\mathcal{N}\}=\sum_{i\in{N}}T_{i}\alpha_{i}\{d_{i},\bm{n}_{i}\},$ (4) $$

where $d_{i}$ denotes the depth and $\bm{n}_{i}$ denotes the normal of each gaussian point. To optimize the normal, we then encourage the rendered normal $\mathcal{N}$ to be well-aligned with the pseudo-normal map $\tilde{\mathcal{N}}$, computed from the rendered depth $\mathcal{D}$ based the local planarity assumption.
After training 3DGS with normal attributes, we can extract the mesh using the Screened poisson surface reconstruction[] algorithm.

Marching Cube for Gaussian Splatting.
In DreamGaussian[], the method attempts to summarize the alpha values of neighboring Gaussian points as the composite density value of marching cube sampling points. The Gaussian points within a pre-defined local voxel are considered neighbors. However, We found that this method often ignores the thin and small structures. To address this issue, we employ ball query Nearest Neighbor searching methods to search only the nearest Gaussian points of grid sampling points. The sampling points that have a neighbor within a pre-defined distance are assigned with a density value of $1$, otherwise with a density value of 0 0. The geometry is then extracted using the Marching Cube. This approach allows for the extraction of thin and small structures, even though they may be slightly larger than the actual surface.
With our Mesh-Gaussian binding strategy, we can achieve high-fidelity rendering with the slightly inaccurate mesh and support smooth Gaussian manipulation.

Neural Implicit Surfaces.
In this work, we also try to extract high-quality surfaces from the implicit representation utilizing the method proposed in NeuS []. NeuS suggests extracting a surface as the zero-level set of a signed distance function (SDF), which has consistently served as a robust and widely used benchmark in the realm of neural surface reconstruction. Nevertheless, the surface derived from NeuS may encompass an excessive number of triangle surfaces, which may have around 1.5 million. A large number of triangles can negatively affect both training and inference speeds, therefore, we employ a mesh cleaning process to eliminate the noisy floaters and utilize mesh decimation techniques to reduce the count of triangles to approximately 300K. We circumvent the need for such a high number of triangles while maintaining high-fidelity rendering.

### 3.3 Binding Gaussian Splatting on Mesh

Owing to the exceptional proficiency in high-fidelity and fast rendering, 3DGS has gained significant attention in differential rendering. However, despite being an explicit 3D representation, it currently lacks a method for effectively manipulating 3DGS while preserving high-quality rendering simultaneously. Mesh editing techniques, such as large-scale deformation, localized manipulation, and simulation, have been widely acknowledged and extensively researched for many years. Our primary objective is to associate the 3DGS with mesh triangles, enabling the manipulation of 3DGS and its rendering results following mesh editing.

Figure: Figure 2: Overview of our method.(1) Firstly, we extract a triangular mesh from 3DGS[] or a neural surface field(NeuS[]). (2) Next, we bind $N$ Gaussians to each triangle in the local triangle space, and optimize the local Gaussian attributes ($\bm{\mu^{l},R^{l},s^{l},o,c}$). The triangle attributes ($\bm{\mu^{t},R^{t},e}$) are calculated based on the triangle vertices. (3) Finally, we manipulate 3DGS by transferring the mesh manipulation directly, thus achieving manipulable rendering.
Refer to caption: x2.png

Given a reconstructed or extracted triangular mesh $\bm{T}$ with $K$ vertices $\{\bm{v}_{i}\}_{i=1}^{K}$ and $M$ triangles $\{\bm{f}_{i}\}_{i=1}^{M}$, the goal of our method is to construct a 3DGS model bound to mesh triangles and optimize each Gaussian attribute $\{\bm{\mu}_{i},\bm{q}_{i},\bm{s}_{i},o_{i},\bm{c}_{i}\}$. To simplify the notation, we will omit the subscript in subsequent sections.

For each triangle $\bm{f}$ in the given mesh $\bm{T}$, which is composed of three vertices $(\bm{v_{1}},\bm{v_{2}},\bm{v_{3}})$, we initialize $N$ Gaussians on this triangle. To be specific,
the mean position $\bm{\mu}$ of initialized Gaussians is formulated as $\bm{\mu}=(w_{1}\bm{v_{1}}+w_{2}\bm{v_{2}}+w_{3}\bm{v_{3}})$, $\bm{w}=(w_{1},w_{2},w_{3})$ is the pre-defined barycentric coordinate of each Gaussians attached on the triangle. And $\bm{w}$ satisfy $(w_{1}+w_{2}+w_{3})=1$.

Gaussians on Mesh. To achieve controllable 3DGS manipulation through the mesh, an intuitive way is to perfectly attach the GS to the triangle, as shown in the SuGaR[]. With the rotation matrix denoted as $\bm{R}=\{\bm{r_{1}},\bm{r_{2}},\bm{r_{3}}\}$ and the scaling vector represented by $\bm{s}=(s_{1},s_{2},s_{3})$, SuGaR train 3DGS with a flat Gaussian distribution on the mesh by setting $s_{1}=\epsilon$, where $\epsilon$ is close to zero. $\bm{r_{1}}$ is defined by the normal vector $\bm{n}$ of the attached triangle. The Gaussians have only 2 learnable scaling factors $(s_{2},s_{3})$ instead of 3, and only 1 learnable 2D rotation rather than a quaternion, to keep the Gaussians flat and aligned with the mesh triangles. This type of binding strategy heavily relies on mesh accuracy, which lacks the flexibility of 3DGS to model complex object rendering. Moreover, the mesh quality obtained from SuGaR is significantly inferior compared to the ground truth and recent neural surface reconstruction results, which increases the difficulty of editing.

Gaussians on Mesh with Offset. To compensate for the inaccuracy of the extracted mesh, it would be better to add an offset $\Delta\bm{\mu}$ to the Gaussians 3D mean $\bm{\mu}$, which enables the Gaussians to move out of the attached triangle $\bm{f}$. Although it could improve the rendering quality of the reconstructed static object, it would result in noisy and unexpected rendering distortion in the manipulated object due to the mismatched localized relative position between Gaussians.

Triangle Shape Aware Gaussian Binding and Adapting. To preserve the high-fidelity rendering results after manipulation, the key lies in maintaining the local rigidity and preserving the relative location between Gaussians, both for 3D means and rotations. Our key insight is to define a local coordinate system in each triangle space.

The first axis direction of triangle space is defined as the direction of the first edge.
The second axis direction of triangle space is defined as the triangle’s normal direction.
The third axis direction of triangle space is defined as the cross product of the first and second axis.
Then the triangle coordinate system rotation can be formulated as:

$$ $\bm{R^{t}}=[\bm{r_{1}^{t}},\bm{r_{2}^{t}},\bm{r_{3}^{t}}]=[\frac{(\bm{v_{2}}- \bm{v_{1}})}{\lVert\bm{v_{2}}-\bm{v_{1}}\rVert},\bm{n^{t}},\frac{(\bm{v_{2}}- \bm{v_{1}})}{\lVert\bm{v_{2}}-\bm{v_{1}}\rVert}\times\bm{n^{t}}]$ (5) $$

where $\bm{v_{1}}$, $\bm{v_{2}}$ is the first and second vertex location repectively, $\bm{n^{t}}$ is the normal vector which can be calulated by:

$$ $\bm{n^{t}}=\frac{(\bm{v_{2}}-\bm{v_{1}})\times(\bm{v_{3}}-\bm{v_{1}})}{\lVert( \bm{v_{2}}-\bm{v_{1}})\times(\bm{v_{3}}-\bm{v_{1}})\rVert}.$ (6) $$

We then optimize the Gaussians’ local position $\bm{\mu^{l}}$ and local rotation $\bm{R^{l}}$ in triangle space instead of the global position and rotation in the original 3DGS.

Then the global rotation, scale and location of 3DGS are as follows:

$$ $\bm{R={R^{t}R^{l}}},\bm{s}={\bm{s}^{l}},\quad\bm{\mu=R^{t}\mu^{l}+\mu^{t}}$ (7) $$

where, $\bm{\mu^{t}}$ is the global coordinate of each triangle center. In practice, we initialize $N$ local Gaussian points and bind them for each Gaussian point, whose initialized position is on the triangle uniformly.

This Gaussian mesh binding method can preserve the relative position and rotation between Gaussians that are bonded on neighboring triangles after mesh manipulation. However, following mesh manipulation, not only does the triangle center change but also the triangle shape. With the altered triangle shape, the local Gaussian position and scaling should adjust accordingly. When the triangle enlarges, it is intuitive that the local scaling and position should expand as well:

$$ $\bm{R={R^{t}R^{l}}},\bm{s}={\beta\bm{e}\bm{s}^{l}},\quad\bm{\mu=eR^{t}\mu^{l}+ \mu^{t}},$ (8) $$

where $\beta$ is a hyper-parameter, adaption vector $\bm{e}=[e_{1},e_{2},e_{3}]$ is designed to make sure that the global scaling $\bm{s}$ is proportionable to the triangle shape. The first axis is along the first edge, so $e_{1}$ is designed as the length $l_{1}$ of the first edge of the triangle. The second axis is along the normal direction, we set $e_{2}=(0.5*(e_{1}+e_{3}))$. The third axis is perpendicular to the first edge, we set $e_{3}$ as the average length of the second and third edges $(0.5*(l_{2}+l_{3}))$.

### 3.4 Manipulate Gaussian Splatting through Mesh

Utilizing our triangle shape aware gaussian binding and adapting strategy, upon the completion of model training and mesh manipulation, the 3DGS is instantly manipulated and adapted.
During mesh manipulation, the attributes in the local triangle space remain unchanged. The triangle rotation, position, and edge length can be calculated instantly. Therefore, the global Gaussian position, scaling, and rotation can be self-adaptively adjusted following our proposed formula.
In this paper, we exhibit the 3DGS manipulation rendering outcomes, such as large-scale deformation, local manipulation, and soft-body simulation, which are driven by the manipulated mesh. Many 3D design software applications possess the capability to execute mesh manipulation. In our experiments, we employ Blender to manipulate the mesh.

## 4 Experiments

### 4.1 Training Details

The first stage of our methods includes a mesh extracting stage, during which we extract triangular mesh from neural implicit surface filed (NeuS[]) or 3D Gaussian splatting (Screened Poisson or Marching Cube) only. However, the extracted mesh always contains enormous triangles, which we try to decimate to approximately 300K.

With the extracted mesh, we conduct the triangle shape aware Gaussian binding and adapting strategy on the mesh. For each triangle, we bind $N=3$ Gaussian on the surface initially. The Gaussian attributes are optimized subsequently with the supervision of multi-view rendering loss in the second stage.
We train our model for 30K iterations in the initial stage to extract mesh and 20K iterations in the second stage. All experiments are conducted on a single NVIDIA A100 GPU. Please see the supplementary material for more details.

### 4.2 Datasets, Metrics and Methods for Comparison

To evaluate our methods, we compare Mani-GS with previous editable novel view synthesis methods, a NeRF-based editing method NeRF-Editing [] and a 3DGS-based editing method SuGaR[]. For the evaluation, we employ the commonly used metrics:PSNR, SSIM, LPIPS. We evaluate our methods mainly on the NeRF Synthetic dataset[] and DTU dataset[].

### 4.3 Evaluation

Static Rendering Table 1 provides a quantitative comparison of all NeRF Synthetic 8 cases between our method and competing methods. We conducted experiments using their official code repository.

Table: Table 1: Quantitative comparison of our methods with NeRF-Editing (N.E.)[] and SuGaR[] on NeRF Synthetic dataset in terms of SSIM, PSNR, LPIPS. ($\uparrow$ means higher is better, $\downarrow$ means lower is better.) The best results are marked in bold.

The numerical results of the NeRF-Editing were not presented in their original paper, we ran their code to provide more visual and numerical comparisons.
As depicted in Table 1, we observed some outliers in "Drums, Ficus" that were lower than even 10 PSNR compared to ours and SuGaR. Therefore, we remove these outliers with a strikeout in the table. As can be observed, our approach surpasses all the baseline methods with respect to PSNR, SSIM, LPIPS, which means we achieve the best rendering quality. We are 3.0 higher than SuGaR in PSNR, 0.013 higher in SSIM, and 0.024 lower in LPIPS.

Figure: Figure 3: Visual comparison between ours, NeRF-Editing[](N.E.) and SuGaR[] for static rendering. It illustrates our proposed method can contain a much more accurate boundary in "Ship" (first row), and detailed results in "Drums" (second row).
Refer to caption: x3.png

In Figure 3, we present qualitative results of our approach and other methods in overview and zoom-in details. Our methods render more detailed and more accurate boundaries. For SuGaR, it attempts to bind 3D Gaussians on the triangle and enforce that the attached Gaussian is closely aligned on the triangle. In practice, they set the scale $s_{1}$ along the triangle’s normal direction to a value close to zero, i.e., $s_{1}=\epsilon$. This binding strategy heavily depends on the accuracy of the mesh. As can be observed in the third column of Figure 3, wrong geometry leads to an inaccurate rendering, especially in the boundary region.

Figure: Figure 4: We provide a visual comparison of manipulation rendering between our proposed approach and SuGaR. Our approach exhibits fewer artifacts and less blurring effect compared to SuGaR. For more details, please zoom in.
Refer to caption: x4.png

Manipulation Rendering
In Figure 4, we showcase our manipulation results. In these four cases, we manipulate the underlying mesh with large deformation, the Chair is stretched, Lego is tapered, Ficus and Mic is bent to left and up respectively. As demonstrated in Chair, Ficus, we have more accurate boundary and shape, as well as in the bottom region. This indicates that when the geometry is not that accurate, SuGaR can not adapt to compensate for geometry error, which results in missing geometry and dilated boundary geometry. For Lego, Mic, we can maintain the high rendering quality even after the large deformation, while SuGaR shows some distortion and noise in rendering results after large deformation. We did not present the manipulation results of NeRF-Editing[] since we did not obtain their relatively reasonable results in these cases.

Figure: Figure 5: Visual results of local manipulation and soft body simulation. In row 1, we demonstrate that our proposed GS-mesh binding method enables us to support local part manipulation, such as blending, reposing, and elastic deformation. In row 2, we showcase a 3DGS soft body simulation demo at different timesteps.
Refer to caption: x5.png

In addition to the large deformation, our method also produces promising results for local manipulation and physics simulation. Here we show an example of soft body simulation. In Figure 5 row 1, we try to blend the red sauce and yellow sauce of Hotdog as shown in the blue box, which shows satisfying editing and reasonable rendering quality. In Drums, we repose a cymbal and elastically deform a cymbal as shown in the blue box. After reposing and elastic deformation, we still preserve the photo-realistic rendering results. Note that the manipulation is achieved by manipulating the triangular mesh directly, the 3DGS rendering is achieved simultaneously with self-adaption.

In the second row of Figure 5, we present the rendering results of soft body simulation at different timesteps. As observed, we can achieve 3DGS soft body simulation by just transferring the mesh simulation to 3DGS, which eliminates the need for a soft body simulation algorithm dedicated to 3DGS.

**Table 2: Quantitative ablation comparison between 3DGS On NeuS Mesh, NeuS Mesh + Offset, Ours with Marching Cube Mesh,Ours with Screened Poisson Mesh on NeRF Synthetic dataset. ($\uparrow$ means higher is better, $\downarrow$ means lower is better.)**
| Method | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |
| --- | --- | --- | --- |
| 3DGS On NeuS Mesh | 30.87 | 0.9521 | 0.0447 |
| NeuS Mesh + Offset | 32.48 | 0.9625 | 0.0341 |
| Ours + Marching Cube Mesh | 32.11 | 0.9602 | 0.035 |
| Ours + Screened Poisson Mesh | 33.42 | 0.9638 | 0.0324 |
| Ours + NeuS Mesh | 33.45 | 0.9646 | 0.0309 |

### 4.4 Ablation Study

We conduct ablation studies to verify the effectiveness of triangle shape aware Gaussian binding and adapting method. We first evaluate the strategy of directly binding 3DGS to the mesh, which implies that the 3D position is fixed on the triangle. As shown in Table 2, the performance significantly drops, with a decrease of approximately 2.6 PSNR compared to our best model. For visual ablation in Figure 6, 3DGS on NeuS Mesh after deformation shows a boundary with many burrs.

Next, we verify the effectiveness of adding 3D offset for 3DGS on Mesh. Although the offset can enhance the fitting of 3DGS to the static scene, as demonstrated in Table 2, it fails to generate satisfactory deformation rendering results because the offset only fits the static scene and remains unchanged during subsequent deformations. Consequently, it leads to significant noise and distortion after manipulation in Table 2.

Figure: Figure 6: Visual Results of Ablation Study. After deformation, (3DGS on Mesh) shows a burring boundary, (Mesh + Offset) leads to significant noise and distortion, (Ours Full) can maintain the high fidelity rendering. In the second row, we demonstrate that even with a low-quality mesh, we can still achieve high-quality editable rendering.
Refer to caption: x6.png

Finally, we conduct experiments with different meshes extracted using different methods. The 3DGS Marching Cube Mesh (3DGS MC) is of low quality, including a dilated boundary and very noisy surface, as can be observed in Figure 6 row 2. The screened poisson mesh (Poisson Recon.) has some unconnected regions and missing parts compared with NeuS mesh (NeuS). However, using our triangle shape aware Gaussian binding and adapting method can still achieve 3DGS manipulation and maintain high-fidelity rendering even after very large deformation as shown in Figure 6 row 2. As shown in Table 2, the numerical results obtained with screened poisson mesh are only slightly lower than those obtained with NeuS mesh. When the mesh is of low quality, such as the MC mesh, the quantitative results are approximately 1 PSNR lower than the best, but still 2 PSNR higher than only binding 3DGS on the best Mesh (NeuS mesh), and 1.5 PSNR higher than SuGaR.

We also evaluate our method using mesh extracted from SuGaR, whose mesh is extremely dense. The PSNR using SuGaR mesh is 33.67, which is about 3 PSNR higher than that of the original SuGaR results.

## 5 Conclusion and Limitation

In this paper, we introduce a triangle shape aware Gaussian binding strategy with self-adaptation, which supports various 3DGS manipulations, maintains rendering quality, and has a high tolerance for mesh accuracy. We evaluate our methods on the NeRF synthetic dataset and demonstrate state-of-the-art results, showcasing various 3DGS manipulations, including large deformations, local manipulations, and soft body simulations.

During our experiments, we noticed that some results still exhibit distortions. When the local region of the manipulated mesh contains highly non-rigid deformations, it can result in rendering distortions. Additionally, during our simulation demos, we found that conducting physics simulations on meshes with more than 35K triangles can take hours. It would be a novel direction to explore fast simulation methods for 3DGS.
Finally, we found that our results may not have accurate boundary rendering when the extracted mesh has a significant discrepancy from the ground truth mesh, such as unconnected regions.

Supplementary Material

In this supplementary material, we describe more implementation details of our method in Sec. A.
Then, we give an efficiency analysis of training time and inference time in Sec. B.
Besides, we also provide more visual rendering and geometry results on the NeRF[] Synthetic dataset including some video demos in Sec. C.
Finally, we further evaluate our methods on the DTU[] dataset and provide qualitative and quantitative results in Sec. D.

## Appendix A Implementation Details

### A.1 Training Details of Mesh Extraction Stage

As outlined in our main paper, the first stage of our approach involves mesh extraction. While we utilize the NeuS[] mesh as the foundation for binding Gaussians, we also explore extracting mesh from Gaussian-Splatting.

In this work, we try to extract triangular mesh using the Screened Poisson surface reconstruction [] method from trained Gaussian-Splatting model. We incorporate a normal attribute $\bm{n}$ for each 3D Gaussian and optimize the normal attribute with the pseudo-normal constraint.

The normal consistency is quantified as follows:

$$ $\mathcal{L}_{n}=\|\mathcal{N}-\tilde{\mathcal{N}}\|_{2}.$ (9) $$

where ${\mathcal{N}}$ is the rendered-normal map, $\tilde{\mathcal{N}}$ is the pseudo-normal map computed from rendered depth map.

Besides the normal constraint $\mathcal{L}_{n}$, the ordinary L1 Loss and Structural Similarity Index (SSIM) loss are also incorporated into optimization by comparing the rendered image $\mathcal{C}$ with the observed image $\mathcal{C}_{gt}$. To address the issue of unwarranted 3D Gaussians in the background region, we employ a mask cross-entropy loss. This loss is defined as follows:

$$ $\begin{split}\mathcal{L}_{mask}=-B^{m}\log{B}-(1-B^{m})\log{(1-B)},\end{split}$ (10) $$

where $B^{m}$ denotes the object mask and $B$ denotes the accumulated transmittance $B=\sum_{i\in{N}}T_{i}\alpha_{i}$.

Then all the loss terms can be summarized as follows:

$$ $\begin{split}\mathcal{L}_{stage1}=\lambda_{1}\mathcal{L}_{1}+\lambda_{2} \mathcal{L}_{SSIM}+\lambda_{3}\mathcal{L}_{n}+\lambda_{4}\mathcal{L}_{mask}, \end{split}$ (11) $$

where $\lambda_{1}=1,\lambda_{2}=0.2,\lambda_{3}=0.01,\lambda_{4}=0.1$. We train this stage for 30K steps with adaptive density control, which is executed at every 500 iterations within the specified range from iteration 500 to 10K. Once the training stage is complete, we proceed with Screened Poisson surface reconstruction using the positions and normals of the Gaussians as input. The mesh extraction process takes less than 1 minute to complete.

In addition to mesh extraction, we also utilize Gaussian-Splatting Marching-Cube to extract the triangular mesh. Our approach involves sampling a grid with a resolution of $256\times 256\times 256$. For each sampling point, we identify its nearest Gaussian points. Sampling points that have the nearest Gaussians within a pre-defined distance threshold $\tau$ are assigned a density value of 1, while those that do not meet the threshold are assigned a density value of 0. $\tau$ is set to 0.01 in practice. The density threshold for Marching-Cube is set to 1e-4.

Based on the visual comparison, the overall mesh quality can be ranked as follows: NeuS > Poisson Reconstruction > Marching-Cube.

### A.2 Training Details of Gaussian-Binding Stage

To ensure an accurate representation of each triangle, we bind $N$ Gaussians to it. Prior to training, we initialize the positions of the Gaussians on the attached triangle. The $N$ initialized position is calculated using a barycentric coordinate, with a predefined barycentric coordinate set of $[1/2,1/4,1/4]$, $[1/4,1/2,1/4]$, $[1/4,1/4,1/2]$. For the hyper-parameter $\beta$ mentioned in main paper equation (8), we set $\beta=10$ in most cases, $\beta=100$ in Materials.

In the Gaussian-Binding stage, we don’t perform adaptive control because we find it doesn’t influence the final performance. We also train 300K iterations in this stage with L1 loss, SSIM loss and mask entropy loss. The overall loss in this can be summarized as follows:

$$ $\begin{split}\mathcal{L}_{stage2}=\lambda_{1}\mathcal{L}_{1}+\lambda_{2} \mathcal{L}_{SSIM}+\lambda_{3}\mathcal{L}_{mask},\end{split}$ (12) $$

where $\lambda_{1}=1,\lambda_{2}=0.2,\lambda_{3}=0.1$.

## Appendix B Efficiency Analysis

The efficiency of GS Binding training and rendering speed depends on the number of Gaussians, which is the product of the triangle number $T$ and the Gaussians number for each triangle $N$. In Table C, We first fixed $T$ and tested different values of $N$. Our results indicate that $N$=3 leads to the best rendering quality while keeping a competitive rendering speed. When $N=1$, the PSNR slightly decreased with a faster training and rendering speed.

We also evaluated the impact of underlying mesh resolution by testing meshes with different triangles (270K, 150K, 70K). As shown in Table C, the rendering quality decreases while efficiency improves with decreasing mesh resolution.

Regarding the editing time, it primarily depends on the time cost of mesh editing. We use Blender for mesh editing, and in our experience, local manipulation and large deformation can be achieved instantly. Soft body simulation can be a more time-consuming process, as it depends on the simulation algorithm employed in Blender.

**Table C: Efficiency Analysis**
|  | $N$=4 | $N$=3 | $N$=1 | $N$=1 | $N$=1 |
| --- | --- | --- | --- | --- | --- |
| Triangles(K) | 270 | 270 | 270 | 150 | 70 |
| Points(K) | 1080 | 810 | 270 | 150 | 70 |
| Training (min) | 16 | 13 | 7 | 5.5 | 4.5 |
| Speed (FPS) | 244 | 300 | 452 | 571 | 572 |
| PSNR | 36.36 | 36.39 | 36.27 | 35.86 | 34.52 |

## Appendix C More Results on NeRF Synthetic Dataset

.
Soft Body Simulation.
In addition to the visual results presented in Figure 5 of the main paper, we also provide the geometry after simulation and rendering at different viewpoints in Figure G. To improve the speed of the mesh simulation, we decimated the original mesh from 300K to 35K triangles. While this may result in some decrease in rendering quality due to the reduced number of triangles as well as Gaussians, it was necessary to ensure reasonable simulation speed.

Demo Video. In order to further demonstrate the effectiveness of our methods, we have provided additional visual videos showcasing large deformation, soft body simulation, and local manipulation. These videos can be accessed through a local webpage by navigating to the mani-gs folder and clicking on index.html.

Numerical Comparison with NeuS and Neu-Mesh. We have compared our methods with NeuS and Neu-Mesh on the NeRF Synthetic dataset, and the results are presented in Table D. Our approach achieves a PSNR score that is 2.5 higher than Neu-Mesh, and a SSIM score that is 0.014 higher than Neu-Mesh, and a LPIPS score that is 0.012 lower than Neu-Mesh. These results demonstrate that our approach has achieved the best overall performance in all metrics compared to NeuS and Neu-Mesh on the NeRF Synthetic dataset.

Binding Gaussians on low-quality mesh. In some extreme cases, the Screend Poisson surface reconstruction method may result in a very low-quality triangular mesh. However, with our mesh-Gaussians binding strategy, we can still generate much better rendered images than those produced by SuGaR [], even though SuGaR may have a much better mesh in such case as shown in Figure H.

Figure: Figure G: Visual results of softbody simulation at different viewpoints. The left column displays the geometry after simulation, while the right three columns showcase the rendering results from three different viewpoints.
Refer to caption: x7.png

Figure: Figure H: Binding Gaussians on a low-quality mesh (ours on the left, SuGaR on the right), we are still able to achieve high-fidelity manipulated rendering results when the mesh we generate of Screened Poisson reconstruction is of low quality. In contrast, SuGaR fails to produce satisfactory results, even though it has a better mesh in this particular case.
Refer to caption: x8.png

## Appendix D More Results on DTU Dataset

**Table D: We compare quantitative rendering quality with NeuS [] and NeuMesh [] on the DTU dataset and the NeRF Synthetic dataset.**
| Methods | DTU | NeRF 360^∘ Synthetic |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ |  |
| NeuS [] | 26.352 | 0.909 | 0.176 | 30.588 | 0.960 | 0.058 |
| Neu-Mesh [] | 28.289 | 0.921 | 0.117 | 30.95 | 0.951 | 0.043 |
| Ours | 31.496 | 0.943 | 0.088 | 33.45 | 0.965 | 0.031 |

We have also evaluated our approach on the DTU dataset, which includes 15 cases with multi-view images as input. The results are presented in Table D, and demonstrate that our approach achieves a PSNR score that is 2.8 higher than Neu-Mesh, and a SSIM score that is 0.022 higher than Neu-Mesh. Additionally, our approach has a LPIPS score that is 0.016 lower than Neu-Mesh, indicating that our approach achieves the best overall performance on the DTU dataset.

We have also presented the manipulation results in Figure I. The left two columns showcase the geometry and rendered image before manipulation, while the right three columns showcase the geometry and rendered image after manipulation. In the first row, the wing of a pigeon is manipulated, while in the second row, the arms of a tiger are swinging down. In the third row, a toy is twisted to the left. The manipulation results presented in Figure I demonstrate that our approach can successfully transfer mesh manipulation to Gaussian-Splatting, resulting in accurate and visually appealing results.

Figure: Figure I: Manipulation rendering results in DTU dataset. The left two columns showcase the geometry and rendered image before manipulation, while the right three columns showcase the geometry and rendered image after manipulation. To highlight the deformed area, we have enclosed it within a red rectangle.
Refer to caption: x9.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 1
questions: - On line 505, you mention that the results using SuGaR mesh are 33.676 dB (You + SuGaR), which is higher than 33.34 (You + NeuS). Why use NeuS if this is the case? If this is the case, the contribution on a quantitative level does not seem to be significant.
- In Table 1, please add, if possible, the rendering results of NeuS so that it can be seen how much the authors improve on the work of NeuS.
soundness: 3
strengths: The strengths of this work rely mainly on addressing a highly relevant problem and achieving great values compared to their chosen state-of-the-art methods.
summary: The authors improve the methodology of manipulating renderings generated by 3D Gaussian Splatting (3DGS). To achieve this, they propose the use of triangular mesh (generated by NeuS) as initial input to the 3DGS. Additionally, the authors propose a triangle-aware 3DGS to improve the manipulation and rendering capability.
weaknesses: This work suffers from a few larger issues:

- Poor writing quality. Here, we mostly mean that the paper heavily introduces and talks about NeRF in the introduction and related work, while this is not relevant for understanding the paper. Further, structurally the paper needs some improvements (for example Figure is mentioned on page 4 but not seen until page 6)
- In general, while the method works decently, the contributions do not seem to be enough
- Compared to SuGaR, the author here uses better meshes that are generated from NeuS (higher training time). The authors should address the differences in the training in their work.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: Please check the weakness section. My main concern is the comparison with GaMeS or Mesh-GS.
soundness: 3
strengths: 1. The paper is well written and straightforward.
2. The paper proposes to bind Gaussians to a local triangle space, which maintains the local rigidity and preserves the relative
location between Gaussians, allowing the method to preserve the high-fidelity rendering results after manipulation.
2. The manipulation results are vivid and interesting, especially the soft body simulation.
3. The authors demonstrate the editability of the method on three different tasks, which shows the capability of the method in various scenarios.
summary: The paper utilizes a given triangular mesh for free-form shape deformation of Gaussian-splatting with self-adaption. By parameterizing each Gaussian in the local triangle space and decoupling the local and global transformations, the proposed method maintains the local rigidity and preserves the relative location between Gaussian, which is robust to inaccurate meshes. The authors demonstrate the method editability in three tasks, large deformation, local manipulation, and soft body simulation.
weaknesses: 1. The authors need to compare their method with GaMeS or Mesh-GS to demonstrate their contributions. In comparison to GaMeS which constrains the Gaussians on the surface exactly, the main contributions of Mani-GS are (1) attaching the Guassians to local space rather than global space, and (2) allowing Gaussians to offset out of the attached triangle. Could the authors provide some qualitative results that support those two contributions? In terms of the rendering quality given an inaccurate mesh and the rendering quality after manipulation. For example, given the Poisson mesh in Fig.8, where part of the pot is missing, Mani-GS can better fill the missing part than GaMeS since it allows the offset. And for example, show a case where the rendering quality of Mani-GS is better than GaMeS after manipulation due to the local triangle space.

2. In line 333 authors propose to use an adaption vector to scale both the offset vector and the scale of the Gaussian. However, the adaption vector solely depends on the length of the three triangle edges. Imagine stretching a triangle along its plane, e1, e2, e3 will all increase as the edge lengths get larger. Since e2 increases, the offset of the Gaussian along the triangle's normal direction will get larger, the Gaussian will move farther from the plane. A concrete example could be the Poisson mesh in Fig. 8, since part of the pot is missing, to be able to reconstruct the pot there must be lots of Gaussians that have large offsets along the normal directions, in that case if you stretch the pot vertically, I'd expect the Gaussians to expand horizontally as well. Does this lead to artifacts empirically? I'm happy to hear any comments on this. 

3. Is the mesh used in Table 1 extracted from SuGaR? If not what's the mesh used there and could you provide the results using the SuGaR mesh for a fair comparison? If yes seems the average PSNR is different from what is mentioned later in line 505.

4. Do you regularize the scale of the local position \mu? I'm concerned that a Gaussian could significantly offset the attached triangle, potentially causing artifacts after manipulation.

5. In line 44 NeRF-Editing is referred to as Yuan et al. 2022, but in the rest of the paper (for example lines 365 and 379) it becomes Liu et al. 2021. The former approach is more relevant for comparison, as it aligns more closely with the context of free-form shape deformation, which the latter approach does not directly address. Is it a typo?

---

confidence: 5
contribution: 1
initial_rating: 5
presentation: 3
questions: The training of 3DGS in the paper is conducted entirely in static scenes, which fails to effectively learn the correspondence that Gaussian and mesh should maintain during motion. If a 3D Gaussian is trained separately and then matched to the mesh surface (transforming coordinates from world space to the local space of each triangle), can good manipulation still be achieved?
soundness: 3
strengths: - The paper is well written, and the analysis is comprehensive.
- The idea of correlating the scale of 3D Gaussians with the shape of the triangles to better handle large-scale deformations is reasonable.
- The experimental results appear to be valid.
summary: This work proposes a method for 3D Gaussian manipulation using a mesh as a proxy. By defining a local coordinate system for each triangle, the paper associates Gaussians with each triangle in a self-adaptive manner. The paper is clearly illustrated and thoroughly demonstrated.
weaknesses: - 3D Gaussian Spatting achieves high-quality rendering results primarily due to its split/clone mechanism, which adaptively adjusts the number of points in the scene. However, this paper limits the number of Gaussians in each triangle face, which may restrict its fitting capability. Nevertheless, the rendering metrics in Table 1 appear to be very high, with some even exceeding those of the original 3DGS; this raises questions.
- The main innovation of this paper lies in the introduction of $e$ in Equation 7 to better handle large-scale deformations. However, this is not evident in the ablation study. In fact, both the 3DGS on Mesh and Mesh + Offset experiments seem not to address the rotation of Gaussians, which is unreasonable.
- The current experimental examples are focused on hard surfaces. However, the greater advantage of 3DGS compared to meshes lies in rendering scenes without well-defined surfaces. How does this method perform on fuzzy geometry (e.g., the data from "Adaptive Shells for Efficient Neural Radiance Field Rendering")?

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Can you please summarize additional recent baselines, including Gaussian Frosting, and compare against those?

2. Please show large deformations or avoid claiming them.
soundness: 3
strengths: The paper is reasonably well written and easy to understand. It addresses the important challenge of editing 3D scenes represented by 3D Gaussians. The qualitative results look compelling and quantitatively outperform the sugar baseline.
summary: This submission describes an approach for deforming a 3D scene represented by 3D Gaussians. Towards this goal, the proposed method extracts a triangle mesh, binds the 3D Gaussians representing the scene to the mesh (on- and off-surface), and then uses the mesh to drive rigid and non-rigid deformations of the 3D Gaussians for object deformation.
weaknesses: There are several weaknesses:

1. The proposed methods is incremental compared with sugar. The proposed method is basically sugar, which binds the optimized 3D Gaussians to the mesh surface. with an additional offset. This seems like a simple extension. More advanced extensions of sugar already exists, including 

Guedon and Lepetit, Gaussian Frosting: Editable Complex Radiance Fields with Real-Time Rendering, ECCV 2024

which model a much broader class of objects than both sugar and the proposed work.

2. The related work discussion is too focused on NeRF, instead of giving a more comprehensive snapshot of approaches that enable animatable / deformable 3D Gaussians. A few examples:

Huang and Yu, GSDeformer: Direct Cage-based Deformation for 3D Gaussian Splatting
Abdal et al., Gaussian Shell Maps for Efficient 3D Human Generation, CVPR 2024
Yang et al., Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction, CVPR 2024

So relevant papers published at CVPR 2024, ECCV 2024, and also SIGGRAPH Asia 2024

3. Some claims on the capabilities of the proposed system seem exaggerated

The presented results look good, but they mainly show local and non-rigid deformations. I did not see examples that show the claimed "large deformations" (see g.g. abstract & introduction)

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. There are some citation errors, such as the reference to Mesh-GS (Waczynska et al., 2024), which actually pertains to the GaMeS paper.
2. Have you tried testing your method on real-world data with backgrounds, such as the LLFF dataset? How effective is it in such cases?
soundness: 2
strengths: 1. The article has a clear logic and provides an in-depth analysis of the problem. For example, when discussing how to bind Gaussians to the mesh (Sec.3.3), the authors compared two alternative methods ("Gaussians on Mesh" and "Gaussians on Mesh with Offset"), analyzing the principles, advantages, and disadvantages of each. Another example is the authors' discussion of the results from different mesh extraction methods (Sec.3.2). 
2. The supplementary materials are meticulously prepared, and the demo presentation is impressive, showcasing excellent results on editing and simulating.
summary: This article focuses on the editing of Gaussians. Its central idea is to first perform mesh modeling of the scene, then bind 3DGS to the mesh for topology-consistent training. To better bind Gaussians to the mesh’s triangular surfaces, the paper proposes a coordinate system definition method based on triangles, allowing the topology to maintain a more stable structure. Once completed, this enables Gaussian editing and simulation similar to mesh manipulation. The authors conducted experiments on NeRF synthetic data and DTU data, achieving the expected editing effects.
weaknesses: 1. The discussion of some works is insufficient. For example, GaMeS and Mesh-GS are mentioned in the related work section, but as the most closely related and recent Gaussian methods, they are not included in the experimental comparisons. Methodologically, I feel that the Gaussian binding approach in this article is very similar to that of GaMeS, yet the authors do not discuss this point. The baselines the authors compare are outdated and are insufficient to demonstrate the superiority of their method.
2. The range of data types this article's method can be applied to is not diverse enough. From the authors' experiments, it currently only supports the editing of small objects and relies heavily on the topology mesh obtained from mesh reconstruction algorithms (e.g., NeuS). If the object becomes more complex or includes a complex background, this approach is likely to produce a degraded-quality mesh, making it impossible to proceed with subsequent binding operations.