Title: D-MiSo: Editing Dynamic 3D Scenes using Multi-Gaussians Soup
Venue: NeurIPS 2024
Accepted: yes
Keywords: Neural rendering

Gaussian splatting

3D

Triangle Soup
Abstract: Over the past years, we have observed an abundance of approaches for modeling dynamic 3D scenes using Gaussian Splatting (GS). These solutions use GS to represent the scene's structure and the neural network to model dynamics. Such approaches allow fast rendering and extracting each element of such a dynamic scene. However, modifying such objects over time is challenging. SC-GS (Sparse Controlled Gaussian Splatting) enhanced with Deformed Control Points partially solves this issue. However, this approach necessitates selecting elements that need to be kept fixed, as well as centroids that should be adjusted throughout editing. Moreover, this task poses additional difficulties regarding the re-productivity of such editing. To address this, we propose Dynamic Multi-Gaussian Soup (D-MiSo), which allows us to model the mesh-inspired representation of dynamic GS. Additionally, we propose a strategy of linking parameterized Gaussian splats, forming a Triangle Soup with the estimated mesh. Consequently, we can separately construct new trajectories for the 3D objects composing the scene. Thus, we can make the scene's dynamic editable over time or while maintaining partial dynamics.

D-MiSo: Editing Dynamic 3D Scenes using Multi-Gaussians Soup

Abstract

Over the past years, we have observed an abundance of approaches for modeling dynamic 3D scenes using Gaussian Splatting (GS). These solutions use GS to represent the scene's structure and the neural network to model dynamics. Such approaches allow fast rendering and extracting each element of such a dynamic scene. However, modifying such objects over time is challenging. SC-GS (Sparse Controlled Gaussian Splatting) enhanced with Deformed Control Points partially solves this issue. However, this approach necessitates selecting elements that need to be kept fixed, as well as centroids that should be adjusted throughout editing. Moreover, this task poses additional difficulties regarding the re-productivity of such editing. To address this, we propose D ynamic M ult i -Gaussian So up (D-MiSo), which allows us to model the mesh-inspired representation of dynamic GS. Additionally, we propose a strategy of linking parameterized Gaussian splats, forming a Triangle Soup with the estimated mesh. Consequently, we can separately construct new trajectories for the 3D objects composing the scene. Thus, we can make the scene's dynamic editable over time or while maintaining partial dynamics.

1 Introduction

Recently introduced Gaussian Splatting (GS) [1] represents the 3D scene structure through Gaussian components. We can combine GS with the neural network (i.e., deform network) to model dynamic scenes [2]. This approach involves the joint training of both the GS components and the neural network. GS characterizes the 3D object's shape and color, while the neural network utilizes time embedding and Gaussian parameters to generate updated initial positions to model dynamic scenes. Such an approach allows for fast rendering and extracting each element of a dynamic scene.

Most existing methods can effectively model dynamic scenes, but generating new 3D objects'

positions remain challenging. Consequently, we cannot edit objects over time when using such

reconstruction

Render

Sub-Triangle

Soup

Modifications

Sub-TriangleSoup

Modified

gV

Core-Gaussians

gV;

Core-Triangle Soup

Core-TriangleV

gV,

gV,

U3

Sub-Gaussians

gv

Sub-TriangleVj

Multi-TriangleSoup

approaches. To tackle this issue, SC-GS (Sparse Controlled Gaussian Splatting) [4] uses Deformed Control Points to manage Gaussians. After the training phase, we can manually modify the model at any point in time. However, this method requires identifying elements to remain static and adjusting 3D objects' centroids (nodes) during editing when a relationship between the selected nodes is visible. For example, by moving the humanoid 3D model's hand, the part of the head or leg is also changed.

To address this issue, we introduce D ynamic M ult i -Gaussian So up (D-MiSo), which is easier to modify (Fig. 1) and obtain renders comparable to SC-GS. D-MiSo estimates the mesh as Triangle Soup, i.e. a set of disconnected triangle faces [5, 6], and uses a dynamic function to control the vertices.

D-MiSo employing Multi-Gaussians, defined as larger Core-Gaussians encompassing smaller ones termed Sub-Gaussians (Fig. 2). SubGaussians are defined in the local coordinate system given by principal components of CoreGaussian. Therefore, by modifying Core-

Renders

Modifications

View

2

View

C

Gaussian, we change all Sub-Gaussians, which allows scene modifications (Fig 3). Core-Gaussians are an alternative to the control points discussed in [4], with the added advantage of allowing individual modifications. Consequently, there is no necessity for static and dynamic markers. In Fig. 4, we present the difference between modification applied by SC-GS and D-MiSo.

Our model uses flat Gaussians. Therefore, based on GaMeS [3], we can approximate Gaussian components using triangle face mesh by parameterizing Gaussian components by the vertices of the mesh face. We denote such transformation by T ( · ) . In practice, as a consequence of parameterizing each Gaussian, we obtain a cloud of triangles called Triangle Soup [5]. Using Triangle Soup, we can control two types of Gaussian components. Accordingly, in D-MiSo, we get: Sub-Triangle Soup, Core-Triangle Soup and Multi-Triangle Soup (Fig. 2, Core-Triangle Soup is marked by red color, and Sub-Triangle Soup is denoted in blue). In D-MiSo, we can select and modify one part of the object like a mesh. In contrast, using SC-GS, static and dynamic points have to be selected, and editing only one part of the object is difficult.

During training, the positions of Core-Gaussians are managed by deformation multilayer perceptron (MLP), while the Sub-Gaussians are collectively manipulated through global transformation and small local deformation. The former describes the general flow of objects in the scene, while the local deformation is responsible for modeling small changes like shadows and light reflections. After training, we can modify our model directly by using the vertex of the Sub-Triangle Soup, or we can generate mesh from the Core-Triangle Soup (Fig. 5).

The contributions of this paper are significant and are outlined as follows:

We introduce the Multi-Gaussian components, which consist of a single large Gaussian response for global transformations and many small components dedicated to rendering. Multi-Gaussian components allow for the modeling of large 3D scene elements.

We propose D-MiSo a model that uses Multi-Gaussian components and two deformation networks for modeling dynamic scenes.

Apparent deficiencies

Too few selected static points

in modeling of hand

element dynamics

S

SC-GS

Render

Nodes

Proper

Challenges incontrolling movingelements andmanaging separability

modification

precise selection of

moving parts

D-MiSo

Render

Sub-Triangle

Hand Rotation

Effortlessmodificationofstaticandmovingelements

Soup

Our D-MiSo allows an object to be edited at a selected moment in time. The edited components are independent, and the editing does not affect other parts of the object. In addition, it also allows for full or partial dynamics to be maintained. Modifications also include scaling and rotation.

2 Related Works

Recent advancements in view synthesis, particularly driven by NeRFs [7], have significantly contributed to the rapid development of novel view synthesis techniques. However, the majority of these approaches model static scenes implicitly using MLP. Moreover, several works have extended classical NeRF to dynamic scenes through the use of deformation fields [8, 9, 10] and [11]. The alternative approaches, such as [12] and [13], represent scenes as 4D radiance fields. Early works on dynamic scenes face difficulties when dealing with monocular settings and uncontrolled or lengthy scenarios. To enhance scene motion modeling, some works utilize flow-based techniques [10, 14, 15]. However, NeRF-based solutions often suffer from long training and rendering times. To address this, grid-plane-based methods [16, 17, 18] have been proposed. In addition, several NeRF-based approaches have also been extended for scene editing purposes [19, 20, 21, 22].

The recently introduced Gaussian Splatting (GS) technique [1] addresses many limitations of other methods, offering multiple advantages due to their explicit geometry representation, enabling easier dynamics modeling. The efficient rendering of the 3D version of GS also avoids densely sampling and querying neural fields, making downstream applications such as free-viewpoint video reconstruction more feasible. A notable extension of 3D GS was proposed in [23], where the authors introduced the concept of anchor points to tackle the problem of overfitting caused by redundant Gaussians. Scaffold-GS addresses this issue by distributing local 3D Gaussians according to anchor points. This approach reduces redundancy, enhances scene coverage, and maintains high-quality rendering with improved robustness to view changes. While the original GS was developed for static scenes, several extensions for dynamic scenes were proposed. Most of the early works operate in multiview setup [24, 25, 26]. For example, [26] utilizes a frame-by-frame approach to model each timestep. However, this method lacks inter-frame correlation and requires high storage overhead for long-term sequences. In [27, 2], MLP is introduced to model changes in Gaussians over time, and in [28] MLP together with decomposed neural voxel encoding algorithm are utilized for training and storage efficiency. In [29], dynamic scenes are divided into dynamic and static parts, optimized separately and rendered together to achieve decoupling. [24] spacetime Gaussian proposes approximating the spatiotemporal 4D volume of a dynamic scene by optimizing a collection of 4D primitives with explicit geometry and appearance modeling. This method uses 4D Gaussians parameterized by anisotropic ellipses and

V

V'

Vgv

V2

V2

V3

Renderobjectattimet

ForeachSub-Gaussianchosethe

TransformGaussiansusing

Vi

Rendermodificationattimet;

closestestimated-meshfaceV;

view-dependent, time-evolved appearances represented by 4D spherical harmonics coefficients. In [25], a novel, real-time and photorealistic scene representation called Spacetime Gaussian Feature Splatting has been introduced. This approach extends 3D Gaussians with temporal opacity and parametric motion/rotation, enabling the capture of static, dynamic, and transient scene content. Additionally, the method incorporates splatted feature rendering to model view- and time-dependent appearances while maintaining a compact representation size. The method is notable for its high rendering quality and speed while also being storage-efficient. Other works enhance dynamic scene reconstruction using external priors. For example, the diffusion priors can be used as regularization terms during optimization [30].

Furthermore, GS was employed for mesh-based scene geometry editing. In [31], 3D Gaussians are defined over an explicit mesh and utilize mesh rendering to guide adaptive refinement. This approach depends on the extracted mesh as a proxy and fails if the mesh cannot be extracted. In contrast, in [32], explicit meshes are extracted from 3D GS representations by regularizing Gaussians over surfaces. However, this method involves a costly optimization and refinement pipeline. Another example of [33] employs sparse control points for 3D scene dynamics, but this method struggles with intense edit movements and necessitates accurate static node selection. Also, [3] combines GS with mesh extraction. However, such an approach only works for static scenes.

The method proposed in [34] combines meshes with GS and reconstructs a high-fidelity and time-consistent mesh from a single monocular video. However, it relies on a Poisson Solver and differentiable Marching Cubes to recover the deformed surface, significantly complicating the pipeline. Moreover, it does not explore geometry modification capabilities, which constitute a significant aspect of our work. Conversely, cage-based methods [35, 36] are an intuitive tool for geometry manipulation. However, they require additional steps for cage-building and may lack the flexibility and precision of manual, vertex-level deformation techniques, potentially missing fine details.

02

Q&2

2

GVi

&

1

2

2

3

In contrast to the listed approaches, we propose a D-MiSo, a mesh-based method specifically designed to handle dynamic scenes. D-MiSo leverages a straightforward pipeline of GS techniques to enable real-time editing of dynamic scenes.

3 Dynamic Multi-Gaussian Soup

Here, we present the main components of D-MiSo. We start with the classical GS to provide the foundations for our model. Next, we introduce the concept of Multi-Gaussians and describe how to

Stage1

Stage2

Renders

Animation

t1

Network

Deform

→(△，,，△R）t

→

t3

Network

Deform

→(△，△u,,△R)t

Network

Deform

Network

Deform

H

Network

Sub-Rot

Network

Sub-Rot

Sub-Gaussians

update Core-Gaussians

batch

update Core-Gaussians

(△,△，△△R）

(△,，△，△R)

update Sub-Gaussians

→(△R’)i∈[1,2,3}

→(△R'),i∈{1,2,3}

multi-view images

camera information

6④

time t;

+ Sub-Gaussians

9v

5

gv:

U3

Core-Gaussians

Multi-Gaussians

Stage2-preparation

estimate the mesh for editing. Finally, we show D-MiSo, which uses Multi-Gaussians in dynamic 3D scenes.

Gaussian Splatting The Gaussian Splatting (GS) technique models 3D scenes using an array of 3D Gaussians, each specified by its mean position, covariance matrix, opacity, and color expressed using spherical harmonics (SH) [37, 38]. The GS algorithm constructs the radiance field by iteratively optimizing the parameters of all Gaussian components. Ultimately, the GS efficiency mainly depends on its rendering method, which involves projecting Gaussian components.

The GS framework employs a dense collection of 3D Gaussians: G = { ( N ( m i , Σ i ) , σ i , c i ) } n i =1 , where m i denotes the position, Σ i the covariance, σ i the opacity, and c i the SH colors for the i -th Gaussian. The GS optimization process involves a repetitive cycle of rendering and comparing the resultant images with the training views. In our work, we will use Multi-Gaussian approaches (Fig. 6).

Multi-Gaussians Multi-Gaussians G multi are dedicated to describing relatively large parts of the 3D scene to allow modification of large blocks instead of modifying each small Gaussian separately. The Multi-Gaussian model comprises a primary large 3D Gaussian (referred to as the Core-Gaussian G core ), which encompasses numerous smaller Gaussians (termed Sub-Gaussians G sub ), all of which are parameterized by the main Core-Gaussian. MultiGaussianare is similar to anchor Gaussians from [39], but we do not use a neural network to produce child components. We parametrize Sub-Gaussians in a local coordinate system.

V2

v2

gv

△R

U3

g'v

V3

time ti

Similarly to classical GS, we parameterize the Core-Gaussian distribution by center m and the covariance parameterized by factorization: Σ = RSSR T , where R is the rotation matrix and S the scaling parameters. More precisely we consider p Core-Gaussians uses flat Gaussians as in [3], and is defined by:

where S = diag( s 1 , s 2 , s 3 ) , s 1 = ε and R is rotation matrix of Core-Gaussian defined as: R = ( r 1 , r 2 , r 3 ) , where r i ∈ R 3 which can be interpreted as a local coordinate system used by Sub-Gaussian.

It is worth noting that Sub-Gaussian can be interpreted as a child of Core-Gausian. We define centers of Sub-Gaussian N sub ( m i , R i , S i ) in the local coordinate system of Core-Gaussian N core ( m , R, S ) by: m i = m + Rα i α i α iT , where m , R is Core-Gaussian position and rotation; and α i α i α i = ( α i 1 , α i 2 , α i 3 ) are trainable parameters used to define the positions of the Sub-Gaussian relative to the Core-Gaussian (Fig. 6). Sub-Gaussians are used for rendering and can be seen as a main component of our model:

where m , R, S are parameters of Core-Gaussian and, α i , S i , R i marks parameters of i -th SubGaussians with opacity σ i , and SH colors c i .

Core-Gaussians is generally dedicated mainly to transformations, hence, in practice opacity or colors are not used during rendering. On the other hand, Sub-Gaussian is devoted to rendering and modification. It has its own opacity and colors.

GaMeS parametrisation of Multi-Gaussian component Multi-Gaussians describe 3D scenes using solid blocks rather than tiny Gaussian distributions. This method enhances our model's suitability for dynamic environments. One of our mode's most important properties is its ability to model dynamic scenes in each time step. To obtain such properties, we parameterize all Gaussian using Triangle Soup following similar approach as in GaMeS [3]. Thanks to a few simple transformations, we can convert the mean m rotation R and scaling S into triangle V = ( v 1 , v 2 , v 3 ) , which parametrizes Gaussian distribution. Such transformation is unambiguous and reversible.

d2<d3

d2<d1

d2

d1

Estimated Mesh

Sub-Gaussian

Let us assume that we have a Gaussian component parameterized by mean m , rotation matrix R = [ r 1 , r 2 , r 3 ] and scaling S = dig( ε, s 2 , s 3 ) . We define three vertex of a triangle (face): V = [ v 1 , v 2 , v 3 ] , , where v 1 = m , v 2 = m + s 2 r 2 , v 3 = m + s 3 r 3 .

Now we froze the vertex of the face V = [ v 1 , v 2 , v 3 ] and reparameterize the Gaussian component by defining ˆ m , ˆ R = [ˆ r 1 , ˆ r 2 , ˆ r 3 ] and ˆ S = dig(ˆ s 1 , ˆ s 2 , ˆ s 3 ) . First, we put ˆ m = v 1 . The first vertex of ˆ R is given by a normal vector:

where × is the cross product. The second one is defined by ˆ r 2 = ( v 2 -v 1 ) ∥ ( v 2 -v 1 ) ∥ . The third one is obtained as a single step in the Gram-Schmidt process [40]:

Scaling parameters can also be easily calculated as s 1 = ε , ˆ s 2 = ∥ v 2 -v 1 ∥ and ˆ s 3 = ⟨ v 3 -v 1 , ˆ r 3 ⟩ . Consequently, the covariance of Gaussian distribution positioned on face is given by:

Initposition

D-Miso

SC-GS

withSub-TriangleSoup

Modification

Modification

Nodes

Init

and correspond with the shape of a triangle V . For one face ( v 1 , v 2 , v 3 ) , we define the corresponding Gaussian component:

Reconstruction

ModificationbyestimatedMeshchange

ModificationbyTriangles Soup(Sub-Gaussians)

directly change

Core-Gaussians

Sub-Gaussians

Estimated Mesh

EstimatedMesh&

Sub-Gaussians

Finally, the Gaussian component is derived from the mesh face parameters. This approach can be applied in a Multi-Gaussian framework. Therefore, we will use invertible transformation between Gaussian parameters and triangle face T and notation:

In our D-MiSo we parameterize the p Core-Gaussians with k Triangle soup:

where V j , α i α i α i , R i , ˆ S i , σ i , c i are trainable parameters and ( ˆ m V j , ˆ R V j , ˆ S V j ) = T -1 ( V j ) . Alternatively, we can parameterize Core-Gaussian and Sub-Gaussians by Triangle Soup:

where ( ˆ m V j , ˆ R V j , ˆ S V j ) = T -1 ( V j ) , V i = T ( ˆ m V + ˆ R V α i α i α iT , ˆ R i , ˆ S i ) .

In D-MiSo, we use the collation of Multi-Gaussian distribution for rendering and Sub-Triangle Soup for editing. The formal definition of our model uses equation (3) since, in training, we store CoreGaussian as a triangle face (Core-Gaussian does not have colors) and Sub-Gaussian as a collection from classical GS components with color and opacity. After training, we parametrize our model to equation (4) for editing.

3.1 Dynamic Multi-Gaussian Soup (D-MiSo)

Previously, we defined Multi-Gaussians and their parametrization using Triangle Soup. Now, we have all the tools to present the D-MiSo model. The overview of our method is illustrated in Fig. 7. The input to our model is a set of images of a dynamic scene, together with the time label and the corresponding camera poses. Our training is divided into two stages. In the first stage, we initialize the Core-Gaussians. In the second, we add Sub-Gaussian components.

Stage 1 First, we train only Core-Gaussians to obtain good initialization for Multi-Gaussins. As Core-Gaussians are mainly employed to capture motion, our model only requires their small amount (Fig. 7). In our approach, the Core-Triangle Soup, constructed via the Core-Gaussians parameterization (as depicted in Fig. 2), is adjusted depending on the time t .

In practice, when random initialization of Gaussians is necessary, redundant Gaussians must be pruned first to ensure that the remaining ones represent the object's shape. To reduce the number of Gaussians

and obtain consistent Core-Gaussians, we train GS on a batch containing a few views instead of one. In practice, we render a handful of views from different positions and use back-propagation.

In particular, we parameterize Gaussians N ( m , R, S ) by face V = ( v 1 , v 2 , v 3 ) to obtain N ( v 1 , v 2 , v 3 ) . Our Deform Network takes as in input the triangle vertices V assigned to the parameterized 3D Core-Gaussians and the current time t and returns updated ψ ( V, t ) = (∆ v 1 ( t ) , ∆ v 2 ( t ) , ∆ v 3 ( t ) , ∆ R V ( t )) . Such updates consist of translation and rotation (Fig. 8):

D-MiSo parameterized Core-Gaussians in time t by: N core ( V ( t ) , σ, c ) = N core ( V ⊙ ψ ( V, t ) , σ, c ) where ψ is a deformable network that moves triangle V according to time t . The opacity and color of the Core-Gaussian are used only in the first stage.

Stage 2: Preparation To move to the second phase of the model, it is imperative to prepare the Multi-Gaussians. This involves attaching k Sub-Gaussians to each Core-Gaussian generated in Stage 1, as shown in Fig. 6. Henceforth, SubGaussians assume responsibility for the resultant rendering. Initially, the Sub-Gaussian adopts the same features as the Core-Gaussian, except for the position.

Stage 2 The primary objective of the second phase is to parallelized Core-Gaussians (Core-Triangles Soup) to enhance understanding of movement and increase rendering quality through the precise training of Sub-Gaussians.

Init position

Small elements animations

Since the centers of Sub-Gaussians are parameterized by the local coordinate system given by the rotation matrix of Core-Gaussian, when the Deform Network ψ changes the Core-Gaussian, all Sub-Gaussians (attached to this Core-Gaussian) are modified by global transformation ψ ( V, t ) .

D-MiSo use an additional deformation network Sub-Rot Network ϕ dedicated to each i -th SubGassian's small changes. Sub-Rot Network takes the Sub-Gaussian rotation matrix R i V and the current time t as input and produces an updated rotation matrix ∆ R i V ( t ) .

The position ˆ m i V ( t ) = ˆ m V ( t ) + ˆ R V ( t ) α i α i α iT of the Sub-Gaussian in time t is determined by the position ˆ m V ( t ) , and rotation ˆ R V ( t ) of the Core-Gaussian (parameterized by triangle V ) and the learning parameter α i α i α i . It should be noted that scale S i , color c i , and opacity σ i of Sub-Gaussian are trainable and do not depend on time. Sub-Rot Network produce updated ϕ ( R i , t ) = ∆ R i ( t ) for rotation parameter of Sub-Gaussians. Finally parameters of SubGaussians in time t depends on Deform Network

ψ , and Sub-Rot Network ϕ , and the Corr-Gaussians parameter V , and Sub-Gaussian parameters R i , S i , c i and σ i :

D-MiSo final model consists of two deformable networks and two levels of Gaussian distributions. This approach enables efficient modeling of object motion over time in dynamic scenes, allowing for adjustments to objects at each time frame.

The result of the model's inference depends on the camera's view angle as well as on the selected time. Only Sub-Gaussian features are used in the rendering process. Hence, the implementation of generating an output image from Gaussians is no different from a vanilla GS.

The modifications involve the appropriate transformation of Sub-Gaussians. We base this on the fact that the transformed Gaussians have exactly the same color properties as before editing. Only the shape of the Gaussians is changed. In our work, we present three editing methods (Fig. 11).

4 Experiments

The experimental section is divided into two parts. First, we show that D-MiSo can model dynamic scenes with high quality; in the second part, we demonstrate that our model allows an easy editing procedure, which is our main contribution.

4.1 Reconstruction of dynamic scenes

Here, we outline the specifics of our implementation and provide a detailed description of the used datasets. We demonstrated the core advantages of our model by conducting experiments using three different datasets. The source code

PSNR

↑

Hook Jumpin. Trex

Bounc. Hell.

D-NeRF [11]

29.25 32.80

31.75 38.93

25.02 31.29

32.79

TiNeuVox-B [41] 31.45 34.23

32.70 40.73

28.17 33.61

35.43

Tensor4D [18]

29.03 24.01

23.51 25.36

31.40 29.99

30.86

K-Planes [17]

28.59 32.27

31.41 40.61

25.27 33.79

34.31

FF-NVS [42]

32.29 33.55

30.71 40.02

27.71 34.97

36.91

4D-GS [28]

30.99 33.59

32.16 38.59

31.39 35.98

35.37

DynMF [2]

33.94 38.04

35.82 41.92

37.51 41.68

41.16

Deform-GS [43]

37.77 39.10

38.40 41.46

42.11 43.73

45.38

Editable

SC-GS [33]

39.87 41.13

41.24 44.91

42.93 45.19

47.89

D-MiSo (our)

38.13 42.05

40.88 41.49

41.49 44.38

47.66

is available on 3 . Our code is developed on top of the GS vanilla code, according to their license. We used NVIDIA GeForce RTX 4090 and A100 GPUs. The experiments focus on a benchmark task of reconstruction. PSNR metric is used to compare our model with other methods. Furthermore, additional numerical comparisons, i.e., SSIM/LPIPS, are available in the supplementary material.

D-NeRF Datasets: Contains seven dynamic objects with realistic materials described with a single camera [11]. This means the model had access to only one view at a given moment. Tab. 1 shows we are very comparable to other methods, and we achieve a higher PSNR on one object. The differences in metrics are small, however, our method presents an easier way to edit (Fig. 4). Providing, among other things, better scaling. Following previous methods, our results are obtained on images using 400 by 400 resolution with a black background.

NeRF-DS[44]: This dataset contains again seven real-world scenarios containing a moving or deforming specular object. Each scene was recorded using two cameras, with the video captured by one of them being used as a training set, while the footage from the second one was treated as a test set. The camera pose was estimated using COLMAP for both cameras. Tab. 2 shows that our method achieves the SOTA results for distinct objects.

PanopticSports Datasets: The dataset comprises six dynamic scenes featuring significant object and actor movements [26, 45]. The scenes are categorized by the activities performed in the video sequences, i.e., juggling, boxing, softball, tennis, football, and basketball. Each scene was recorded using 31 cameras over 150 timesteps. Following the official data split, we use footage from 27 cameras for training and the remaining 4 cameras for testing. Numerical results are shown in Tab. 3. The results show that the model achieved SOTA for five objects according to PSNR metrics and six objects according to LPIPS metrics.

These results demonstrate that D-MiSo is comparable to other methods. Moreover, our main contribution in comparison to other methods is the very easy editing of the resulting object (Fig. 4 11).

4.2 Editing of dynamic scenes

As mentioned earlier, we proposed three new methods of output object modification. The first one focuses on moving and preparing a formal mesh, which is connected in contrast to Triangle Soup. We apply a simple meshing strategy on Core-Gaussians. We used the basic Alpha Shape algorithm [46, 47] and showed that the mesh does not have to estimate the surface perfectly to be effective. Then, we reparametrize the Sub-Gaussian by finding the closest face from the mesh (Fig. 9). In practice, we represent each Sub-Gaussian in a local coordinate system (analogically to MultiGaussians). Therefore, each Sub-Gaussian is assigned to the nearest face (Fig. 9). Each face can have a different number of Sub-Gaussians prescribed in such a configuration. This method allows us

to maintain the consistency bestowed by the mesh. The whole process is shown in Fig. 5. Thanks to such representation, we can edit our connected mesh to produce the correct edition of the dynamic scene (Fig. 2).

The second editing method allows us to define the connections, i.e., editing Sub-Triangles Soup directly, e.g. moving a hand or bending horns (Fig. 10) or rotating a human body (Fig. 3). Changes are possible because we can also transform a group of Sub-Triangles instead of individual ones. This method allows for even very subtle changes like raising the thumb (Fig. 10), turning the hand over (Fig 4), opening or closing the hand (Fig. 12). These edits would be difficult in other approaches based on adjusting the 3D objects' centroids (nodes) since the

Table 2: PSNR comparisons on a NeRF-DS dataset showing that D-MiSo gives comparable results with other models.

PSNR

↑

Bell Sheet Press Basin Cup Sieve Plate

HyperNeRF [9]

24.0 24.3

25.4

20.2

20.5 25.0

18.1

NeRF-DS [44]

23.3 25.7

26.4

20.3

24.5 26.1

20.8

TiNeuVox-B [41] 23.1 21.1

24.1

20.7

20.5 20.1

20.6

Editable

SC-GS [4]

25.1 26.2

26.6

19.6

24.5 26.0

20.2

D-MiSo (our)

25.3 25.8

25.6

19.8

24.5 26.5

20.8

space of nodes is limited in details area (Fig. 4). Nodes' use is preeminent for defining movement, which was insignificant in these places.

With this method, we can also change complicated objects like a 360 ◦ scene without losing the dynamics-related model. For example, the rotation of a person stacking boxes (Fig. 3).

The third method focuses on transforming the space in which the object is located. Similar to the second method, the third one also works directly on the SubTriangle Soup. We can achieve such an effect by applying a certain function to the selected plane. In practice, the object, or a portion of it, is modified, for instance, through the use of sinusoidal functions. It allows us to obtain fluidity and more easily define the physical nature of the movement (Fig. 1, 11, 13).

Our methods are also scalable, so we can easily remove or duplicate elements from an image. Moreover, the duplicated elements can be given their own dynamics. Examples of these effects are shown in Fig. 13, where we removed the plate and both duplicated and rescaled the blue balls multiple times.

5 Conclusion

D-MiSo is a novel method based on Gaussian Splatting parameterization, which produces a cloud of triangles called Triangle Soup. The method allows easy editing of objects created in inference with the possible transformations, including moving, scaling, and rotating. By defining Multi-Gaussians, the obligatory separability of modified elements seen in other models is combated. In addition, certain elements of objects can be duplicated and removed (Fig. 13). Furthermore, the D-MiSo method can facilitate giving different dynamics to separate parts of an object (Fig. 1).

Limitation The method allows for complex changes at a given moment in time. However, if some area is not well represented in the training set, it is impossible to edit them. For example, a person's hand can be changed but not fingers (Fig. 4). This is due to the liminality of Triangle Soup relative to a well-fitted mesh.

Broader impact Our model significantly improves rendering quality and advances 3D scene reconstruction and rendering, impacting multiple domains by enabling more realistic and efficient 3D modeling and animation. This technology could enhance VR/AR experiences [48], robotics [49], and medical imaging [50]. It could also be used for interactive education [50], scientific visualization, and a plethora of other commercial applications like product design and real estate [51].

Acknowledgments The work of P. Borycki was supported by the National Centre of Science (Poland) Grant No. 2021/41/B/ST6/01370. The work of J. Kaleta was supported by the National Science Centre, Poland, grant no 2022/47/O/ST6/01407. The work of P. Spurek and J. Waczy´ nska was supported by the National Centre of Science, Poland Grant No. 2021/43/B/ST6/01456.

References

Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics , 42(4), 2023.

Agelos Kratimenos, Jiahui Lei, and Kostas Daniilidis. Dynmf: Neural motion factorization for real-time dynamic view synthesis with 3d gaussian splatting. arXiV , 2023.

Joanna Waczy´ nska, Piotr Borycki, Sławomir Tadeja, Jacek Tabor, and Przemysław Spurek. Games: Mesh-based adapting and modification of gaussian splatting. arXiv preprint arXiv:2402.01459 , 2024.

Yi-Hua Huang, Yang-Tian Sun, Ziyi Yang, Xiaoyang Lyu, Yan-Pei Cao, and Xiaojuan Qi. Sc-gs: Sparse-controlled gaussian splatting for editable dynamic scenes. arXiv preprint arXiv:2312.14937 , 2023.

Hayong Shin, J.C. Park, B.K. Choi, Y.C. Chung, and S. Rhee. Efficient topology construction from triangle soup. In Geometric Modeling and Processing, 2004. Proceedings , pages 359-364, 2004.

Félix Paulano, Juan Jimenez-Delgado, and Rubén Pulido. An application to interact with 3d models reconstructed from medical images. volume 3, 01 2014.

Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV , 2020.

Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. ICCV , 2021.

Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M. Seitz. Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields. ACM Trans. Graph. , 40(6), dec 2021.

Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6498-6508, 2021.

Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-NeRF: Neural Radiance Fields for Dynamic Scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021.

Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocular video. In Proceedings of the IEEE International Conference on Computer Vision , 2021.

T. Li, M. Slavcheva, M. Zollhoefer, S. Green, C. Lassner, C. Kim, T. Schmidt, S. Lovegrove, M. Goesele, R. Newcombe, and Z. Lv. Neural 3d video synthesis from multi-view video. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5511-5521, Los Alamitos, CA, USA, jun 2022. IEEE Computer Society.

Yilun Du, Yinan Zhang, Hong-Xing Yu, Joshua B Tenenbaum, and Jiajun Wu. Neural radiance flow for 4d view synthesis and video processing. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 14304-14314. IEEE Computer Society, 2021.

Kaichen Zhou, Jia-Xing Zhong, Sangyun Shin, Kai Lu, Yiyuan Yang, Andrew Markham, and Niki Trigoni. Dynpoint: Dynamic neural point for view synthesis. Advances in Neural Information Processing Systems , 36, 2024.

Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. CVPR , 2023.

Sara Fridovich-Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In CVPR , 2023.

Ruizhi Shao, Zerong Zheng, Hanzhang Tu, Boning Liu, Hongwen Zhang, and Yebin Liu. Tensor4d: Efficient neural 4d decomposition for high-fidelity dynamic reconstruction and rendering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2023.

Yu-Jie Yuan, Yang-Tian Sun, Yu-Kun Lai, Yuewen Ma, Rongfei Jia, and Lin Gao. Nerf-editing: geometry editing of neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18353-18364, 2022.

Chengwei Zheng, Wenbin Lin, and Feng Xu. Editablenerf: Editing topologically varying neural radiance fields by key points, 2023.

Kacper Kania, Kwang Moo Yi, Marek Kowalski, Tomasz Trzci´ nski, and Andrea Tagliasacchi. CoNeRF: Controllable Neural Radiance Fields. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2022.

Yuze Wang, Junyi Wang, Yansong Qu, and Yue Qi. Rip-nerf: Learning rotation-invariant point-based neural radiance field for fine-grained editing and compositing. Proceedings of the 2023 ACM International Conference on Multimedia Retrieval , 2023.

Tao Lu, Mulin Yu, Linning Xu, Yuanbo Xiangli, Limin Wang, Dahua Lin, and Bo Dai. Scaffoldgs: Structured 3d gaussians for view-adaptive rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20654-20664, 2024.

Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. Spacetime gaussian feature splatting for real-time dynamic view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8508-8520, 2024.

Zeyu Yang, Hongye Yang, Zijie Pan, and Li Zhang. Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting. arXiv preprint arXiv:2310.10642 , 2023.

Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. In 3DV , 2024.

Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20331-20341, 2024.

Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Wang Xinggang. 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint arXiv:2310.08528 , 2023.

Yiqing Liang, Numair Khan, Zhengqin Li, Thu Nguyen-Phuoc, Douglas Lanman, James Tompkin, and Lei Xiao. Gaufre: Gaussian deformation fields for real-time dynamic novel view synthesis, 2023.

Tingyang Zhang, Qingzhe Gao, Weiyu Li, Libin Liu, and Baoquan Chen. Bags: Building animatable gaussian splatting from a monocular video with diffusion priors, 2024.

Lin Gao, Jie Yang, Bo-Tao Zhang, Jia-Mu Sun, Yu-Jie Yuan, Hongbo Fu, and Yu-Kun Lai. Mesh-based gaussian splatting for real-time large-scale deformation, 2024.

Antoine Guédon and Vincent Lepetit. Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering. arXiv preprint arXiv:2311.12775 , 2023.

Yi-Hua Huang, Yang-Tian Sun, Ziyi Yang, Xiaoyang Lyu, Yan-Pei Cao, and Xiaojuan Qi. Sc-gs: Sparse-controlled gaussian splatting for editable dynamic scenes. arXiv preprint arXiv:2312.14937 , 2023.

Isabella Liu, Hao Su, and Xiaolong Wang. Dynamic gaussians mesh: Consistent mesh reconstruction from monocular videos. arXiv preprint arXiv:2404.12379 , 2024.

Jiajun Huang and Hongchuan Yu. Gsdeformer: Direct cage-based deformation for 3d gaussian splatting. arXiv preprint arXiv:2405.15491 , 2024.

Ying Jiang, Chang Yu, Tianyi Xie, Xuan Li, Yutao Feng, Huamin Wang, Minchen Li, Henry Lau, Feng Gao, Yin Yang, et al. Vr-gs: a physical dynamics-aware interactive gaussian splatting system in virtual reality. In ACM SIGGRAPH 2024 Conference Papers , pages 1-1, 2024.

Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In CVPR , pages 5501-5510, 2022.

Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACMTransactions on Graphics (ToG) , 41(4):115, 2022.

Ziyi Yang, Xinyu Gao, Yangtian Sun, Yihua Huang, Xiaoyang Lyu, Wen Zhou, Shaohui Jiao, Xiaojuan Qi, and Xiaogang Jin. Spec-gaussian: Anisotropic view-dependent appearance for 3d gaussian splatting, 2024.

Ake Björck. Numerics of Gram-Schmidt orthogonalization. Linear Algebra and Its Applications , 197:297-316, 1994.

Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIGGRAPH Asia 2022 Conference Papers , SA '22. ACM, November 2022.

Xiang Guo, Jiadai Sun, Yuchao Dai, Guanying Chen, Xiaoqing Ye, Xiao Tan, Errui Ding, Yumeng Zhang, and Jingdong Wang. Forward flow for novel view synthesis of dynamic scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 16022-16033, October 2023.

Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. arXiv preprint arXiv:2309.13101 , 2023.

Zhiwen Yan, Chen Li, and Gim Hee Lee. Nerf-ds: Neural radiance fields for dynamic specular objects, 2023.

Hanbyul Joo, Hao Liu, Lei Tan, Lin Gui, Bart Nabbe, Iain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh. Panoptic studio: A massively multiview system for social motion capture. In The IEEE International Conference on Computer Vision (ICCV) , 2015.

H. Edelsbrunner, D. Kirkpatrick, and R. Seidel. On the shape of a set of points in the plane. IEEE Transactions on Information Theory , 29(4):551-559, 1983.

Herbert Edelsbrunner. Alpha shapes-a survey. Tessellations in the Sciences , 01 2010.

Linning Xu, Vasu Agrawal, William Laney, Tony Garcia, Aayush Bansal, Changil Kim, Samuel Rota Bulò, Lorenzo Porzi, Peter Kontschieder, Alja v z Bo v zi v c, Dahua Lin, Michael Zollhöfer, and Christian Richardt. VR-NeRF: High-fidelity virtualized walkable spaces. In SIGGRAPH Asia Conference Proceedings , 2023.

Daniil Lisus, Connor Holmes, and Steven Waslander. Towards open world nerf-based slam. In 2023 20th Conference on Robots and Vision (CRV) . IEEE, June 2023.

Shu-Sheng Liaw Hsiu-Mei Huang and Chung-Min Lai. Exploring learner acceptance of the use of virtual reality in medical education: a case study of desktop and projection-based display systems. Interactive Learning Environments , 24(1):3-19, 2016.

Deep Learning of Cross-Modal Tasks for Conceptual Design of Engineered Products: A Review , volume Volume 6: 34th International Conference on Design Theory and Methodology (DTM) of International Design Engineering Technical Conferences and Computers and Information in Engineering Conference , 08 2022.

Artur Kasymov, Bartosz Czekaj, Marcin Mazur, and Przemysław Spurek. Neggs: Negative gaussian splatting. arXiv preprint arXiv:2405.18163 , 2024.

Dawid Malarz, Weronika Smolak, Jacek Tabor, Sławomir Tadeja, and Przemysław Spurek. Gaussian splatting with nerf-based color and opacity.

Joanna Kaleta, Kacper Kania, Tomasz Trzcinski, and Marek Kowalski. Lumigauss: High-fidelity outdoor relighting with 2d gaussian splatting. arXiv preprint arXiv:2408.04474 , 2024.

Joanna Waczy´ nska, Tomasz Szczepanik, Piotr Borycki, Sławomir Tadeja, Thomas Bohné, and Przemysław Spurek. Mirage: Editable 2d images using gaussian splatting. arXiv preprint arXiv:2410.01521 , 2024.

Piotr Borycki, Weronika Smolak, Joanna Waczy´ nska, Marcin Mazur, Sławomir Tadeja, and Przemysław Spurek. Gasp: Gaussian splatting for physic-based simulations. arXiv preprint arXiv:2409.05819 , 2024.

Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9054-9063, 2021.

Chung-Yi Weng, Brian Curless, Pratul P Srinivasan, Jonathan T Barron, and Ira KemelmacherShlizerman. Humannerf: Free-viewpoint rendering of moving people from monocular video. In Proceedings of the IEEE/CVF conference on computer vision and pattern Recognition , pages 16210-16220, 2022.

Yue Chen, Xuan Wang, Xingyu Chen, Qi Zhang, Xiaoyu Li, Yu Guo, Jue Wang, and Fei Wang. Uv volumes for real-time rendering of editable free-view human performance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16621-16631, 2023.

Zhe Li, Zerong Zheng, Lizhen Wang, and Yebin Liu. Animatable gaussians: Learning posedependent gaussian maps for high-fidelity human avatar modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19711-19722, 2024.

Wojciech Zielonka, Timur Bagautdinov, Shunsuke Saito, Michael Zollhöfer, Justus Thies, and Javier Romero. Drivable 3d gaussian avatars. arXiv preprint arXiv:2311.08581 , 2023.

Jing Wen, Xiaoming Zhao, Zhongzheng Ren, Alexander G Schwing, and Shenlong Wang. Gomavatar: Efficient animatable human modeling from monocular video using gaussians-onmesh. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2059-2069, 2024.

Shenhan Qian, Tobias Kirschstein, Liam Schoneveld, Davide Davoli, Simon Giebenhain, and Matthias Nießner. Gaussianavatars: Photorealistic head avatars with rigged 3d gaussians. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20299-20309, 2024.

Jun Xiang, Xuan Gao, Yudong Guo, and Juyong Zhang. Flashavatar: High-fidelity digital avatar rendering at 300fps. arXiv preprint arXiv:2312.02214 , 2023.

Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero. Learning a model of facial shape and expression from 4d scans. ACM Trans. Graph. , 36(6):194-1, 2017.

A Extended related work and potential applications of D-MiSo

The detail-capturing capabilities of Sub-Gaussians can be effectively adapted to other GS-based approaches, particularly in cases where native methods lack dedicated support for high-detail reconstruction [52, 53, 54] or edition [55, 56].

Recent studies have investigated the integration of human faces, avatars, and meshes for 3D human modeling and rendering from video, presenting a promising avenue for enhancing detailed reconstruction.

For instance, [57] introduces a human body representation where neural features across frames share a latent code anchored to a deformable mesh, facilitating cross-frame integration and efficient 3D representation learning in sparse monocular video scenarios. Similarly, HumanNeRF [58] optimizes a volumetric human representation in a canonical T-pose, using a decomposed motion field to handle both skeletal and non-rigid deformations. This method provides high-quality, free-viewpoint renderings in challenging settings. UV-Volumes [59] further address the high computational costs associated with NeRF, achieving real-time, editable video rendering by encoding appearance details into 2D neural texture stacks for efficient 3D density and texture estimation.

Gaussian-based representations have gained traction for modeling dynamic avatars, as in [60, 61, 62, 63, 64], with most approaches leveraging video data and human pose/face priors like FLAME fitting [65]. Building on these advances, we propose that D-MiSo concepts could be effectively adapted for dynamic motion modeling and small detail capture in avatar representations. Specifically, Stage 1 could focus on integrating such models for capturing fine-grained avatar motion with anchored details.

B Extension of numerical results from the main paper

This section contains a numerical comparison of the D-MiSo with other models regarding the experiments described in the main document. We used additional LPIPS and SSIM metrics. During the experiments, we used the RTX4090 GPU and densification until 5000 iterations (1st Stage). We see that we get comparable competitive results. We can see that our method achieves better results for five objects from the D-NeRF dataset in terms of the LPIPS metric (Tab. 4). Full numerical comparison with metrics for the NeRF-DS dataset shows that we are comparable to other methods in the reproduction task (Tab. 5).

Table 6: Storage cost, training time, fps, and batch study on a D-NeRF dataset

In our method, a batch of images is taken as input to the model. Tab. 6 shows a numerical comparison using batch: 4, 8 on D-NeRF. Training takes 80 thousand iterations, and the second stage starts at the 5 thousandth iteration. Each Core-Gaussian has attached 25 Sub-Gaussians. This is comparable to the SC-GS implementation. Tab. 6 also presents the training time and storage cost, as well as the FPS for rendering for each dataset. These results suggest that our model is memory efficient and that the time required to train it is minimal. With batch equal to 8 in all cases, storage costs decreased, and performance improved with a trade-off of increased training time.

A similar batch study has been done for the NeRF-DS dataset presented in Tab.7. We can see that batch is playing a bigger role in the dataset, which is considered a more pronounced move.

We contrasted our model with and without the Sub-Rotation Network. The Sub-Rot Network, despite being a simple single-layer network, plays a crucial role in the second stage. Tab. 8 shows a numerical comparison of PSNR for models with and without the Sub-Rot Network. Furthermore, we would like to emphasize that due to its shallow architecture, incorporating the Sub-Rot Network does not significantly impact training time. We decided to present the results on the jumpingjack dataset distinguishing between batches of 4, 8. These results show that the Sub-Rot Network is crucial to obtaining SOTA PSNR. With Sub-Rot Network, we obtain approximately 0.5 PSNR crucial to obtain the reconstruction of small elements (like human fingers in the jumpingjack dataset)

Tab. 9 shows how the size of the network influences training time and render speed. These results show that deformable networks and Sub-Rot Networks are not costly in terms of rendering time, resulting in real-time rendering. In practice, batch size has a higher impact than deformation network depth.

Our model operates on several hyperparameters, primarily derived from the basic GS framework. Given the introduction of additional stages and the multi-Gaussian component, one of the new hyperparameters is the number of Core-Gaussians and Sub-Gaussians (Tab. 10). The ablation study also focuses on the influence of the Sub-Rot Network on training time and PSNR (Tab. 8). Tables show training time and PSNR analysis.

The number of Core-Gaussians is determined automatically during stage 1, utilizing the pruning mechanism implemented in GS. The Tab. 10 illustrates the training time corresponding to these parameters using the jumpingjacks dataset as an example with batch equal to 8. We can see the speed of the 1st stage, which is used for Core-Gaussian preparation. Moreover, too few Core-Gaussians can cause a drop in render quality (PSNR metric). The experiments suggest the number of Core-Gaussians is sufficient after the first phase. In all cases, a larger number of Sub-Gaussians improved the quality of renders.

C Extension of examples modification

Below is an example of scene modification from the PanopticSports dataset (Fig. 14). The dynamics of the object were stopped, so as to show the possibility of changing the position of the ball at a given time instant t .

Due to the characteristics of Multi-Gaussians and the ease of their control, we show that we can easily duplicate and/or scale selected objects (Fig. 15).

The third method of modification allows us to give new dynamics and fluidity to the object (Fig. 16). We can see that even difficult edits like bending the back, changing the shape of the face, and bending the hand are possible while the result is natural from the point of view of graphics. This is made possible by editing the scale and rotation of Gaussians in an explicitly defined way.

Reconstruction

Ball position modification

econstructionRenders

Example of modification

Time (new dynamic)

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly outline the claims made in the paper, detailing the contributions, key assumptions, and scope. The claims are consistent with experimental results, indicating the extent to which the findings can be generalized to other settings.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper clearly presents both technical as well as ethical limitations in a dedicated paragraph.

Guidelines:

The answer NA means that the paper has no limitation, while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper provides comprehensive technical details necessary for reproducing the code, along with datasets and experimental settings. The authors will also release the code and resulting checkpoints to ensure complete reproducibility.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code, along with the checkpoints used to report results, is publicly available. Link is provided in Experiments section. All experiments were conducted using publicly available datasets.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All results were obtained using official data splits which are stated in the paper. The hyperparameters of the model and their influence are also thoroughly discussed.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Statistical significance is not provided due to the large computational resources required. No related work provides such statistics, making it unfeasible to recompute all results for comparison with our method.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Justification: Compute resources used for experiments are listed in the Experiments section.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the broader impact of our work in the Broader impact paragraph in Conclusion section.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cited all utilized works and resources according to the citing guidelines.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not relate new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing or research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the D-Miso pipeline, which includes multi-Gaussian components and two deformation networks to model global transformation and local deformation. A good training strategy is introduced to fit the proposed methods. The rendering quality looks good, and the editing results seem better than those of current motion-based sparse control point deformation using 3D Gaussians.\n\nSTRENGTHS:\n1. The paper proposes the multi-Gaussian component, allowing large 3D scene elements. It seems like a novel idea.\n2. D-Miso uses multi-Gaussian components and two deformation networks for modeling dynamic scenes.\n3. The editing results seem better than those of SC-GS.\n\nWEAKNESSES:\n1. Multi-Gaussian representations are similar to SC-GS's control points. Since 3D/4D Gaussians themselves can model large 3D scene elements, this is not a strength. I think the contribution is minor, and it seems that no ablation study is presented in the paper.\n2. Most demos are provided in synthetic datasets, which is not convincing. Results on real-world datasets such as [9, 13] are recommended to be included.\n3.The \"Related Works\" section should be reorganized. More discussions on multi-view dynamic scenes and flow-based radiance fields should be included. \n4. If your method is based on [3], a brief review should be included in the main text as a preliminary to improve the presentation.\n5. More details such as storage costs and training time (for each stage) should be included. Additionally, would the rendering speed be affected by the large deformation network?\n6. Minor issues: Line 140: \"Guassinas\" -> \"Gaussians,\" Line 197: \"vertixes\" -> \"vertices,\" Fig. 10: \"possition\" should be \"position.\"\n7. What is n_core as mentioned in Eq.1?\n\nQUESTIONS:\n1. More comparison and discussions with SC-GS should be included since SC-GS can also support editing and share the similar ideas with control points/Gaussians.\n2. As I understand, deformation-based methods find it challenging to synthesize correct novel views with large motion. Could the authors provide comprehensive insight into this issue?\n\nLIMITATIONS:\nYes, the limiations have been addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is developed based on SC-GS, which enhanced 3DGS with Deformed Control Points to model low-rank dynamics and modify dynamic objects over time. SC-GS necessitates selecting elements that need to be kept fixed and centroids that should be adjusted throughout editing, and it poses additional difficulties regarding re-productivity editing. This paper proposes Dynamic Multi-Gaussian Soup (D-MiSo), which serves as a kind of mesh representation of dynamic GS, links parameterized Gaussian splats, forming a Triangle Soup with the estimated mesh, and separately constructs new trajectories for the 3D objects composing the scene, which makes the scene’s dynamic editable over time or while maintaining partial dynamics.\n\nSTRENGTHS:\nD-MiSo estimates the mesh as a set of disconnected triangle faces in Multi-Gaussians and uses a dynamic function to control the vertices, which makes dynamic 3D-GS easier to modify than SC-GS. \n\nMulti-Gaussians = Core-Gaussians + Sub-Gaussians. Core-Gaussians are an alternative to the control points discussed in SC-GS, with the added advantage of allowing individual modifications. Sub-Gaussians are defined by principal components of core-Gaussian, such that modifying Core-Gaussians will also change Them, which allows scene modifications.\n\nAlthough the positions of Core-Gaussians are learned by deformation MLP during training, this paper can modify dynamic objects by using the vertex of the Sub-Gaussians or generate mesh from the Core-Gaussians during inference.\n\nD-MiSo can also handle affine transformation and object scaling.\n\nWEAKNESSES:\nLack of training time comparisons.\n\nLack of comparison against gs2mesh-based methods and cage-based methods.\n\nQUESTIONS:\nLack of discussion with respect to the human-NeRF (Neural Body/HumanNeRF/UV-Volumes) and human-GS methods (Animatable Gaussians/D3GA/GoMAvatar).\n\nLIMITATIONS:\nThe authors adequately addressed the limitation.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel framework for modeling and editing dynamic scenes. The authors used a two-pass Multi-Gaussian approach to represent the entire scene. First, they obtained relatively stable Core Gaussians through initialization, and then used the Core Gaussians to drive Sub-Gaussians to fit the entire scene. To better edit motion, the authors parameterized each Gaussian with triangle soup. Experimental results show that this method not only matches the rendering quality of SC-GS but also enables the editing of more extreme large motions.\n\nSTRENGTHS:\nThis paper is clear and easy to follow. The dynamic reconstruction method based on Gaussian splatting inherently has advantages in motion editing, but this direction has not been well-explored by the community. Therefore, I am very grateful to the authors for focusing on improving the motion editing capabilities of dynamic scenes and achieving impressive editing results (as shown in Fig. 4). The triangle-soup-based motion editing proposed by the authors really makes sense. Additionally, the honest comparison of rendering metrics in both synthetic and real scenes is also appreciated.\n\nWEAKNESSES:\n1. I think the following papers should also be cited, because they have made significant contributions to the early dynamic scenes reconstruction based on Gaussian splatting.:\n   - (CVPR 2024) Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis by Zhan Li et al.\n   - (ICLR 2024) Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting by Zeyu Yang et al.\n2. L136, The concept of anchor Gaussian originally comes from Scaffold-GS [1], not from Spec-Gaussian. It would be even better if Scaffold-GS could be cited.\n3. It would be even better if Deformable-GS [2] were included in the comparison of quantitative metrics (such as Tabs. 1-2), as it is the first deformation-based dynamic Gaussian splatting method and serves as the baseline for SC-GS.\n4. I think the ablation study is not thorough enough. Although the ablation on batch size is appreciated, I am unclear about the roles of other components of the method. Corresponding ablation studies are needed to make the paper more robust. For example, I am very interested in understanding the impact of the `Sub-Rot Network` mentioned in L218 on the rendering metrics.\n\n\n\n[1] Tao Lu, Mulin Yu, Linning Xu, Yuanbo Xiangli, Limin Wang, Dahua Lin, and Bo Dai. 2023. Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering. arXiv preprint arXiv:2312.00109 (2023)\n\n[2] Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. arXiv preprint arXiv:2309.13101,2023.\n\nQUESTIONS:\nA small tip: In Tab. 1, the authors used the metrics from the SC-GS paper. However, SC-GS did not use a consistent background; for example, the `Bounce` (and maybe `Trex`) scene used a white background. The authors clearly used a black background consistently. Although the reported metrics in the table show a slight decrease compared to SC-GS, I greatly appreciate the authors' honesty.\n\nLIMITATIONS:\nPlease refer to the weakness part.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a Dynamic Gaussian Splatting representation that allows for easier object shape editing at test time. This is achieved through the use of Dynamic Multi-Gaussian Soup (D-MiSo), a mesh-inspired multi-gaussian system. Specifically, a GS is divided into two components: Core-Gaussian, which models the transformation of groups of sub-gaussians, and sub-Gaussian, which handles the final geometry and color. This enables different levels of editing by either changing the core-Gaussian or sub-Gaussians for more fine-grained control.\n\nSTRENGTHS:\n1) The paper demonstrates substantial improvement in the level of control and editability of Gaussian Splatting (GS) at test time, while maintaining competitive PSNR/SSIM scores.\n2) I appreciate the inclusion of mesh visualization estimates (Figures 9 and 11), which make the method more convincing.\n3) The method is tested on various datasets with varying complexity and shows good performance.\n\nWEAKNESSES:\nI find the paper's writing style difficult to follow and some sentences do not even make sense. Additionally, there are numerous components involved in the pipeline that are challenging to keep track of: multi-Gaussian, core-Gaussian, sub-Gaussian, core-triangle soup, multi-triangle soup, and sub-Gaussian soup (which may be the same thing?). I would appreciate a revision in writing to improve clarity.\n\nQUESTIONS:\n1) Is there a strategy for selecting hyperparameters such as the number of Gaussian and Sub-Gaussian components?\n2) It is unclear why the proposed method would perform worse than SC-GS on metrics like PSNR/SSIM for NeRF-DS and D-NeRF. Does this suggest that increased editability comes at the cost of model performance, or is it simply a matter of hyperparameter tuning?\n\nLIMITATIONS:\nThe authors have discussed the limitations of their method, but I was unable to locate a statement regarding the broader impacts of the proposed method in the Limitations section, as claimed by the author.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-29"}