D-MiSo: Editing Dynamic 3D Scenes using Multi-Gaussians Soup
Authors: Joanna Waczynska
Piotr Borycki
Joanna Kaleta
Slawomir Tadeja
Przemysław Spurek
Venue: NeurIPS 2024 poster (2024)
Keywords: Neural rendering
Gaussian splatting
3D
Triangle Soup
Primary area: deep_learning_architectures

Abstract: Over the past years, we have observed an abundance of approaches for modeling dynamic 3D scenes using Gaussian Splatting (GS). These solutions use GS to represent the scene's structure and the neural network to model dynamics. Such approaches allow fast rendering and extracting each element of such a dynamic scene. However, modifying such objects over time is challenging. SC-GS (Sparse Controlled Gaussian Splatting) enhanced with Deformed Control Points partially solves this issue. However, this approach necessitates selecting elements that need to be kept fixed, as well as centroids that should be adjusted throughout editing. Moreover, this task poses additional difficulties regarding the re-productivity of such editing. To address this, we propose Dynamic Multi-Gaussian Soup (D-MiSo), which allows us to model the mesh-inspired representation of dynamic GS. Additionally, we propose a strategy of linking parameterized Gaussian splats, forming a Triangle Soup with the estimated mesh. Consequently, we can separately construct new trajectories for the 3D objects composing the scene. Thus, we can make the scene's dynamic editable over time or while maintaining partial dynamics.

D-MiSo: Editing Dynamic 3D Scenes using Multi-Gaussians Soup
Joanna Waczyúska" +
Piotr Borycki
Doctoral School of Exact and Natural Sciences Jagiellonian University
Faculty of Mathematics and Computer Science Jagiellonian University
Joanna Kaleta
Slawomir Tadeja
Przemyslaw Spurek
Warsaw University of Technology Sano Centre for Computational Medicine
Department of Engincering University of Cambridge
Jagiellonian University
IDEAS NCBR
Abstract
Over the past years, have observed an abundance of approaches for modeling dynamic 3D scenes using Gaussian Splatting (GS). These solutions use GS represent the scene's structure and the neural network to model dynamics. Such ap- proaches allow fast rendering and extracting each element of such dynamic scene. However, modifying such objects over time is challenging. SC-GS (Sparse Con- trolled Gaussian Splatting) enhanced with Deformed Control Points partially solves this issue. However, this approach necessitates selecting elements that need to be kept fixed, as well as centroids that should be adjusted throughout editing. More- over, this task poses additional difficulties regarding the re-productivity of such editing. To address this, we propose Dynamic Multi-Gaussin Gaussian Soup (D-MiSo), which allows us model the mesh-inspired representation of dynamic GS. Addi- tionally, we propose a strategy of linking parameterized Gaussian splats, forming a Triangle Soup with the estimated mesh. Consequently, we can separately construct new trajectories for the 3D objects composing the scene. Thus, we can make the scene's 's dynamic editable over time or while maintaining partial dynamics.
Introduction
Fender reconstrucion
Suo-Trinnge Soup
Modifictione
Modilled Sup toan-innnie Soup
Recently introduced Gaussian Splatting (GS) [I] represents the 3D scene structure through Gaus- sian components. We can combine GS with the neural network (i.e., deform network) to model dynamic scenes (2). This approach involves the joint training of both the GS components and the neural network. GS characterizes the 3D object's shape and color, while the neural network utilizes time embedding and Gaussian parameters generate updated initial positions to model dynamic scenes. Such approach allows for fast rendering and extracting each element of dynamic scene.
Most existing methods can effectively model dynamic scenes, but generating new 3D objects'
positions remain challenging. Consequently, cannot edit objects over time when using such
Figure 1: D-MiSo model parameterized dynamic
scenes by Triangle Soup (disjoint triangles cloud), which allows modification of objects during time.
Equual contribution
j waw.aaawwccssssaaaooooootocccaoa waczyyskaadboctoral..t edw.uu edu.pl
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

4n
Core-Ganssiass 9v. Core-Trangle V;
Sub Goussians 0% Sub Trannie V,
Core: Triangla Soup
Muli-Triangle Soup
Figure 2: Each object using the D-MiSo model is represented by Mare-Gaussians and Sub- Gaos-aassinn, which form Multi-Gaussians. Each Gaussian is related to a triangle using parameterization proposed in GaMeS (3. Triangles define the Gaussian shape (i.e., location, scale, rotation), and triangles clouds form Triangles Soups.
approaches. To tackle this issue, SC-GS (Sparse Controlled Gaussian Splatting) 4 uses Deformed Control Points to manage Gaussians. After the training phase, we can manually modify the model at any point in time. However, this method requires identifying elements to remain static and adjusting 3D objects' centroids (nodes) during editing when a relationship between the selected nodes is visible. For example, by moving the humanoid 3D model's hand, the part of the head or leg is also changed.
To address this issue, we introduce Dynamic Multi-Gaussian Soup (D-MiSo), which is easier to modify (Fig. [T] and obtain renders compara- ble to SC-GS. D-MiSo estimates the mesh Triangle Soup, i.c. a set of disconnected trian- gle faces 6). and uses dynamic function to control the vertices.
Renders
Modifications
View Naw 2
Jaww
D-MiSo employing Multi-Gaussians, defined as larger Core-Gaussians encompassing smaller ones termed Sub-Gaussians (Fig. 2 Sub- Gaussians are defined in the local coordinate system given by principal components of Core- Gaussian. Therefore, by modifying Core- Gaussian, change all Sub-Gaussians, which allows scene modifications (Fig 3). Core-Gaussians are an alternative to the control points discussed 44, with the added advantage of allowing individ- ual modifications. Consequently, there is necessity for static and dynamic markers. In Fig. 4 present the difference between modification applied by SC-GS and D-MiSo.
Figure 3:
D-MiSo allows us to modify scenes in similar ways as classical mesh-based models.
Our model uses flat Gaussians. Therefore, based on GaMeS 3 we can approximate Gaussian components using triangle face mesh by parameterizing Gaussian components by the vertices the mesh face. We denote such transformation by T(.). In practice, as a consequence of parameterizing each Gaussian, we obtain a cloud of triangles called Triangle Soup [5] Using Triangle Soup, we can control two types of Gaussian components. Accordingly. in D-MiSo, Multi-Triangle Triangle Soup (Fig. 2 Core-Triangle Soup is marked by red color, and Sub-Triangle Soup is denoted in blue). In D-MiSo, we can select and modify one part of the object like a mesh. In contrast, using SC-GS, static and dynamic points have to be selected, and editing only one part of the object is difficult.
get: Sub- hub-Triangle Soup,
Core-Triame Triangge Soup and Multi-
During training. the positions of Core- More-Gaussians are managed by deformation multilayer perceptron (MLP), while the Sub-Gaussians are collectively manipulated through global transformation and small local deformation. The former describes the general flow of objects in the scene, while the local deformation is responsible for modeling small changes like shadows and light reflections. After training, we can modify our model directly by using the vertex of the Sub-Triangle Soup, or we can generate mesh from the Core- ToranTri Soup (Fig. 5).
The contributions of this paper are significant and are outlined follows:
We introduce the Multi Malti-Gaussian components, which consist of single large Gaussian response for global transformations and many small components dedicated to rendering. Multi-Gaussian components allow for the modeling of large 3D scene elements.
We propose D-MiSo a model that uses Multi-Gaussian components and two deformation networks for modeling dynamic scenes.

---

Appare d-d derc sereer dyranic
appesprian jamp soally selcee) namber Lame rntoa paftes
The selected sbric
SC-GS
Render
Nodes
Proper modification
Challenges in controling moving elements managing separabilty
prodte selection mowig paro
D-MiSo
Render
Suib-Tringgee Soup
Hand Rotation
Effortless modification moving elemnnts
Comparison of possible modifications in D-MiSo and the SC-GS. In the latter, authors nodes while D-MiSo apply Sub-Triangle Soup (see the second column). We also must add static (pink) and dynamic (yellow) points in SC-GS to obtain modification by editing dynamic points. In practice, we have to use many static points to stop artifacts. Moreover, SC-GS is not an affine invariant and produces space when change the size of the objects. In the case of D-MiSo, marked points and applied modifications. Our model is superior in handling object scaling.
Figure 4:
Our D-MiSo allows object to be edited at selected moment in time. The edited components are independent, and the editing does not affect other parts the object. In addition, it also allows for full partial dynamics to be maintained. Modifications also include scaling and rotation.
2 Related Works
Recent advancements in view synthesis, particularly driven by NeRFs 77. have significantly con- tributed to the rapid development of novel view synthesis techniques. However, the majority of these approaches model static scenes implicitly using MLP. Moreover, several works have extended classical NeRF to dynamic scenes through the use deformation fields 8 10 and [1]. The
alternative approaches, such 212 and [13] represent scenes as 4D radiance fields. Early works
dynamic scenes face difficulties when dealing with monocular settings and uncontrolled or lengthy scenarios. To enhance scene motion modeling,
works utilize flow-based techniques (TO] [4] 15).
However, FeRs-b solutions often suffer from long training and rendering times. To address this, crid-plane-based methods [16 17 18 have been proposed. In addition. several NeRF-based
approaches have also been extended for scene editing purposes [19 20 21 22).
The recently introduced Gaussian Splatting (GS) technique []] addresses many limitations of other methods, offering multiple advantages due to their explicit geometry representation, enabling easier dynamics modeling. The efficient rendering of the 3D version of GS also avoids densely sampling and querying neural fields, making downstream applications such as vie-viie video reconstruction more feasible. A notable extension of 3D GS was proposed in [23], where the authors introduced the concept of anchor points to tackle the problem of overfitting caused by redundant Gaussians. Scaffold-GS addresses this issue by distributing local 3D Gaussians according anchor points. This approach reduces redundancy, enhances scene coverage, and maintains high-quality rendering with improved robustness view changes. While the original GS was developed for static scenes, several extensions for dynamic scenes were proposed. Most of the early works operate in multiview setup 24 25 26]. For example, (26) utilizes Trame-by-frame approach to model each timestep. However, this method lacks inter-frame correlation and requires high storage overhead for long-term term sequences. In [2 iLP introduced to model changes in Gaussians over time, and in 28 MLP together with decompased neural voxel encoding algorithm are utilized for training and storage efficiency. In
29 dynamic scenes are divided into dynamic and static parts, optimized separately and rendered
together to achieve decoupling. [24] spacetime Gaussian proposes approximating the spatiotemporal 4D volume of a dynamic scene by optimizing collection of 4D primitives with explicit geometry and appearance modeling. This method uses 4D Gaussians parameterized by anisotropic ellipses and
3

---

Vol
Giyy
Vi
Ve
Vg
V
4,
For oach Sun Gaussian chosa the closest estimated-mmaah -eahh face V.
Traneform Gauesions using
modiliation
Figure 5: One way to modify the object at the selected time to is to take Core-Gaussians and apply a meshing strategy to obtain the correct mesh instead of Triangle Soup. Then, we can parametrize Sub-Gaussian in the coordinate system given bay mesh faces instead of Core- Soup. Finally, we can modify our mesh to obtain new modifications.
view-dependent, time-evolved appearances represented by 4D spherical harmonics coefficients. In 225 novel, real-time and photorealistic scene representation called Spacetime Gaussian Feature Splatting has been introduced. This approach extends 3D Gaussians with temporal opacity and parametric motion/rotation, enabling the capture of static, dynamic, and transient scene content. Additionally, the method incorporates splatted feature rendering to model view- and time-dependent appearances while maintaining a compact representation size. The method is notable for its high rendering quality and speed while also being storage-efficient Other works enhance dynamic scene reconstruction using external priors. For example, the diffusion priors be used regularization terms during optimization 30.
Furthermore, GS was employed for mesh-based scene geometry editing. In 31 3D Gaussians are defined over an explicit mesh and utilize mesh rendering to guide adaptive refinement. This approach depends on the extracted mesh as a proxy and fails if the mesh cannot be extracted. In contrast, in [32, explicit meshes are extracted from 3D GS representations by regularizing Gaussians over surfaces. However, this method involves costly optimization and refinement pipeline. Another example of [33] employs sparse control points for 3D scene dynamics, but this method struggles with intense edit movements and necessitates accurate static node selection. Also, combines GS with mesh extraction. However, such an approach only works for static scenes.
The method proposed in 34 combines meshes with GS and reconstructs a high-fidelity and time-consistent mesh from a single monocular video. However, it relies on a Poisson Solver and differentiable Marching Cubes to recover the de- formed surface, significantly complicating the pipeline. Moreover, it does not explore geom- etry modification capabilities, which constitute significant aspect of work. Conversely, are an intuitive tool for geometry manipulation. However, they require additional steps for cage-building and may lack the flexibility and precision of manual, vertex-level deformation techniques, potentially missing fine details.
Gv
V/
Figure 6: Multi-Gaussian, consisting of one Core- Gaussian Gv, and a Lab-Gaussian 92z- The Core Gaussian is parametrized by a V,-triangle, and the Sub-Gaussian by Vi- Vi-triangle. The relative dis- tance of the center of the Sub-Gaussian from the Core-Gaussir Gaussian indicated by a (af ag of)).
cage-basee based methods [5 36
In contrast to the listed approaches, we propose a D-MiSo, a mesh-based method specifically designed to handle dynamic scenes. D-MiSo leverages straightforward pipeline of GS techniques to enable real-time editing of dynamic scenes.
3 Dynamic Multi-Gaussian Soup
Here, we present the main components D-MiSo. We start with the classical GS to provide the foundations for our model. Next, we introduce the concept of Multi- Malti-Gaussian and describe how to

---

Stage
Stege
Fanders
Araratan
Oulorm
(fey Art AN1
Network
Dalem
Rehaore
Figure 7: D-MiSo model diagram. The input of the model consists of images at different moments in time and information regarding the position of the camera. Training distinguishes two main phases (i.c., stages). The first includes the preparation of Core- Gaussians, describing the movement of the object. The second focuses on the fitting of Sub-Gaussians responsible for the render's quality. The model has the ability to produce high-quality renders create unination/modification of the object due to Saussaasss (i.e., Sub- Triangles Soup) shape modification.
estimate the mesh for editing. Finally, we show D-MiSo, which uses scenes.
in dynamic 3D
Gaussian Splatting The Gaussian Splatting (GS) technique models 3D scenes using an array of 3D Gaussians, each specified by its mean position, covariance matrix, opacity, and color expressed using spherical harmonics (SH) [37 38 The GS algorithm constructs the radiance field by iteratively optimizing the parameters of all Gaussian components. Ultimately, the GS efficiency mainly depends on its rendering method, which involves projecting Gaussian components.
The GS framework employs dense collection of 3D Gaussians: G = [(,(m, Σj), o. c)) where m, denotes the position, Σ, the covariance, d; the opacity, and c; the SH colors for the i-th Gaussian. The GS optimization process involves a repetitive cycle of rendering and comparing the resultant images with the training views. our work, we will use Multi Gaussian approaches (Fig. 6).
Mausi-Ga Multi- multi-Gaussians Gmuttr dedicated to describing relatively large parts of the 3D scene allow modification of large blocks instead of modifying each small Gaus- sian separately. The Multi-Gaussian model com- prises a primary large 3D Gaussian (referred to the Gaue-Gansiinn Goore), which encom- passes numerous smaller Gaussians (termed Sub-Gaussians Guur), all of which param- eterized by the main Core-Gaussian. Multi- anchor Gaussians
Gr
AR
g'
de time
time
Figure 8: Representation of change over time act- ing on Core-Gaussianl using neural network re- sponsible for movement. In practice, t; an ab- stract time. The network's 's output returns informa- tion about the change location AUy. scale (Avg. Avg), and rotation AR.
Gaussianare is similar from [39]. but we do not use a neural network to produce child components. We parametrize Sub- Sob-Gaussians in local coordinate system.
Similarly to classical GS, parameterize the Core-Gaussian Gaussian distribution by center m and the covariance parameterized by factorization: Σ = RSSRR" where R is the rotation matrix and the scaling parameters. More precisely consider p Core- Gaussians uses flat Gaussians as in 3 and defined by:
Geore {WNore (m, R, S,), c.)c
where S = diag(81,500 82, 83), and R is rotation matrix of Core-Gaussian defined as: R = r2, r3), where r; E R3 which can be interpreted as a local coordinate system used by Sub-Gaussian.
5

---

It is worth noting that Sub- Gaussian can be interpreted as child of Core- Gausian. We define centers of Bab-Gaussian (mn',,,, R', .5*) in the local coordinate system of Sore-Gaussian N Wore! (,, R. S) by: m' m + Ra where m. are trainable parameters used to define the positions of the Sub-Gausee Gaussian relative to the Core-Gaussian (Fig. 6 Sub-Gaussians are used for rendering and can be seen main component of our model:
Core-Gaussian position and rotation; and a = (aj 0 a$))
Geub (Neub (m + Ra RR, , S*) o², )
(2)
where m, S are parameters of Fore-Gaussian and, a², St, Nt marks parameters of i-th Sub- Gaussians with opacity o², and SH colors ct.
Core-Gaussians generally dedicated mainly transformations, hence, in practice opacity or colors are not used during rendering. On the other hand, Sub-Gaussian is devoted to rendering and modification. has its own opacity and colors.
4g < dg
GaMeS parametrisation of Multi-Gaussian component Multi-Gaussians describe 3D scenes using solid blocks rather than tiny Gaus- sian distributions. This method enhances our model's ' suitability for dynamic environments. One of our mode's ' most important properties is its ability to model dynamic scenes in each time step. To obtain such properties, we parameter- ize all Gaussian using Triangle Soup following similar approach as in GaMeS 3. Thanks to few simple transformations, we can convert the mean m rotation It and scaling S into triangle V= (vi, V2, V3). which parametrizes Gaussian distribution. Such transformation is unambigu- ous and reversible.
dg < dr
de
d2
d
Estimated Mesh
Sub- Lub-Gaussian
Figure 9: One way to animate is to assign the near- est triangle (face) to each Sub-Gaussian from the estimated mesh. The mesh modification changes the assigned Gaussian.
Let us assume that we have a Gaussian component parameterized by mean m, rotation matrix R [ri, r2, r3] and scaling = dig(e, 82, 83). We define three vertex of triangle (face): V m 8212, V3 m + 8313.
[V1, V2, V3l,, where V1= m,
Now we froze the vertex of the face V [V1, V2, V3l and reparameterize the Gaussian component by defining m, R [f,, f2. is] and S dig( $1, $2. 5z). First, we put m V1. The first vertex of R is
given by a normal vector:
(V2 v1) x (V3 v1) (V3 v1)ll'
f
11(v2 v
where is the cross product. The second one
Int position with Sub-Trisngle Soup
D-Misa Modiicalion
SC Mediliction
(v2-x1) defined by in2 I[v=-vv)]ll The third one is obtained as a single step in the Gram-Schmidt process 40)
f3- orth(v3 v1)
Nocho
Scaling parameters can also be easily calcu- lated as 81 E, 82 = ||v vill and & (V3 - vI, vi, fa). Consequently, the covariance of Gaussian distribution positioned on face is given by:
Êv= Sy.
An example of Sub- Triangle Soup mod- ification using D-MiSo and the render obtained by this change from a different viewpoint. is possi- ble not only change the position of the hand but also to raise the thumb. Comparison with SC-GS similar modifications, highlighting the challenge of editing small elements individually.
Figure 10:
and correspond with the shape of triangle V. For one face (V1, V2, V3). we define the corre- sponding Gaussian component:
v2)) N(mv, Rv, Sv).

---

Recomssrucion
Modificaionn by ectimated Meeh change
Moditiaatoon by Triangles Soup (Sub Gausslans) dircclly change
Care-Gaussaars
Sub Geussians
Esimated Mash
Eatimetedd Meah & Sub Gaussiens
Figure 11: Reconstruction and three ways of modification of the output object. The first involves modifying the estimated mesh, which does not have to be accurate. The next two focus on Sub- Triangle Soup editing. The red box shows the direct modification of the Triangle Soup in logical way (e.g., raising a hand). The yellow box shows a change in space, i.e., giving fluidity to an object by creating abstract modification.
Finally, the Gaussian component derived from the mesh face parameters. This approach can be applied in Multi-Gaussian framework. Therefore, will use invertible transformation between Gaussian parameters and triangle face T and notation:
N(V) ===(T = Nmiv, Rv,
In our D-MiSo we parameterize the Core-Gaussians with k Triangle soup:
Gmellti =(NooreeV))) (Nous (mvv + Rv, R R* . . of )
(3)
j=1
where Vg, a², R. Ri σ², are trainable parameters and (my, Rv, Sv,)= (Vj). Alternatvelyy and Lub-Gaussiang by Triangle Soup:
can parameterize
Gmalti (Nonrre (V)): (Wand (Vj),
of ())
|=1
where (mn, Ry, Sv,) -T))=T-((V)) V T(my + Rya S")
In D-MiSo, we use the collation Multi-Gaussian distribution for rendering and Sub-Triangle Soup for editing. The formal definition of our model uses equation (3) since, in training, we store Core- Gaussian triangle face (Core-Gaussial does not have colors) and Sub-Gaute SabbGaussian as a collection from classical GS components with color and opacity. After training, we parametrize our model to equation (4) for editing.
3.1 Dynamic Multi-Gaussian Soup (D-MiSo)
Previously, we defined Multi-Gaussians and their parametrization using Triangle Soup. Now, we have all the tools to present the D-MiSo model. The overview of our method illustrated Fig. 7 The input to our model is set of images of dynamic scene, together with the time label and the corresponding camera poses. Our training divided into two stages. the first stage, we initialize the
In the second, we add Sub-Gaussian components.
Stage 1 First, we train only Core-Gaussiaa to obtain good initialization for Multi-Gaussin As Core-Gaussian mainly employed to capture motion, model only requires their small amount (Fig. 77) our approach, the Core-Triangle Soup, constructed via the Gaussaasss parameterization (as depicted in Fig. 2. is adjusted depending on the time l.
In practice, when random initialization of Gaussians necessary, redundant Gaussians must pruned first to ensure that the remaining represent the object's 's shape. To reduce the number of Gaussians
7

---

and obtain consistent Core- Sare-Gaussinss we train GS on a batch containing a few views instead of one. In practice, we render a handful of views from different positions and use paok-propngation.
In particular, we parameterize Gaussians (m, R, S) by face V = (V1, V2, Va) to ob- tain (Vv1; V2, V3). Our Deform Network takes in input the triangle vertices V as- signed to the parameterized 3D Core-Gaussians and the current time t and returns updated (V, (t), (12 Ava(t), Av Such updates consist of translation and rota- tion (Fig. 80:
V(t) = V o (V (v1 + Avi + Av2(t), V3 + Avs(t)) (Ry
D-MiSo parameterized Core-Gaussians in time by: Neore (V),, Noore (V o (V, where ie is deformable network that moves triangle according to time L. The opacity and color of the Core- Gore-Gausiinn are used only in the first stage.
Stage 2: Preparation To move to the second phase of the model, it is imperative to prepare the Multi-Gaussiann Gaussians. This involves attaching k Sub-Gaussin Gabssaasss to each Core- Gaussian generated in Stage I, as shown in Fig. 6 Henceforth, Sub- Gaussians assume responsibility for the resultant rendering. Initially, the Sub-Gaussian adopts the same features as the Core-Garr Gaussian, except for the position.
position
Small elements animations
Stage The primary objective of the sec- ond phase is to parallelized Core-Gaussian (Core- Triangles Soup) to enhance understand- ing of movement and increase rendering quality through the precise training of Sub-Gaussians.
Figure 12: Limbs render and modification obtained with D-MiSo. possible to close the hand.
worth noting that is also
Since the centers of aab-Gaussians parameterized by the local coordinate system given by the rotation matrix of Core-Gaussian, when the Deform Network &! changes the Core-Gaussian, all Sub-Gaussians (attached this Core-Gaussian) are modified by global transformation (VV,t) t).
D-MiSo use an additional deformation network Sub-Rot Network e dedicated to each i-th Sub- Gassian's 's small changes. Sub-Rot Network takes the Sub-Gaussian rotation matrix Ri and the current time as input and produces an updated rotation matrix A ARt (1).
The position ()) myvo) + of of the Sub-Gaussian time determined by the position ()) and rotation Rv of the Core-Gaussian (parameterized by triangle V) and the learning parameter a'. It should be noted that scale S color C,, and opacity o, of Sub- Gaussian are trainable and do not de- pend time. Sub-Rot Network produce up- dated db R2', I) = RR for rotation parameter of Sub-Gaussians. Finally parameters of Sub- Gaussians in time t depends on Deform Nenwork tb and Sub-Rot Network o, and the Norr-Gaussians parameter V, and Sub-Gaussian parameters Rt, St, and o;:
Figure 13: Examples of object modifications. The first method allows for smooth modification (bending) and also removes (e.g. plate), scales and/or adds (e.g. small blue balls) objects.
(L) ((Nand (m + R Ri + (,,
D-MiSo final model consists of two deformable networks and two levels of Gaussian distributions. This approach enables efficient modeling of object motion over time in dynamic scenes, allowing for adjustments to objects at each time frame.
The result of the model's inference depends on the camera's view angle well as the selected time. Only Sub-Gaussian features are used in the rendering process. Hence, the implementation of generating output image from Gaussians is no different from vanilla GS.
8

---

The modifications involve the appropriate transformation of Sub-Gaussian We base this on the fact that the transformed Gaussians have exactly the same color properties as before editing. Only the shape of the Gaussians is changed. In our work, we present three editing methods (Fig.)
4 Experiments
Table 1: Quantitative comparisons (PSNR) on a D-NeRF dataset showing that D-MiSo gives com- parable results with other models.
The experimental section divided into two parts. First, we show that D-MiSo can model dynamic scenes with high quality; in the sec- ond part, we demonstrate that our model allows an easy editing procedure, which is our main contribution.
PSNR
Hook Jumpin.
Trex
Bounc.
Hell
Mutant
Stan.
D-NeRF 11
29.25 32.80
31.75 38.93
25.02 31.29
32.79
TiNeuVox-B B
31.45 34.23
32.70 40.73
28.17 33,61
35.43
Tensor4D L3
29.03 24.01
23.51 25.36
31.40
29.99
30.86
K-Planes
28.59 32.27
31.41 40.61
25.27 33.79
34.31
4.1 Reconstruction of dynamic scenes
FF-NVS 32
32.29 33.55
30.71
40.02
27.71 34.97
36.91
4D-GS 25
30.99 33.59
38.59
31.39
35.37
DynMF 2
33.94 38.04
35.82 41.92
37,51
41.68
41.16
Here, we outline the specifics of our implemen- tation and provide a detailed description of the used datasets. We demonstrated the core advan- tages of our model by conducting experiments using three different datasets. The source code is available on 3 Our code is developed on top of the GS vanilla code, according to their license. We used NVIDIA GeForce RTX 4090 and A100 GPUs. The experiments focus on a benchmark task of reconstruction. PSNR metric is used to compare our model with other methods. Furthermore, additional numerical comparisons, i.e., SSIM/LPIPS, are available in the supplementary material.
Deform-GS 43
33.77
39.10
38.40
41.46
#2.11
43.73
45.38
Editable
SC-GS A5
39.87 41.13
41.24
44.91
42,93
45.19
47.89
D-MiSo (our)
38 42.05
40.88 41.49
41,49
44.38
47.66
Contains seven dynamic objects with realistic materials described with single camera [.]. This means the model had access only one view at a given moment. Tab. [ shows we are very comparable to other methods, and we achieve a higher PSNR on one object. The differences in metrics are small, however, our method presents an easier way to edit (Fig. Providing, among other things, better scaling. Following previous methods, our results are obtained on images using 400 by 400 resolution with black background.
D-NeRF Datasets:
This dataset contains again seven real-world scenarios containing a moving or deforming specular object. Each scene was recorded using two cameras, with the video captured by one of them being used as training set, while the footage from the second one was treated as a test set. The camera pose was estimated using COLMAP for both cameras. Tab. 2 shows that our method achieves the SOTA results for distinct objects.
NeRF-DS(44]:
PanopticSports Datasets: The dataset comprises six dynamic scenes featuring significant object and actor movements 26 45) The scenes are categorized by the activities performed in the video sequences, i.e., juggling, boxing, softball, tennis, football, and basketball. Each scene was recorded using 31 cameras over 150 timesteps. Following the official data split, we use footage from 27 cameras for training and the remaining 4 cameras for testing. Numerical results are shown in Tab. 3 The results show that the model achieved SOTA for five objects according to PSNR metrics and six objects according to L.IPS metrics.
These results demonstrate that D-MiSo is comparable to other methods. Morcover, our main contribution in comparison to other methods is the very easy editing of the resulting object (Fig.
4.2 Editing of dynamic scenes
As mentioned earlier, we proposed three new methods of output object modification. The first one focuses on moving and preparing formal mesh, which is connected in contrast to Triangle Soup. We apply a simple meshing strategy on Core-Gaussians. We used the basic Alpha Shape algorithm [6, 47 and showed that the mesh does not have to estimate the surface perfectly to be effective. Then, we reparametrize the Sub-Gaussian by finding the closest face from the mesh (Fig. (9). In practice, represent each Sub-Gaussa Gaussian in local coordinate system (analogically to Multi- Gaussians). Therefore, each Sub-Gaussian is assigned to the nearest face (Fig. D) Each face can have different number of Sub- Sub-Gaussians prescribed in such configuration. This method allows us
hbttss//gihhbbccom/wwweeooo////

---

to maintain the consistency bestowed by the mesh. The whole process is shown in Fig. 5 Thanks to such representation. we can edit our connected mesh to produce the correct edition of the dynamic scene (Fig. -
The second editing method allows to define the con- nections, i.c., editing Sub-Triang -riinniie Soup directly, e.g. moving hand or bending horns (Fig. I0 or rotating human body (Fig. 3). Changes are possible because we can also transform group of Sub- Triangles instead of individual ones. This method allows for even very sub- tle changes like raising the thumb (Fig. [O). turning the hand over ) opening or closing the hand (Fig. [2). These edits would be difficult in other approaches based on adjusting the 3D objects' centroids (nodes) since the space of nodes limited in details area (Fig. 4 Nodes' use is preeminent for defining movement, which was insignificant in these places.
Table 2: PSNR comparisons NeRF-DS dataset showing that D-MiSo gives compa- rable results with other models.
PSNR 7
Bell Sheet Press Baxin Cup Sieve Plate
HyperNeRF
24.0
20.2
25.0
18.1
NeRF-DS [+]]
23.3
25.7
26.4
20.3
24.5
26.1
20.8
Vox-uVox- B 41
23.1 21.1
24.1 20.7
20.5
20.1
20.6
Editable
SC-GS !
25.1 26.2
26.6
19.6
24.5
26.0
20.2
D-MiSo (our)
25.3 25.8
25.6
19.8
24.5
26.5
20.8
With this method, can also change complicated objects like 360 scene without losing the dynamics-related model. For example, the rotation of a person stacking boxes (Fig. 3
The third method focuses transform- ing the space in which the object is lo- cated. Similar to the second method, the third one also works directly on the Sub- Triangle Soup. We can achieve such an effect by applying a certain function to the selected plane. In practice, the ob- ject, or a portion of it, is modified, for instance, through the use of sinusoidal functions. It allows us to obtain fluid- ity and more easily define the physical nature of the movement (Fig.
Table 3: Comparison on PanopticSports dataset.
Metries
Mothed
JugeleBoxes (Sottaol Footss BasketbaliMean
3DGS 0
28.19 28.77
28.77
28.03
28.49
27.02
28.
PSNR
Dyn3DG 26
28.22
28.7
D-MiSo (our)
29.79 29.39
28.6
29.02
28.99
28.49
29.04
3DGS 0
0.91
0.91
0.91
0.90
0.90
0.89
0.90
SSIM
Dyn3DG 26
0.92
0.91
0.91
0.91
0.91
0.91
0.91
M-MiSo (our)
0.92
0.92
0.92
3DGS i
0.15
0.15
0.14
0.16
0.16
0.18
0.16
LPIPS
Dyn3DG 26
0.15
0.17
0.19
0.17
0.19
0.18
0.17
D-MiSo (our)
0.13
0.13
0.15
0.14
0.13
0.15
0.14
Our methods are also scalable, so can easily remove duplicate elements from image. Moreover, the duplicated elements can be given their own dynamics. Examples of these effects shown in Fig. 13 where we removed the plate and both duplicated and rescaled the blue balls multiple times.
5 Conclusion
D-MiSo is novel method based on Gaussian Splatting parameterization, which produces cloud of triangles called Triangle Soup. The method allows easy editing of objects created in inference with the possible transformations, including moving, scaling, and rotating. By defining Multi-Gaussian the obligatory separability of modified elements seen in other models is combated. In addition, certain elements of objects can be duplicated and removed (Fig. 13 Furthermore, the D-MiSo method can facilitate giving different dynamics to separate parts of an object (Fig. D
Limitation The method allows for complex changes at given moment in time. However, if some area is not well represented in the training set, it is impossible to edit them. For example, a person's 's hand can be changed but not fingers (Fig. This due to the liminality of Triangle Soup relative to a well-fitted mesh
Broader impact Our model significantly improves rendering quality and advances 3D scene recon- struction and rendering, impacting multiple domains by enabling more realistic and efficient 3D modeling and animation. This technology could enhance VR/AR experiences (48) robotics [49], and medical imaging [50]. It could also be used for interactive education [50]. scientific visualization,
and a plethora of other commercial applications like product design and real estate (5I].
Acknowledgmentts The work of P. Borycki was supported by the National Centre of Science (Poland) Grant No. 1/2//41/B/ST601 1370. The work of J. Kaleta was supported by the National Science Centre, Poland, grant no 2022/47/O/ST6/0/ 407. The work of P. Spurek and J. Waczyńska was supported by the National Centre of Science, Poland Grant No. 2021/43/B/ST6/01456
10

---

References
[1] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), 2023.
[2] Agelos Kratimenos, Jiahui Lei, and Kostas Daniilidis. Dynmf: Neural motion factorization for real-time dynamic view synthesis with gaussian splatting. arXiV, 2023.
[3] Joanna Waczyńska, Piotr Borycki, Slawomir Tadeja, Jacek Tabor, and Przemystaw Spurek. Games: Mesh-based adapting and modification of gaussian splatting. arXiv preprint WrXiv2240.001452 2024.
[4] Yi-Hua Huang, Yang-Tiar Sun, Ziyi Yang, Xiaoyang Lyu, Yan-Pei Cao, and Xiaojuan Qi. Sc-gs: Sparse-controlled gaussian splatting for editable dynamic scenes. arXiv preprint arXiv:2312. 14937, 2023.
[5] Hayong Shin, J.C. Park, B.K. Choi, Y.C. Chung, and S. Rhee. Efficient topology construction from triangle soup. In Geometric Modeling and Processing, 2004. Proceedings, pages 359-364, 2004.
[6] Félix Paulano, Juan Limenez-Delgadee and Rubén Pulido. An application interact with 3d models reconstructed from medical images. volume 3, 01 2014.
[7] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV, 2020.
[8] Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. ICCV, 2021.
[9] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin- Brualla, and Steven M. Seitz. Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields. ACM Trans. Graph., 40(6), dec 2021.
[10] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6498-6508, 2021.
[11] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguer. D-NeRF: Neural Radiance Fields for Dynamic Scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[12] Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocular video. In Proceedings of the IEEE International Conference on Computer Vision, 2021.
[13] T. Li, M. Slavcheva, M. Zollhoefer, S. Green, C. Lassner, C. Kim, T. Schmidt, S. Lovegrove, M. Goesele, R. Newcombe, and Z. Lv. Neural 3d video synthesis from multi-view video. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5511-5521, Los Alamitos, CA, USA. jun 2022. IEEE Computer Society.
[14] Yilun Du, Yinan Zhang, Hong-Xing Yu, Joshua B Tenenbaum, and Jiajun Wu. Neural radiance flow for 4d view synthesis and video processing. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 14304-14314, IEEE Computer Society, 2021.
[15] Kaichen Zhou, Jia-Xing Zhong, Sangyun Shin, Kai Lu, Yiyuan Yang, Andrew Markham, and Niki Trigoni. Dynpoint: Dynamic neural point for view synthesis. Advances in Neural Information Processing Systems, 36, 2024.
[16] Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. CVPR, 2023.

---

[17] Sara Fridovich- Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In CVPR, 2023.
[18] Ruizhi Shao, Zerong Zheng. Hanzhang Tu, Boning Liu, Hongwen Zhang, and Yebin Liu. Tensor4d: Efficient neural 4d decomposition for high-fidelity dynamic reconstruction and rendering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2023.
[19] Yu-Jie Yuan, Yang-Tian Sun, Yu-Kun Lai, Yuewen Ma, Rongfei Jia, and Lin Gao. Nerf-editing: geometry editing of neural radiance fields. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 18353-18364, 2022.
[20] Chengwei Zheng, Wenbin Lin, and Feng Xu. Editablenerf: Editing topologically varying neural radiance fields by key points, 2023.
[21] Kacper Kania, Kwang Moo Yi, Marek Kowalski, Tomasz Trzciński, and Andrea Tagliasacchi. CoNeRF: Controllable Neural Radiance Fields. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2022.
[22] Yuze Wang, Junyi Wang, Yansong Qu, and Yue Qi. Rip-nerf: Learning rotation-invariane point-based neural radiance field for fine-grained editing and compositing. Proceedings of the 2023 ACM International Conference on Multimedia Retrieval, 2023.
[23] Tao Lu, Mulin Yu, Linning Xu, Yuanbo Xiangli, Limin Wang, Dahua Lin, and Bo Dai. Scaffold- gs: Structured 3d gaussians for view-adaptive rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20654- 20664, 2024.
[24] Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. Spacetime gaussian feature splatting for real-time dynamic view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8508 08880,850,000 2024.
[25] Zeyu Yang, Hongye Yang. Zijie Pan, and Li Zhang. Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting. arXiv preprint 10X1v:2310.10642, 2023.
[26] Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. In 3DV, 2024.
[27] Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20331-2033 10341, 2024.
[28] Guanjun WU, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Wang Xinggang. 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint arXiv:2310.08528, 2023.
[29] Yiqing Liang, Numair Khan, Zhengqin Li, Thu Nguyen-Phuoc, Douglas Lanman, James Tompkin, and Lei Xiao. Gaufre: Gaussian deformation fields for real-time dynamic novel view synthesis, 2023.
[30] Tingyang Zhang, Qingzhe Gao, Weiyu Li, Libin Liu, and Baoquan Chen. Bags: Building animatable gaussian splatting from a monocular video with diffusion priors, 2024.
[31] Lin Gao, Jie Yang, Bo-Tao Zhang, Jia-Mu Sun, Yu-Jie Yuan, Hongbo Fu, and Yu-Kun Lai. Mesh-based gaussian splatting for real-time large-scale deformation, 2024.
[32] Antoine Guédon and Vincent Lepetit. Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering. arXiv preprint arXiv:2311.12772 2023.
[33] Yi-Hua Huang, Yang-Tian Sun, Ziyi Yang, Xiaoyang Lyu, Yan-Pei Cao, and Xiaojuan Qi. Se-gs: Sparse-controlled gaussian splatting for editable dynamic scenes. arXiv preprint a7X1v 2023. 2023.
12

---

[34] Isabella Liu, Hao Su, and Xiaolong Wang. Dynamic gaussians mesh: Consistent mesh recon- struction from monocular videos. arXiv preprint vFXiv:240..12379 2024.
[35] Jiajun Huang and Hongchuan Yu. Gsdeformer: Direct cage-based deformation for 3d gaussian splatting, arXiv preprint arXiv:2405. 15491,, 2024.
[36] Ying Jiang, Chang Yu, Tianyi Xie, Xuan Li, Yutao Feng, Huamin Wang, Minchen Li, Henry Lau, Feng Gao, Yin Yang, et al. Vr-gs: a physical dynamics-aware interactive gaussian splatting system virtual reality. In ACM SIGGRAPH 2024 Conference Papers, pages 1-1, 2024.
[37] Sara Fridovich-Keil. Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In CVPR, pages 5501-5510, 2022.
[38] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM Transactions on Graphics (ToG), 41()) 15, 2022.
[39] Ziyi Yang, Xinyu Gao, Yangtian Sun, Yihua Huang, Xiaoyang Lyu, Wen Zhou, Shaohui Jiao, Xiaojuan Qi, and Xiaogang Jin. Spec-gaussian: Anisotropic view-dependent appearance for 3d gaussian splatting, 2024.
[40] Ake Björck. Numerics of Gram-Schmidt orthogonalization. Linear Algebra and Its Applications, 197:297-316, 1994.
[41] Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIG- GRAPH Asia 2022 Conference Papers, SA '22. ACM, November 2022.
[42] Xiang Guo, Jiadai Sun, Yuchao Dai, Guanying Chen, Xiaoqing Ye, Xiao Tan, Errui Ding, Yumeng Zhang, and Jingdong Wang. Forward flow for novel view synthesis of dynamic scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 16022-16033, October 2023.
[43] Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. De- formable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. arXiv preprint 2r09v:2309. 13101, 2023.
[44] Zhiwen Yan, Chen Li, and Gim Hee Lee. Nerf-ds: Neural radiance fields for dynamic specular objects, 2023.
[45] Hanbyul Joo, Hao Liu, Lei Tan, Lin Gui, Bart Nabbe, lain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh. Panoptic studio: A massively multiview system for social motion capture. In The IEEE International Conference on Computer Vision (ICCV), 2015.
[46] H. Edelsbrunner, D. Kirkpatrick, and R. Seidel. On the shape of a set of points in the plane. IEEE Transactions on Information Theory, 294))551-555, 1983.
[47] Herbert Edelsbrunner. Alpha shapes survey. Tessellations the Sciences, 01 2010.
[48] Linning Xu, Vasu Agrawal, William Laney, Tony Garcia, Aayush Bansal, Changil Kim, Samuel Rota Buld, Lorenzo Porzi, Peter Kontschieder, Aljavz Bovzive, Dahua Lin, Michael Zollhöfer, and Christian Richardt. VR-NeRF; High- fighh-fieeitty virtualized walkable spaces. SIGGRAPH Asia Conference Proceedings, 2023.
[49] Daniil Lisus, Connor Holmes, and Steven Waslander. Towards open world nerf-based slam. In 2023 20th Conference on Robots and Vision (CRV). IEBE, June 2023.
[50] Shu-Sheng Liaw Hsiu-Mei Huang and Chung-Min Lai. Exploring learner acceptance of the use of virtual reality in medical education: a case study desktop and projection- based display systems. Interactive Learning Environments, 24(1):3-19, 2016.
13

---

[51] Deep Learning of Cross-Modal Tasks Conceptual Design of Engineered Products: A Review, volume Volume 6: 34th International Conference on Design Theory and Methodology (DTM) of International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, 08 2022.
[52] Artur Kasymov, Bartosz Czekaj, Marcin Mazur, and Przemystaw Spurek. Neggs: Negative gaussian splatting. arXiv preprint arXiv:2405./816. 2024.
[53] Dawid Malarz, Weronika Smolak, Jacek Tabor, Stawomir Tadeja, and Przemyslaw Spurek Gaussian splatting with nerf-based color and opacity.
[54] Joanna Kaleta, Kacper Kania, Tomasz Trzcinski, and Marek Kowalski, Lumigauss: High-fidelity outdoor relighting with 2d gaussian splatting. arXiv preprint arXiv:2408.044: 2024.
[55] Joanna Waczyúska, Tomasz Szezepanik, Piotr Borycki, Stawomir Tadeja, Thomas Bohné, and Przemyslaw Spurek. Mirage: Editable 2d images using gaussian splatting. arXiv preprint arXiv:24000015/1, 2024.
[56] Piotr Borycki, Weronika Smolak, Joanna Waczyúska, Marcin Mazur, Slawomir Tadeja, and Przemystaw Spurek. Gasp: Gaussian splatting for mhyyic-basedd simulations. arXiv preprint arXiv:2409.05819, 2024.
[57] Sida Peng. Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9054-9063. 2021.
[58] Chung Yi Weng, Brian Curless, Pratul P Srinivasan, Jonathan T Barron, and Ira Kemelmacher- Shlizerman. Humannerf: Free-viewpoint rendering of moving people from monocular video. In Proceedings of the IEEE/CVF conference on computer vision and pattern Recognition, pages 16210-16220, 2022.
[59] Yue Chen, Xuan Wang. Xingyu Chen, Qi Zhang, Xiaoyu Li, Yu Guo, Jue Wang, and Fei Wang. Uv volumes for real-time rendering of editable free-view human performance. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16621-16631, 2023.
[60] Zhe Li, Zerong Zheng, Lizhen Wang, and Yebin Liu. Animatable gaussians: Learning pose- dependent gaussian maps for high-fidelity human avatar modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19711-19722. 2024.
[61] Wojeiech Zielonka, Timur Bagautdinov, Shunsuke Saito, Michael Zollhöfer, Justus Thies, and Javier Romero. Drivable 3d gaussian avatars. arXiv preprint arXiv:2311.0858/ 2023.
[62] Jing Wen, Xiaoming Zhao, Zhongzheng Ren, Alexander G Schwing, and Shenlong Wang. Gomavatar: Efficient animatable human modeling from monocular video using gaussians-on- mesh. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2059-2066 2024.
[63] Shenhan Qian, Tobias Kirschstein, Liam Schoneveld, Davide Davoli, Simon Giebenhain, and Matthias NieBner. Gaussianavatars: Photorealistic head avatars with rigged 3d gaussians. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20299-20309, 2024.
[64] Jun Xiang, Xuan Gao, Yudong Guo, and Juyong Zhang. Flashavatar: High-fidelity digital avatar rendering at 300fps. arXiv preprint arXiv:23/2.02211 2023.
[65] Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero. Learning a model of facial shape and expression from 4d scans. ACM Trans. Graph., 36(6):194-1,2011
14

---

A Extended related work and potential applications of D-MiSo
The detail-capturing capabilities of Sub-Gaussians can be effectively adapted to other GS-based approaches. particularly cases where native methods lack dedicated support for high-detail recon- struction [52 53 54] or edition [55 [6]]
Recent studies have investigated the integration of human faces, avatars, and meshes for 3D hu- man modeling and rendering from video, presenting a promising avenue for enhancing detailed reconstruction.
For instance, [57] introduces human body representation where neural features across frames share a latent code anchored to a deformable mesh, facilitating cross-frame integration and efficient 3D representation learning in sparse monocular video scenarios. Similarly, HumanNeRR (58] optimizes volumetric human representation in canonical T-pose, using decomposed motion field to handle both skeletal and non-rigid deformations. This method provides high-quality, tree-vie renderings in challenging settings. UV-Volumes [59] further address the high computational costs associated with NeRF, achieving real-time, editable video rendering by encoding appearance details into 2D neural texture stacks for efficient 3D density and texture estimation.
Gaussian-basee representations have gained traction for modeling dynamic avatars, as in [60 61 62 63 (64,, with most approaches leveraging video data and human pose/face priors like FLAME fitting 665 Building on these advances, we propose that D-MiSo concepts could be effectively adapted for dynamic motion modeling and small detail capture in avatar representations, Specifically, Stage 1 could focus on integrating such models for capturing fine-gratd tne-grained avatar motion with anchored details.
B Extension of numerical results from the main paper
This section contains numerical comparison of the D-MiSo with other models regarding the experiments described in the main document. We used additional LPIPS and SSIM metrics. During the experiments,
used the RTX4090 GPU and densification until 5000 iterations (Ist Stage). We see that we get comparable competitive results. We can see that our method achieves better results for five objects from the D-NeRF dataset in terms of the LPIPS metric (Tab. 4). Full numerical comparison with metrics for the NeRF-DS dataset shows that we are comparable to other methods in the reproduction task (Tab. 55.
Table 4: Quantitative comparison on D-NeRF dataset
Methods
Hook
Jumpingjacks
Trex
BouncingBalls
PSNR
SSIM
LLPIPS
PSNR
1SSIM [LPIP
PSNR
1SSM
LPIPS
PSNR
1SSIM LLPIPS
D-NeRF
29.25
.968
.1120
32.80
.981
.0381
31.75
.974
.0367
38.93
.987
.1074
TiNeuVox-BB B
31.45
.971
.0569
34.23
.986
.0383
32.70
.987
.0340
40.73
.991
.0472
Tensor4D
29.03
.955
.0499
24.01
.919
.0768
23.51
.934
.0640
25.36
.961
.0411
K-Planes
28.59
.953
.0581
32.27
.971
.0389
31.41
.980
.0234
40.61
.991
.0297
FF-NVS
32.29
.980
.0400
33.55
.980
.0300
30.71
.960
.0400
40.02
.990
.0400
4D-GS
30.99
.990
.0248
33.59
.990
.0242
32.16
.988
.016
38.59
.993
.0267
SC-GS
39.87
.997
.0076
41.13
.998
.0067
41.24
.998
.0046
44.91
.998
.0166
D-MiSo (our)
38.13
.0086
42,05
.995
.0049
40.88
.996
.0029
41.49
.993
Methods
Hellwarrior
Mutant
Standup
Average
PSNR
SSIM tLPIPS J. PSNN
1SSIM LLPIPS
PSNR
tSSIM
LLPPPS
PSNR
tSSIM 1LPIPS L
D-NeRF
25.02
.0633
31.29
.0212
32.79
.991
.0241
31.69
.975
.0775
Vox-B
28.17
.978
.0706
33.61
.982
.0388
35.43
.991
.0230
33.76
.983
.0441
Tensor4D
31.40
.925
.0675
29.99
.951
.0422
30.86
.964
.0214
27.62
.947
.0471
K-Planes
25.27
.948
.0755
33.79
.982
.0207
34.31
.984
.0944
32.32
.973
.0382
FF-NVS
27.71
.970
.0500
34.97
.980
.0300
36.91
.990
.0200
33.73
.979
.0357
4D-GS
31.39
.974
.0436
35.98
.996
.0120
35.37
.994
.0136
34.01
.987
.0316
SC-GS
42.93
.994
.0155
45.19
.999
.0028
47.89
.999
.0033
43.31
.977
.0063
D-MiSo (our)
41.49
.986
.0173
44.38
.997
.0026
47.66
.998
.0166
42.27
.993
.0055
15

---

Table 5: Quantitative comparison on a NeRF-DS dataset
Methods
Bell
Sheet
Press
Basin
PSNR
SSIM
MLPIPS 4 PSNR
1SSIM 1LPIPS
PSNR
1SSIM 1LPIPS PSNR
1SSIM 1LPIPS 4
HypenNeRR
24.0
8884
.159
24.3
.844
.148
25.4
.873
.164
20.2
.829
.168
NeRF-DS
23.3
.872
.134
25.7
.918
.115
26.4
.911
.123
20.3
.668
.127
Vox-
23.1
876
.13
21.1
.745
.234
24.1
.892
.133
20.7
.86
.105
SC-GS
25.1
918
.117
26.2
.988
.142
26.6
.901
.135
19.6
.846
.154
D-MiSo (our)
25.3
846
.174
25.8
.875
.211
25.6
.867
.206
19.8
.788
.217
Methods
Cup
Sieve
Plant
Average
PSNR
SSIM 1LPIPS J. PSNR
1SSIM 1LPIPS PSNR
1SSIM 1LPIPS
PSNR
1SSIM 1LPIPS +
HyperNeRR
20.5
.705
.318
25.0
.909
.129
18.1
.714
.359
22.5
.827
.206
NeRF-DS
24.5
.916
.118
26.1
.935
.108
20.8
.867
.164
23,9
.898
.127
Vox-u
20.5
.806
.182
20.1
.822
.205
20.6
.863
.611
21.5
.843
.162
SC-GS
24.5
.916
.115
26.0
.919
.114
20.2
.837
.202
24.1
.891
D-MiSo (our)
24.5
.874
.185
26.5
.881
.159
20.8
.815
.234
24.0
.449
Table 6: Storage cost, training time, fps, and batch study on a D-NeRF dataset
PSNR Trex
Hook
Jumpin.
Boune.
Hell.
Mutant
Standup
Batch:
4
4
PSNR
37.77
40.42
39.56
40.63
41.44
43.38
46.07
Time 1st stage
00:02:48
00:02:22
00:02:41
00:03:12
00:02:27
00:03:46
00:03:17
Time 2nd stage
1:40:15
1:16:18
2:14:09
1:43:08
1:01:27
1:48:49
1:19:32
Storage cost
76MB
53.5MB
122MB
131MB
27MB
73MB
43MB
Rendering time [fps]
138
175
90
123
185
138
169
Batch:
8
8
8
PSNR
38.07
41.65
40.74
40.55
41.59
44.40
47.22
Time 1st stage [h]
00:05:30
0:05:17
00:05:38
00:05:16
00:04:28
00:05:51
00:05:56
Time 2nd stage [h]
2:27:08
2:03:25
2:42:22
2:40:30
1:43:33
2:18:43
2:12:12
Storage cost
32MB
24MB
51MB
80MB
16MB
28MB
20MB
Rendering time/fps]
192
190
143
153
205
188
194
In our method, batch of images taken input to the model. Tab. 6|shows numerical comparison using batch: 4, 8 on D-NeRF. Training takes 80 thousand iterations, and the second stage starts at the 5 thousandth iteration. Fach the SC-GS implementation. Tab, also presents the training time and storage cost, as well as the FPS for rendering for each dataset. These results suggest that our model is memory efficient and that the time required to train it is minimal. With batch equal to 8 in all cases, storage costs decreased, and performance improved with a trade-off of increased training time.
has attached 25 Sub-Gaussians. This is comparable to
similar batch study has been done for the NeRF-DS dataset presented in T We can see that batch is playing bigger role in the dataset, which is considered more pronounced move.
We contrasted our model with and without the Sub-Rotation Network. The Sub-Rot Network, despite being simple single-layer network, plays crucial role in the second stage. Tab. [8shows numerical comparison of PSNR for models with and without the Sub-Rot Network. Furthermore, we would like emphasize that due its shallow architecture, incorporating the Sub-Rot Network does not significantly impact training time. We decided to present the results on the jumpingjack dataset distinguishing between batches of 4, 8. These results show that the Sub-Rot Network is crucial to obtaining SOTA PSNR. With Sub- Rot Network, we obtain approximately 0.5 PSNR crucial to obtain the reconstruction of small elements (like human fingers in the jumpingjack dataset)
Tab. shows how the size of the network influences training time and render speed. These results show that deformable networks and Sub-Rot Networks are not costly in terms of rendering time, resulting real-time rendering. In practice, batch size has a higher impact than deformation network depth.
16

---

Table 7: Batch study on NeRF-DS dataset
Batch size
Sheet *SIIM 1IPPIP
Press
Basin
PSNR
SSIM -TPPPS
PSNR
PSNN
*SSIM 11.PIPS
PSNR 1SSIM 1LPIPS
25.1
851
165
24.8
863
.219
25.2
.58
.205
19.7
.788
.217
24.5
.332
.190
25.1
.833
.202
25.3
.60
.215
19.8
.781
.245
25.3
846
.174
25.8
875
.211
25.6
867
.206
19.6
.785
.077
25.1
844
.190
25.6
.873
.216
24.3
.842
.273
19.7
.785
.229
Batch size
Cup
Sieve
Plant
Average
PSNR
SSIM *LPIPS
JIPSNR
*SSIM "LPIPS J JPPNN
*SSIM 1LPIPS JIPSNR 1SSIM 1LPIPS
24.0
884
.163
26.5
881
.158
20.2
.812
.226
23.6
.848
.193
24.2
886
.163
25.8
.871
.167
20.8
.815
.234
23.6
.845
.202
24.4
888
166
26.0
.85
.169
20.6
.817
.232
23.9
.849
.195
24.5
.874
185
25.8
864
.95
20.5
.808
.245
23.6
.841
.219
Our model operates on several hyperparame- ters, primarily derived from the basic GS frame- work. Given the introduction of additional stages and the multi Gaussian component, one of the new hyperparameters is the number of Grre-Gasssians and Gaussinns (Tab. TО). The ablation study also focuses on the influence of the Sub-Rot Network training time and PSNR (Tab. [). Tables show training time and PSNR analysis.
Table 8: Training time corresponding the num- ber of Sub-Gaussians and iteration start of the de- form network using the jumpingjack dataset.
batch 4
batch 88
Sub-Rot Network PSNR
with without with without
40.42
39.98
41.65 41.27
Training time [h]
1:18 1:12
2:08 1:45
Rendering time [fps]
175
186
190 259
The number of Core-Gaussians is determined automatically during stage 1, utilizing the pruning mechanism implemented in GS. The Tab. [00 illustrates the training time corresponding parameters using the jumpingjacks dataset as an example with batch equal to 8. We can see the speed of the 1st stage, which used for Bare-Gaussian preparation. Moreover, too few More-Gaussians can cause a drop in render quality (PSNR metric). The experiments suggest the number Wore-Gaussians sufficient after the first phase. In all cases, larger number of Sub-Gaussians improved the quality of renders.
these
Table Deformation network depth (numbers of layers) and batch study on a jumpingjack dataset from D-NeRF.
Def. net. depth
4
6
8
10
6
8
10
Batch
4
8
8
PSNR
40.91
40.75
40.42
40.26
41.86
42.01
41.65
41.37
Training time [h]
1:21
1:22
1:18
1:31
2:00
1:58
2:08
2:07
Render time [fpss]
138
167
175
144
227
221
190
192
Table 10: Training time corresponding to the number of Sub-Gaussians and iteration start of the deform network using the jumpingjack dataset.
n s_b
1
10
25
1
25
10
25
Iter. start of def. net.
2000
2000
2000
3000
3000
3000
5000
5000
5000
core after 1st stage
3290
3340
3434
2841
2974
2936
1803
1798
1807
PSNR
38.94
41.35
41.65
38.52
41.14
41.41
33.63
37.14
37.47
Train time 1st stage [h]
00:05:14
00:04:48
00:05:17
00:04:17
00:04:33
00:04:16
00:03:27
00:03:02
00:03:21
Train time 2nd stage [h]
1:51:17
1:44:21
2:03:25
1:39:39
1:53:18
1:50:25
1:48:54
1:41:12
1:56:28
Sum: Train time [h]
1:56:31
1:49:09
2:08:42
1:43:56
1:57:51
1:54:41
1:52:21
1:44:14
1:59:49
C Extension of examples modification
Below is example of scene modification from the PanoptieSports dataset (Fig. 14). The dynamics of the object were stopped, so as to show the possibility of changing the position of the ball at a given time instant
Due to characteristics Multi-Gaussians and the ease of their control, we show that we can casily duplicate and/or scale selected objects (Fig. 15
17

---

The third method of modification allows us to give new dynamics and fluidity to the object (Fig. T6p. We can see that even difficult edits like bending the back, changing the shape the face, and bending the hand are possible while the result is natural from the point of view of graphics. This is made possible by editing the scale and rotation of Gaussians an explicitly defined way.
Reconstruction
Ball position modification
Figure 14: Example ball position modification on 360 scene from PanopticSports Datasets
econstruction Renders
Example of modification
Figure 15: Example of animation by duplicating elements or changing their scale
Time (new dynamic)
Figure 16: Example of third way of modification: transformation of space, allowing to give new dynamics.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly outline the claims made the paper, detailing the contributions, key assumptions, and scope. The claims consistent with experimental results, indicating the extent to which the findings can be generalized to other settings.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The paper clearly presents both technical as well as ethical limitations in dedicated paragraph.
Justification:
Guidelines:
The answer NA means that the paper has no limitation, while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
19

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the suppiemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: The paper provides comprehensive technical details necessary for reproducing the code, along with datasets and experimental settings. The authors will also release the code and resulting checkpoints to ensure complete reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
20

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The code, along with the checkpoints used to report results, is publicly available. Link is provided Experiments section. All experiments were conducted using publicly available datasets.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
gewiii/gguee//eeeeeeeioiiiiiseee CodeeSumiiiseeeeee onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All results were obtained using official data splits which are stated in the paper. The hyperparameters of the model and their influence also thoroughly discussed.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Statistical significance is not provided due to the large computational resources required. No related work provides such statisties, making it unfeasible recompute all results for comparison with our method,
Guidelines:
The answer NA means that the paper does not include experiments.
21

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Justification:
Compute resources used for experiments are listed the Experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics / http://awwraarcooouuuu/ccccccaaa
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the broader impact of our work the Broader impact paragraph in Conclusion section.
Guidelines:
22

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cited all utilized works and resources according to the citing guidelines.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
23

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not relate new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codeemoodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "obyzpYxBKN", "reviewer": "Reviewer_u5NP", "summary": "This paper introduces a Dynamic Gaussian Splatting representation that allows for easier object shape editing at test time. This is achieved through the use of Dynamic Multi-Gaussian Soup (D-MiSo), a mesh-inspired multi-gaussian system. Specifically, a GS is divided into two components: Core-Gaussian, which models the transformation of groups of sub-gaussians, and sub-Gaussian, which handles the final geometry and color. This enables different levels of editing by either changing the core-Gaussian or sub-Gaussians for more fine-grained control.", "questions": "1) Is there a strategy for selecting hyperparameters such as the number of Gaussian and Sub-Gaussian components?\n2) It is unclear why the proposed method would perform worse than SC-GS on metrics like PSNR/SSIM for NeRF-DS and D-NeRF. Does this suggest that increased editability comes at the cost of model performance, or is it simply a matter of hyperparameter tuning?", "limitations": "The authors have discussed the limitations of their method, but I was unable to locate a statement regarding the broader impacts of the proposed method in the Limitations section, as claimed by the author.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1) The paper demonstrates substantial improvement in the level of control and editability of Gaussian Splatting (GS) at test time, while maintaining competitive PSNR/SSIM scores.\n2) I appreciate the inclusion of mesh visualization estimates (Figures 9 and 11), which make the method more convincing.\n3) The method is tested on various datasets with varying complexity and shows good performance.", "weaknesses": "I find the paper's writing style difficult to follow and some sentences do not even make sense. Additionally, there are numerous components involved in the pipeline that are challenging to keep track of: multi-Gaussian, core-Gaussian, sub-Gaussian, core-triangle soup, multi-triangle soup, and sub-Gaussian soup (which may be the same thing?). I would appreciate a revision in writing to improve clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for the constructive remarks regarding our manuscript, to which we responded next to the posted questions below. We will revise the manuscript in accordance with the raised concerns.\n\n***W1 “I find the paper's writing style difficult to follow …  I would appreciate a revision in writing to improve clarity.”***\n\nThank you for this comment. We appreciate the complex nature of our manuscript, and we will edit all the relevant parts for clarity. In the paper, we introduced a Multi-Gaussian component, which is comprised of a Core-Gaussian \"parent\" and Sub-Gaussian \"child,\" each parameterized by a triangle. Accordingly, the Core-Triangle corresponds to the Core-Gaussian, and the Sub-Triangle corresponds to the Sub-Gaussian. The terms \"Triangle\" or \"Gaussian\" are used contextually:\n\n- When discussing rendering or properties such as color and transparency, we use \"Gaussian.\"\n\n-  When referring to parameterized Gaussians and their modifications (e.g., location, scale, rotation), we use \"Triangle.\"\n\nFig. 3 illustrates these differences. **We will explain all these objects and clarify them in the Introduction.**\n\n***Q1:  Is there a strategy for selecting hyperparameters such as the number of Gaussian and Sub-Gaussian components?***\n\nOur model operates on several hyperparameters, primarily derived from the basic Gaussian Splatting (GS) framework. Given the introduction of additional stages and the multi-Gaussian component, one of the new hyperparameters is the number of Core-Gaussians and Sub-Gaussians. We are pleased to see this novel aspect being acknowledged. The number of Core-Gaussians is determined automatically in stage 1, utilizing the pruning mechanism implemented in GS. \nThe Table below illustrates the training time corresponding to these parameters using the *jumpingjacks* dataset as an example. During the experiments, we used the RTX4090 GPU and densification until 5000 iterations (1st stage). We can see the speed of the 1st stage, which is used for Core-Gaussians preparation. Moreover, too few Core-Gaussians can cause a drop in quality (PSNR metric). The sub-rot network is shallow enough (it is a single layer in all of our experiments) that the increased cost of training time is not apparent. Therefore, the experiments suggest the number of Core-Gaussians is sufficient after the first phase. In all cases, a larger number of Sub-Gaussians improved the quality of renders. \n\n|n_sub|1|10|25|1|10|25|1|10|25|\n|---|---|---|---|---|---|---|---|---|---|\n|Iteration - start of deform network|2000|2000|2000|3000|3000|3000|5000|5000|5000|\n|N_core  after 1st stage|3290|3340|3434|2841|2974|2936|1803|1798|1807|\n|PSNR|38.94|41.35|41.65|38.52|41.14|41.41|33.63|37.14|37.47|\n|Train time 1st stage [h]|00:05:14|00:04:48|00:05:17|00:04:17|00:04:33|00:04:16|00:03:27|00:03:02|00:03:21|\n|Train time 2nd stage [h]|1:51:17|1:44:21|2:03:25|1:39:39|1:53:18|1:50:25|1:48:54|1:41:12|1:56:28|\n|Sum: Train time [h]|1:56:31|1:49:09|2:08:42|1:43:56|1:57:51|1:54:41|1:52:21|1:44:14|1:59:49|\n\n***Q2: It is unclear why the proposed method would perform worse than SC-GS on metrics like PSNR/SSIM for NeRF-DS and D-NeRF. Does this suggest that increased editability comes at the cost of model performance, or is it simply a matter of hyperparameter tuning?***\n\nOur main contribution is the ability to edit a dynamic scene better and more efficiently. As pointed out, there is generally a trade-off between editing and the quality of the reconstruction. However, **in our model, such effects are minimal**. Consequently, we obtained markedly worse results on synthetic datasets (Tab 1 in our draft paper) **and better results achieved on full scenes** (Tab. 3 and Tab. 4  in our draft paper). We tentatively believe this is caused by the suboptimal algorithm used to choose core points in SC-GS. SC-GS uses a heuristic approach based on distances to select core points, which might be problematic on a large scale. Our solution uses classical GS optimization procedures to establish the Gaussian number and size. Therefore, D-Miso covers 3D scenes better. \n\n***L1: The authors have discussed the limitations of their method, but I was unable to locate a statement regarding the broader impacts of the proposed method in the Limitations section, as claimed by the author.***\n\nOur model significantly improves rendering quality and advances 3D scene reconstruction and rendering, impacting multiple domains by enabling more realistic and efficient 3D modeling and animation. This technology could enhance VR/AR experiences [a1], robotics [a2], and medical imaging [a3]. It could also be used for interactive education [a3], scientific visualization, and a plethora of other commercial applications like product design and real estate [a4]. We will edit our manuscript to include these considerations and potential application fields of our method.\n\n[a1] Linning et al. 2023, “VR-NeRF: High-Fidelity Virtualized Walkable Spaces”, SIGGRAPH Asia 2023 Conference Papers\n\n[a2] Lisus et al., 2023, \"Towards Open World NeRF-Based SLAM,\" 2023 20th Conference on Robots and Vision (CRV), Montreal, Canada, pp. 37-44, doi: 10.1109/CRV60082.2023.00013\n\n[a3] Huang et al. 2013, “Exploring Learner Acceptance of the Use of Virtual Reality in Medical Education: A Case Study of Desktop and Projection-Based Display Systems,” Interactive Learning Environments 24 (1): 3–19.\n\n[a4] Li et al. 2022, “Deep Learning of Cross-Modal Tasks for Conceptual Design of Engineered Products: A Review”, ASME 2022 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QQoDJ7WbQY", "reviewer": "Reviewer_hxyh", "summary": "This paper proposes a novel framework for modeling and editing dynamic scenes. The authors used a two-pass Multi-Gaussian approach to represent the entire scene. First, they obtained relatively stable Core Gaussians through initialization, and then used the Core Gaussians to drive Sub-Gaussians to fit the entire scene. To better edit motion, the authors parameterized each Gaussian with triangle soup. Experimental results show that this method not only matches the rendering quality of SC-GS but also enables the editing of more extreme large motions.", "questions": "A small tip: In Tab. 1, the authors used the metrics from the SC-GS paper. However, SC-GS did not use a consistent background; for example, the `Bounce` (and maybe `Trex`) scene used a white background. The authors clearly used a black background consistently. Although the reported metrics in the table show a slight decrease compared to SC-GS, I greatly appreciate the authors' honesty.", "limitations": "Please refer to the weakness part.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is clear and easy to follow. The dynamic reconstruction method based on Gaussian splatting inherently has advantages in motion editing, but this direction has not been well-explored by the community. Therefore, I am very grateful to the authors for focusing on improving the motion editing capabilities of dynamic scenes and achieving impressive editing results (as shown in Fig. 4). The triangle-soup-based motion editing proposed by the authors really makes sense. Additionally, the honest comparison of rendering metrics in both synthetic and real scenes is also appreciated.", "weaknesses": "1. I think the following papers should also be cited, because they have made significant contributions to the early dynamic scenes reconstruction based on Gaussian splatting.:\n   - (CVPR 2024) Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis by Zhan Li et al.\n   - (ICLR 2024) Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting by Zeyu Yang et al.\n2. L136, The concept of anchor Gaussian originally comes from Scaffold-GS [1], not from Spec-Gaussian. It would be even better if Scaffold-GS could be cited.\n3. It would be even better if Deformable-GS [2] were included in the comparison of quantitative metrics (such as Tabs. 1-2), as it is the first deformation-based dynamic Gaussian splatting method and serves as the baseline for SC-GS.\n4. I think the ablation study is not thorough enough. Although the ablation on batch size is appreciated, I am unclear about the roles of other components of the method. Corresponding ablation studies are needed to make the paper more robust. For example, I am very interested in understanding the impact of the `Sub-Rot Network` mentioned in L218 on the rendering metrics.\n\n\n\n[1] Tao Lu, Mulin Yu, Linning Xu, Yuanbo Xiangli, Limin Wang, Dahua Lin, and Bo Dai. 2023. Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering. arXiv preprint arXiv:2312.00109 (2023)\n\n[2] Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. arXiv preprint arXiv:2309.13101,2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for the vote of confidence in our manuscript and for the in-depth feedback and suggestions, which we are happy to incorporate and feel will improve the manuscript. We respond to all the questions and concerns raised in the answers below.\n\n***W1: I think the following papers should also be cited, because they have made significant We contributions to the early dynamic scenes reconstruction based on Gaussian splatting …***\n\nThank you for suggesting this; we will cite these papers in our manuscript. Method [b1] proposes approximating the spatiotemporal 4D volume of a dynamic scene by optimizing a collection of 4D primitives with explicit geometry and appearance modeling. This method uses 4D Gaussians parameterized by anisotropic ellipses and view-dependent, time-evolved appearances represented by 4D spherical harmonics coefficients. In [b2], a novel scene representation called Spacetime Gaussian Feature Splatting has been introduced. This approach extends 3D Gaussians with temporal opacity and parametric motion/rotation, enabling the capture of static, dynamic, and transient scene content. Additionally, the method incorporates splatted feature rendering to model view- and time-dependent appearances while maintaining a compact representation size. The method is notable for its high rendering quality and speed while also being storage-efficient. We agree that both these works have significantly contributed to the reconstruction of dynamic scenes and align well with the topic under discussion. **Both of them will be included in our manuscript.**\n\n[b1] Li et al., 2024, “Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition\n\n[b2] Yang et al., 2024. “Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting”, Proceedings of the International Conference on Representation Learning \n\n***W2: L136, The concept of anchor Gaussian originally comes from Scaffold-GS [1], not from Spec-Gaussian. It would be even better if Scaffold-GS could be cited.***\n\nThank you for this excellent comment. In [c1], the authors introduce the concept of anchor points to tackle the problem of overfitted models caused by redundant Gaussians. Scaffold-GS addresses this issue by distributing local 3D Gaussians according to anchor points and predicting attributes based on viewing direction and distance. This approach reduces redundancy, enhances scene coverage, and maintains high-quality rendering with improved robustness to view changes. **Therefore, we will provide an extended description of [c1] in Related Works.**\n\n[c1] Lu et al., “Scaffold-GS: Structured 3D Gaussians for view-adaptive rendering,” Proceedings of the IEEE/CVF, CVPR 2024\n\n***W3: It would be even better if Deformable-GS [2] were included in the comparison of quantitative metrics (such as Tabs. 1-2), as it is the first deformation-based dynamic Gaussian splatting method and serves as the baseline for SC-GS.***\n\nThe model Deformable-GS [d1] is indeed a prototype of the SC-GS model. Deformable-GS is one of the first works targeting deformable/dynamic scenes. It introduces an MLP-based deformation network to adapt Gaussian parameters to a given timestep. Below are the PSNR results of our comparison with SC-GS and Deformable-GS, which we will include in our article. \n\n|Dataset:|Hook|Jumpin|Trex|Bounce|Hell|Mutant|Standup|\n|---|---|---|---|---|---|---|---|\n|Deformable-GS|37.42|37.72|38.10|41.01|41.54|42.63|44.62|\n|SC-GS |39.87|41.13|41.24|44.91|42.93 |45.19 |47.89|\n|D-MiSo (our) |38.13|42.05|40.88 |41.49 |41.49 |44.38|47.66|\n\n\n[d1] Yang et al., 2023, “Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction”, arXiv preprint arXiv:2309.13101.\n\n***W4 I think the ablation study is not thorough enough. Although the ablation on batch size is appreciated, I am unclear about the roles of other components of the method. Corresponding ablation studies are needed to make the paper more robust. For example, I am very interested in understanding the impact of the Sub-Rot Network mentioned in L218 on the rendering metrics.***\n\nThe Sub-Rot Network, despite being a simple single-layer network, plays a crucial role in the 2nd stage. We appreciate it being highlighted for further analysis, as this allows us to demonstrate its significance. Below is a numerical comparison of PSNR for models with and without the Sub-Rot Network. Furthermore, we would like to emphasize that due to its shallow architecture, incorporating the Sub-Rot Network does not significantly impact training time. We decided to present the results on the *jumpingjack* dataset distinguishing between batch = {4, 8}. The experiments we performed using the RTX4090 GPU.\n\n|   | batch = 4  |batch = 4   |batch = 8   |  batch = 8 |\n|---|---|---|---|---|\n| Sub-Rot Network  | with  |without   | with  |  without |\n|PSNR | 40.42 | 39.98 | 41.65 | 41.27|\n|Training time [h] | 1:18|1:12|2:08|1:45|\n|Rendering time [fps] |175 |186|190|259|\n\nSince the Sub-Rot Network used is not deep and has only a single layer, it does not visibly affect the training time. Additionally, Reviewer 1 (u5NP) pointed us to consider the role of a number of Sub-Gaussian/Core-Gaussian. We agree that this is an interesting analysis, and we will incorporate it in the revised manuscripts. \n\n***Q1: A small tip: In Tab. 1, the authors used the metrics from the SC-GS paper. However, SC-GS did not use a consistent background; for example, the Bounce (and maybe Trex) scene used a white background. The authors clearly used a black background consistently. Although the reported metrics in the table show a slight decrease compared to SC-GS, I greatly appreciate the authors' honesty.***\n\nThank you for raising this point. We believe it is important to use a unified framework for evaluation. Therefore, following most papers, we introduced results obtained on a black background.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GsanJ6El8u", "reviewer": "Reviewer_qmWF", "summary": "This paper is developed based on SC-GS, which enhanced 3DGS with Deformed Control Points to model low-rank dynamics and modify dynamic objects over time. SC-GS necessitates selecting elements that need to be kept fixed and centroids that should be adjusted throughout editing, and it poses additional difficulties regarding re-productivity editing. This paper proposes Dynamic Multi-Gaussian Soup (D-MiSo), which serves as a kind of mesh representation of dynamic GS, links parameterized Gaussian splats, forming a Triangle Soup with the estimated mesh, and separately constructs new trajectories for the 3D objects composing the scene, which makes the scene’s dynamic editable over time or while maintaining partial dynamics.", "questions": "Lack of discussion with respect to the human-NeRF (Neural Body/HumanNeRF/UV-Volumes) and human-GS methods (Animatable Gaussians/D3GA/GoMAvatar).", "limitations": "The authors adequately addressed the limitation.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "D-MiSo estimates the mesh as a set of disconnected triangle faces in Multi-Gaussians and uses a dynamic function to control the vertices, which makes dynamic 3D-GS easier to modify than SC-GS. \n\nMulti-Gaussians = Core-Gaussians + Sub-Gaussians. Core-Gaussians are an alternative to the control points discussed in SC-GS, with the added advantage of allowing individual modifications. Sub-Gaussians are defined by principal components of core-Gaussian, such that modifying Core-Gaussians will also change Them, which allows scene modifications.\n\nAlthough the positions of Core-Gaussians are learned by deformation MLP during training, this paper can modify dynamic objects by using the vertex of the Sub-Gaussians or generate mesh from the Core-Gaussians during inference.\n\nD-MiSo can also handle affine transformation and object scaling.", "weaknesses": "Lack of training time comparisons.\n\nLack of comparison against gs2mesh-based methods and cage-based methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for the feedback and for pointing out improvements to our paper. We respond to these concerns in the points below.\n\n***W1: Lack of training time comparisons.***\n\n\nHere, we showcase a time comparison between SC-GS and D-Miso models, with the former operating noticeably faster than the latter. This can be attributed to two causes. First, unlike SC-GS, D-Miso benefits significantly from a larger batch size during training. Second, D-Miso incorporates two layers of Gaussian components (Core-Gaussians and Sub-Gaussians). On the other hand, SC-GS uses landmark points, making it faster in training but more constrained in editing and processing full scenes. Furthermore, our model obtains better results on real scenes, see Tab. 3 and Tab. 4.  \n\n\nDataset:| Hook|Jumpin|Trex|Bounce|Hell|Mutant|Standup|\n|---|---|---|---|---|---|---|---|\n| | | |**D-Miso**| | | | |\nBatch:|1|1|1|1|1|1|1|\nTime 1st stage [h] | 0:00:53 | 0:00:56 | 0:01:15 | 0:01:21 | 0:00:46 | 0:01:10 | 0:00:56|\nTime 2nd stage [h] | 0:59:03|0:53:41|1:13:51|1:23:45|0:24:35|1:09:26|0:41:41|\nBatch:|4|4|4|4|4|4|4|\nTime 1st stage [h] |00:02:48|00:02:22|00:02:41|00:03:12|00:02:27|00:03:46|00:03:17\nTime 2nd stage  [h] |1:40:15|1:16:18|2:14:09|1:43:08|1:01:27|1:48:49|1:19:32\nBatch:|8|8|8|8|8|8|8|\n|Time 1st stage [h] |00:05:30|0:05:17|00:05:38|00:05:16|00:04:28|00:05:51|00:05:56|\n|Time 2nd stage [h] |2:27:08|2:03:25|2:42:22|2:40:30|1:43:33|2:18:43|2:12:12|\n| | | |**SC-GS**| | | | |\nBatch:|1|1|1|1|1|1|1|\n|Time|0:24:52|0:21:31|0:30:05|0:30:02|0:17:37|0:27:26|0:20:23|\n\n\n\n\n***W2: Lack of comparison against gs2mesh-based methods and cage-based methods.***\n\n**- GS2Mesh based methods:**\nWe have already included the following mesh-based Gaussian splatting works, which operate in static setups [3, 26, 27].\nTo address the Reviewer’s remark, we will also include a more recent work that combines meshes with Gaussians and operates in dynamic scenarios [e1]. This framework reconstructs a high-fidelity and time-consistent mesh from a single monocular video. Building on this representation, DG-Mesh recovers high-quality meshes from the Gaussian points and can track mesh vertices over time, enabling applications such as texture editing on dynamic objects. However, the method relies on a Poisson Solver and differentiable Marching Cubes to recover the deformed surface, significantly complicating and slowing down the pipeline. Moreover, it does not explore geometry modification capabilities, which constitute a significant aspect of our work.\n\n**- Cage based methods:**\nWe thank the reviewer for suggestions regarding cage-based methods. In response, we will include two recent works that utilize the cage concept. Cage-based methods like [e2, e3] are crucial for the efficient and intuitive manipulation of 3D scenes. By providing a direct deformation approach, these methods simplify a traditionally complex task, making it more accessible. However, they require complex cage-building steps, which complicates the pipeline. These works in their current form do not address training on dynamic scenes so direct comparison to our method is not feasible. Additionally, cage-based methods may lack the flexibility and precision of manual, vertex-level deformation techniques, potentially missing fine details. In contrast, our work emphasizes small and precise element movement, addressing this limitation. \n\n[e1] Liu et al. 2024, “Dynamic Gaussians Mesh: Consistent Mesh Reconstruction from Monocular Videos,” arXiv preprint arXiv:2404.12379. Retrieved from https://arxiv.org/abs/2404.12379\n\n[e2] Huang et al. 2024, “GSDeformer: Direct Cage-based Deformation for 3D Gaussian Splatting,” arXiv preprint arXiv:2405.15491. Retrieved from https://arxiv.org/abs/2405.15491 \n\n[e3] Jiang et al. 2024, “VR-GS: A Physical Dynamics-Aware Interactive Gaussian Splatting System in Virtual Reality,” arXiv preprint arXiv:2401.16663\n\n***Q1: Lack of discussion with respect to the human-NeRF (Neural Body/HumanNeRF/UV-Volumes) and human-GS methods (Animatable Gaussians/D3GA/GoMAvatar).***\n\nSeveral works combine human faces/avatars and meshes, and those based on Gaussian representations are particularly relevant to this discussion. Examples include [f1-f5] - all of them can model dynamic avatars, most based on video data. However, these specialized works rely on human pose/face priors (e.g., FLAME fitting [f6]). We believe it would be valuable to explore the potential of adapting D-MiSo concepts in this context, building on existing works in the field. We envision Stage 1 focusing on effectively integrating such models in motion while the anchoring concept could be adapted to efficiently capture the small details of moving avatars. However, human avatars are a broad topic, with several notable and advanced works emerging each month. A thorough and accurate analysis of related works is required to fully investigate the best utilization and adaptation of our method in this context. We include the following papers in the Related Work and Future Work Sections.\n\n[f1] Li et al. 2024, “Animatable Gaussians: Learning Pose-dependent Gaussian Maps for High-fidelity Human Avatar Modeling”, in Proceedings of the IEEE/CVF CVPR\n\n[f2] Zielonka et al. 2023, “Drivable 3D Gaussian Avatars”, arXiv preprint arXiv:2311.08581.\n\n[f3] Wen et al. 2024, “Gomavatar: Efficient Animatable Human Modeling from Monocular Video Using Gaussians-on-Mesh,” in Proceedings of the IEEE/CVF CVPR\n\n[f4] Qian et al. 2024, \"GaussianAvatars: Photorealistic head avatars with rigged 3d Gaussians,\" in Proceedings of the IEEE/CVF CVPR\n\n[f5] Xiang et al. 2023, \"Flashavatar: High-fidelity digital avatar rendering at 300fps.\" arXiv preprint arXiv:2312.02214\n\n[f6] Li et al. 2017, “Learning a model of facial shape and expression from 4D scans”, in ACM Transactions on Graphics (Proc. SIGGRAPH Asia), 36(6)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bd3RALZDQ6", "reviewer": "Reviewer_ZSU4", "summary": "This paper proposes the D-Miso pipeline, which includes multi-Gaussian components and two deformation networks to model global transformation and local deformation. A good training strategy is introduced to fit the proposed methods. The rendering quality looks good, and the editing results seem better than those of current motion-based sparse control point deformation using 3D Gaussians.", "questions": "1. More comparison and discussions with SC-GS should be included since SC-GS can also support editing and share the similar ideas with control points/Gaussians.\n2. As I understand, deformation-based methods find it challenging to synthesize correct novel views with large motion. Could the authors provide comprehensive insight into this issue?", "limitations": "Yes, the limiations have been addressed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper proposes the multi-Gaussian component, allowing large 3D scene elements. It seems like a novel idea.\n2. D-Miso uses multi-Gaussian components and two deformation networks for modeling dynamic scenes.\n3. The editing results seem better than those of SC-GS.", "weaknesses": "1. Multi-Gaussian representations are similar to SC-GS's control points. Since 3D/4D Gaussians themselves can model large 3D scene elements, this is not a strength. I think the contribution is minor, and it seems that no ablation study is presented in the paper.\n2. Most demos are provided in synthetic datasets, which is not convincing. Results on real-world datasets such as [9, 13] are recommended to be included.\n3.The \"Related Works\" section should be reorganized. More discussions on multi-view dynamic scenes and flow-based radiance fields should be included. \n4. If your method is based on [3], a brief review should be included in the main text as a preliminary to improve the presentation.\n5. More details such as storage costs and training time (for each stage) should be included. Additionally, would the rendering speed be affected by the large deformation network?\n6. Minor issues: Line 140: \"Guassinas\" -> \"Gaussians,\" Line 197: \"vertixes\" -> \"vertices,\" Fig. 10: \"possition\" should be \"position.\"\n7. What is n_core as mentioned in Eq.1?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the remarks and excellent comments and appreciate the time taken to review our manuscript. We have responded to these remarks below.\n\n***W1 Multi-Gaussian representations are similar to SC-GS's control points ...***\n\nOur solution differs in architecture and training strategy from SC-GS, which uses control points while we employ two hierarchies of Gaussian components. **More significantly, we have an extremely large possibility of editing dynamic scenes.** All other Reviewers agree that D-Miso introduces significant novelty, as underscored by Reviewer 2 (“Experimental results show that this method not only matches the rendering quality of SC-GS but **also enables the editing of more extreme large motions**”). The SC-GS authors use control points, which influence a large part of the object. Therefore, when we modify some object element (e.g., the hand of the object, see Fig.4 in our manuscript), the full object changes (in the case of the example from Fig. 4, the head is moving). Thanks to using two levels of Gaussian components, we can modify objects easily without artefacts (see Fig. 4 in the main paper). **Consequently, our new GS representation is better suited for editing dynamic scenes than SC-GS, and it is the paper's main contribution.**\n\n***W2A Most demos are provided in synthetic datasets, which is not convincing. Results on real-world datasets such as [9, 13] are recommended to be included.***\n\nThe SC-GS and other baseline models, which modify dynamic scenes, use only synthetic examples. In contrast, **our paper includes examples of full scenes (please see Fig. 3)**, where we modified real-world dynamic scenes. Furthermore, in  Appendix C and Fig. 14, we will also provide additional real-scenes modifications. We also add movies (GIFs) from real scene modification to supplementary materials. We agree that such experiments are valuable, and **we will move more examples from the appendix to the main paper.**\n\n***W2B The \"Related Works\" section should be reorganized. More discussions on multi-view dynamic scenes and flow-based radiance fields should be included.*** \n\nEarly works on dynamic scenes face difficulties when dealing with monocular settings and uncontrolled or lengthy scenarios. To enhance scene motion modeling, some works in **NeRFs utilize flow-based techniques** [10, g1, g2]. Method [10] extends the static scenario by including time in the domain and explicitly modeling 3D motion as dense scene flow fields. For a given 3D point and time, the model predicts both reflectance and opacity and forward and backward 3D scene flow. In [g1], the model learns scene appearance, density, and motion jointly. NeRFlow internally represents scene appearance and density with a neural radiance field and scene dynamics with a flow field. Finally, [g2] focuses on predicting explicit 3D correspondence between neighboring frames to achieve information aggregation. The correspondence prediction is achieved through the estimation of consistent depth and scene flow information across frames with the use of dedicated networks. \n\nFollowing suggestions from another Reviewer (**hxyh**; W1), we plan to include **additional works that operate in dynamic scene reconstruction:** [b1, b2]. These are early Gaussian splatting works that significantly contributed to the dynamic scene reconstruction field, and we plan to include them in the revised version of the manuscript. These methods, however, require a multi-view setup similar to those already included [21].\n\n[g1] Yilu et al. 2021, “Neural radiance flow for 4D view synthesis and video processing,” in Proceedings of the IEEE/CVF ICCV\n\n[g2] Zhou et al. 2024, \"DynPoint: Dynamic Neural Point For View Synthesis,\" in NeurIPS 36\n\n***W3 “If your method is based on [3], a brief review should be included in the main text as a preliminary to improve the presentation.”***\n\nOur model is based upon the Gaussian parameterization framework outlined in [3], with a detailed elaboration provided in Appendix A. To enhance readability, this section will be moved to the main text.\n\n***W4 More details such as storage costs and training time (for each stage) should be included. Additionally, would the rendering speed be affected by the large deformation network?”***\n\nDue to the rebuttal length, ablation studies are included in the global rebuttal.\n\n**W6 “What is n_core as mentioned in Eq.1?”**\n\nWe have introduced the notation N_Core​ (Eq. 1) and N_Sub​ (Eq. 3) to clarify our references to Gaussian distributions for Core-Gaussians versus Sub-Gaussians.  \n\n***Q1  “More comparison and discussions with SC-GS should be included since SC-GS can also support editing and share similar ideas with control points/Gaussians.”***\n\nWe provide numerical comparisons with SC-GS in the main paper body (Tab 1, Tab 2, Appendix: Table 5 and Table 6 in our draft paper). Fig. 4 presents a visual comparison with the SC-GS model, highlighting the placement of key points in SC-GS. The SC-GS model lacks key points necessary for hand rotation, as demonstrated in our method (third image, second column). Additionally, overextending the SC-GS model results in \"tearing\" the human's arm, whereas our method handles scaling more effectively. Additionally, we will add more visual comparisons to illustrate our approach's robustness clearly.\n\n***Q2  “As I understand, deformation-based methods find it challenging to synthesize correct novel views with large motion. Could the authors provide comprehensive insight into this issue?“***\n\nModels based on deformable networks, including our model and SC-GS, create representations of key components (such as Core-Gaussians in our case and landmarks points in SC-GS) by positioning them optimally for ease of adjustment at each time instant. However, when there is significant movement in the training set, the network responsible for component localization may face challenges. Such a problem is the same in all models which use deformable networks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the Reviewers for their excellent comments and constructive remarks regarding our paper, as well as for their positive feedback. We are also thankful for noticing the main contribution of our model, which is “enabling the editing of more extreme large motions” that “matches the rendering quality of SC-GS”, as underlined by Reviewer 2. The main two strands of the received feedback concern the need for further ablation studies and providing a more comprehensive literature review. We feel that these changes will enhance our paper, and we will incorporate them into our manuscript.

**Ablation studies and hyperparameter analysis**

Most of the comments received concerned the need for an ablation study. First, we contrasted our model with and without the Sub-Rotation Network, as shown in the Table below, utilizing the commonly used *jumpingjacks* dataset. **These results show that Sub-Rot Network is crucial to obtaining SOTA PSNR. With Sub-Rot Network, we obtain approximately 0.5 PSNR crucial to obtain the reconstruction of small elements** (like human fingers in the *jumpingjacks* dataset).


|   | batch = 4  |batch = 4   |batch = 8   |  batch = 8 |
|---|---|---|---|---|
| Sub-Rot Network  | with  |without   | with  |  without |
|PSNR | 40.42 | 39.98 | 41.65 | 41.27|
|Training time [h] | 1:18|1:12|2:08|1:45|
|Rendering time [fps] |175 |186|190|259|

We also present how the size of the network influences training time and render speed in the next Table utilizing the commonly used *jumpingjacks* dataset. **These results show that deformable networks and Sub-Rot Networks are not costly in terms of rendering time, resulting in real-time rendering.** 
In practice, batch size has a higher impact than deformation network depth.

|Deformation network depth (numbers of layers)|4|6|8|10|
|---|---|---|---|---|
Batch:|4|4|4|4|
|PSNR|40.91|40.75|40.42|40.26|
|Training time [h]|1:21|1:22|1:18|1:31|
|Render time [fps]|138|167|175|144|
Batch:|8|8|8|8|
PSNR|41.86|42.01|41.65|41.37|
Training time[h]|2:00|1:58|2:08|2:07|
Render speed(fps)|227|221|190|192|

The following Table presents the training time and storage cost, as well as the FPS for rendering for each dataset. **These results suggest that our model is memory efficient and that the time required to train it is minimal.**  We performed the experiments for batches 4 and 8 to show the effect of batch size. In all cases, storage costs decreased, and performance improved with a trade-off of increased training time.


Dataset:| Hook|Jumpin|Trex|Bounce|Hell|Mutant|Standup|
|---|---|---|---|---|---|---|---|
Batch:|4|4|4|4|4|4|4|
PSNR|37.77|40.42|39.56|40.63|41.44|43.38|46.07
Time 1st stage|00:02:48|00:02:22|00:02:41|00:03:12|00:02:27|00:03:46|00:03:17
Time 2nd stage |1:40:15|1:16:18|2:14:09|1:43:08|1:01:27|1:48:49|1:19:32
Storage cost|76MB|53.5MB|122MB|131MB|27MB|73MB|43MB
Rendering time  (fps)|138|175|90| 123| 185 | 138 | 169
Batch:|8|8|8|8|8|8|8
|PSNR|38.07|41.65|40.74|40.55|41.59|44.40|47.22|
|Time 1st stage [h] |00:05:30|0:05:17|00:05:38|00:05:16|00:04:28|00:05:51|00:05:56|
|Time 2nd stage [h] |2:27:08|2:03:25|2:42:22|2:40:30|1:43:33|2:18:43|2:12:12|
|Storage cost|32MB|24MB|51MB|80MB|16MB|28MB|20MB|
|Rendering time(fps)|192|190|143|153|205|188|194

**Related Works**

Another repeated feedback concerned extending the body of referenced prior works. In line with this feedback, we will reorganize and extend the literature review to include the suggested related works and make it more complete. We are also keen to acknowledge that these additional papers present lower scores than SG-CS and D-Miso. Moreover, we reckon that our D-Miso model has a stronger ability to edit dynamic scenes.

Meta-review: This paper introduces a Dynamic Gaussian Splatting representation that enables easier object shape editing at test time. There is an overall consensus among reviewers that this is an interesting submission and should be accepted (2x Borderline Accept, 1x Weak Accept, and 1x Accept). Therefore, I recommend acceptance.

For the camera-ready version, please polish the paper to make it more reader-friendly and include the new experiments and discussions.
Decision: Accept (poster)