Dynamic Gaussians Mesh: Consistent Mesh Reconstruction from Dynamic Scenes
Authors: Isabella Liu
Hao Su
Xiaolong Wang
Venue: ICLR 2025 Poster (2025)
Keywords: dynamic scene reconstruction
4D reconstruction
4D gaussian splatting
video reconstruction
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: DG-Mesh reconstructs high-fidelity, time-consistent meshes from dynamic observations and supports tracking and editing of dynamic mesh sequences.

Abstract: Modern 3D engines and graphics pipelines require mesh as a memory-efficient representation, which allows efficient rendering, geometry processing, texture editing, and many other downstream operations. However, it is still highly difficult to obtain high-quality mesh in terms of detailed structure and time consistency from dynamic observations. To this end, we introduce Dynamic Gaussians Mesh (DG-Mesh), a framework to reconstruct a high-fidelity and time-consistent mesh from dynamic input. Our work leverages the recent advancement in 3D Gaussian Splatting to construct the mesh sequence with temporal consistency from dynamic observations. Building on top of this representation, DG-Mesh recovers high-quality meshes from the Gaussian points and can track the mesh vertices over time, which enables applications such as texture editing on dynamic objects. We introduce the Gaussian-Mesh Anchoring, which encourages evenly distributed Gaussians, resulting better mesh reconstruction through mesh-guided densification and pruning on the deformed Gaussians. By applying cycle-consistent deformation between the canonical and the deformed space, we can project the anchored Gaussian back to the canonical space and optimize Gaussians across all time frames. During the evaluation on different datasets, DG-Mesh provides significantly better mesh reconstruction and rendering than baselines.

Published as a conference paper at ICLR 2025
DYNAMIC GAUSSIANS MESH: CONSISTENT MESH RE- CONSTRUCTION FROM DYNAMIC SCENES
Isabella Liu, Hao Su', Xiaolong Wang' UC San Diego, Equal advising
Figure I: We propose DG-Mesh, a framework reconstructs high-fidelity time-consistent mesh for dynamic scenes with complex non-rigid deformations. Given dynamic input and the camera parameters, our method reconstructs the high-quality surface and its appcarance, as well as the mesh vertice motion across time frames. Our method reconstruct mesh with fiexible topology change as shown above. Additional results be found on: https: //wp:///wwww////// sisuussslllle la. com/ DG-Mesh.
ABSTRACT
Modern 3D engines and graphics pipelines require mesh memory-effiiint representation, which allows efficient rendering, geometry processing, texture editing, and many other downstream operations. However, is still highly diffi- cult to obtain high-quality mesh in terms of detailed structure and time consistency from dynamic observations. To this end, we introduce Dynamic Gaussians Mesh (DG-Mesh), framework to reconstruct high-fidelity and time-consistent mesh from dynamic input. Our work leverages the recent advancement 3D Gaus- sian Splatting to construct the mesh sequence with temporal consistency from dynamic observations. Building on top of this representation, DG-Mesh recov- ers high-quality meshes from the Gaussian points and can track the mesh vertices over time, which enables applications such as texture editing on dynamic objects. We introduce the Gaussian-Mest Anchoring, which encourages evenly distributed Gaussians, resulting better mesh reconstruction through mesh -eesh- densifica- tion and pruning the deformed Gaussians. By applying cycle-consistent de- formation between the canonical and the deformed space, we can project the an- chored Gaussian back the canonical space and optimize Gaussians across all time frames. During the evaluation on different datasets, DG-Mesh provides sig- nificantly better mesh reconstruction and rendering than baselines.
Codes and data are publicly available ht //ggt//ttttt com/ sabel 1a98li www.iu//// h/tgu//oGG-MM---

---

Published as a conference paper at ICLR 2025
INTRODUCTION
The birds fly, the butterflies flutter, and the flowers sway with the breeze from the wind our natural world is dynamic, and objects present in the human eyes as an entanglement of structure, motion, and color. A pivotal goal computer vision is to empower machines with the human-like ability to recover object geometry and motion from visual cues in the environment.
The advent of neural rendering techniques has sparked a surge in research focused on extracting the geometry and motion of dynamic scenes from videos using neural representations. Most stud- ies have concentrated on learning deformable neural radiance fields using volumetric representa- tions (Mildenhall et al., 2021). However, these volumetric models often fall short in terms of memory efficiency and explicit geometric details. The recent introduction of 3D Gaussian Splat- ting (Kerbl et al., 2023) shifts towards point cloud representations that not only are more memory- efficient but also can provide explicit geometries and superior rendering quality. Further advance- ments demonstrate how 3D Gaussian Splatting's 's explicit geometry can effectively represent dynamic scenes by tracking each point's 's movement across frames (Luiten et al., 2023; Wu et al., 2023a; Yang al., 2023d).
This paper takes a significant step in the realm of explicit geometry and motion estimation for dy- namic scenes. We introduce method extract high-fidelity meshes and their motions by tracking vertices over time from videos, as illustrated in Figure 3. Meshes, in contrast to volume, offer a more memory-efficient format. The mapping of correspondences on mesh vertices greatly simpli- fies texture editing and propagation on target objects. Crucially, the dynamic mesh extracted via our method can be seamlessly integrated into a physical simulator, enabling rapid, physics-based rendering adaptable to diverse materials and lighting conditions.
Our method, Dynamic Gaussians Mesh (DG-Mesh), performs a joint optimization procedure of both 3D Gaussians and the corresponding meshes. Our method not only allows the mesh have flexi- ble topology changes but also builds the correspondence across meshes over time. Specifically, our method constructs set of deformable 3D Gaussians by optimizing the 3D Gaussians in canonical space and learning the deformation module for transforming the 3D Gaussians in different time steps for rendering. For each time step, we transform the deformed Gaussians into mesh in differen- tiable manner using a combination of Poisson solver and marching cube algorithm, which is then rendered with differentiable rasterizer for training. Notably, this fully differentiable pipeline allows the change of mesh topology while maintaining cross-fram consistency implicitly due to the intro- duction of the canonical space. But how do we find the correspondences given these independent meshes?
We achieve this by decomposing the mesh-to-mesh cor- respondence into two separate correspondences that are easier to acquirr: the tresh-to-points correspondence that maps the mesh faces to the Gaussian points in each time frame; and the points-to-canen cotatssto-ccnonncl--piinn
correspondence that helps tracking the movement of mesh across time. Our key observation is that the direct training of 3D Gaus- sians mentioned above will lead to uneven spreading of Gaussians in 3D space (as shown in Figure left). How- ever, this detrimental for mesh reconstruction perfor- mance due to the violation of the Possion solver assump- tion that points should be evenly distributed; worse still, non-uniform Gaussians make it trickier to track across a long time period. Thus, we propose the Gaussian- Anchoring procedure during training, which yields uniformly distributed Gaus- sians at each frame. Concretely, first, we encourage deformed 3D Gaussians in each time step to align with the corresponding mesh surface that has uniform face distribution due to the underly- ing iso-surface algorithm; and second, we allow adding Gaussians in the world frame at each time step if certain mesh faces are not covered by existing deformed Gaussians, as well as deduplicting Gaussians for mesh faces that are covered multiple times. We call the new set of Gaussians with 1-1 correspondences to mesh faces as anchored Gaussians. As depicted in Figure 2, anchored Gaussians spread more uniformly in 3D.
Ref, image
Wio anchor
W/ anchor
Figure 2: The 3D Gaussian centers before and after the Gaussian-Mesh Ancharing.
Finally, we need to update the canonical space to accommodate the added and removed Gaussians in the world frame that canonical and anchored Gaussians are always consistent. This can be done
2

---

Published as a conference paper at ICLR 2025
by introducing a backward deformation network that computes the deformation from the anchored Gaussians to the canonical Gaussians, by which we can inject new Gaussians in the canonical space or remove unnecessary Gaussians. We define Cycle-Consistent Deformation loss inspired by (Yang et al., 2023b), where the cycle is formed with: a forward deformation from the canonical Gaussians to deformed Gaussians, anchoring process modifying the deformed Gaussians to the anchored Gaussians, backward deformation from anchored Gaussians to canonical Gaussians. As a result, we obtain uniformly distributed 3D Gaussians in each time step, and they are aligned with the mesh faces. This makes finding the correspondences between meshes much easier.
We demonstrate high-fideling mesh reconstruction and tracking in our experiments. A few examples can be found in Figure I, where the correspondence is shown as the red curves connecting the ver- tices across time. Even for challenging geometries (e.g., thin structures like bird wings), our method can still perform reconstruction and achieve much better results than previous approaches, which usually fail to reconstruct the geometry or construct an oversized mesh. We also demonstrate down- stream applications, such texture editing with our extracted mesh and correspondence (Figure 26), which is much harder to achieve with non-mesh representations. To the best of our knowledge, this is the first framework that recovers high-fidelity meshes with cross- fross-frame correspondences from dynamic observations.
2 RELATED WORK
View Synthesis in a Dynamic Scene. The rise of Neural Radiance Fields (NeRFs) (Mildenhall et al., 2021) has largely transformed 3D scene reconstruction and novel view sythesis. Its success has also been extended to dynamic scenes in two directions. One direction to build dynamic NeRFs (Du et al., 2021; Gao et al., 2021; Li et al., 2021; Xian et al., 2021) which models the motion of the scene by extending the radiance field with an extra time dimension or a latent code. To achieve faster rendering speed and more efficient use of memory, recent works perform volume factorization converting the 4D volume into multiple lower dimension planers or tensors (Cao & Johnson, 2023; Fridovich- Keil et al., 2023; Shao et al., 2023; Xu et al., 2023). Another direction is to construct an additional deformation field that maps point coordinates in different time frames into a canonical space, where large motion and geometry changes can be captured and learned (Park et al., 2021a; Pumarola et al., 2021; Park et al., 2021b; Tretschk et al., 2021; Fang et al., 2022). The recently proposed 3D Gaussians Splatting (Kerbl et al., 2023) extends the volumetric rendering in NeRF by accommodating point clouds. This not only largely improves the speed of neural rendering but also extracts an explicit point cloud structure from images. When applied to dynamic scenes, the same idea of building a deformation field is applied with 3D Gaussians (Yang et al., 2023d; Wu et al., 2023a; Luiten et al., 2023; Li et al., 2024; Yang et al., 2023c). But instead of an implicit field, the deformation field here explicitly tracks the canonical point clouds over time. Inspired by this line of research, we push forward the reconstruction of explicit geometry representation by using meshes with the tracks of the mesh vertices over time from dynamic observations.
Dynamic Mesh Reconstruction. Mesh plays an important role in modern 3D engines, which are widely used for simulation, modeling, and rendering applications. Recovering mesh from static scenes has been extensively studied over the years. Mcsh template-based methods (Hanocka et al., 2020; Xue et al., 2023; Pan et al., 2019; Wang et al., 2018; Kanazawa et al., 2018; Li et al., 2020) optimize the vertex positions from a template to align with the object surface. They usually assume the template is sphere or given prior shape, and learn to deform the template into the desired shape to match with the input images. However, when extended to dynamic scenes, these methods often fail because of the topology change during the deformation. Another line of work extract meshed from the implicit field, which can easily handle topology change during deformation. In particular, the mesh can be extracted by identifying and triangulating the zero level sets with methods like Marching Cubes (Cubes, 1987), Marching Tetrahedra (Treece et al., 1999), and Dual Contouring (Ju al., 2002). Recently, differentiable mesh extraction methods have been proposed (Liao et al., 2018; Chen et al., 2022; Remelli et al., 2020; Shen et al., 2021; 2023; Wei et al., 2023), directly optimizing mesh through implicit filed. Besides static scenes, several works have been studying extracting the surface geometry of deformable objects from dynamic monocular inputs (Yang al., 2023a; 2022; Wu et al., 2023b; Yang et al., 2021a;b; Tulsiani et al., 2020; Johnson et al., 2023), which more closely related to real-world scenarios. Due to the limited information provided by monocular inputs, these methods either rely on category template information or require strong regularization data modalities to output satisfying results.
3

---

Published as a conference paper at ICLR 2025
y(0
Gradient flow
Forward flow
Anchored Gaussians
Fa
Surtace
Backward Deform
Nydiffrast
+/-
Mesh Image
SIM Points
Canonical
1:
Gaussians
DPSR Dift. M.C.
Fr
Gaussian Rasterization
Deformed
Forward
Gaussians
Gaussian Image
Deform
Figure 3: Main pipeline of DG-Mesh. We maintain a set of canonical 3D Gaussians. Under each time step, we transform it into a deformed space. We treat each set of deformed Gaussian points and apply differentiable Poisson Solver and differentiable Marching Cubes recover the deformed surface. propose Gaussian-Ment Auchoring to adjust deformed Gaussians to be uniformly aligned with the mesh faces. During anchoring, Gaussian densification and pruning are performed. We use a backward deformation module to project the newly adjusted Gaussian points back to the canonical space.
oriented point cloud
METHOD
In this section, we present our framework shown Figure 3. Given inputs from a dynamic scene, as well as the time label and camera parameters for each input, reconstruct the dynamic mesh and its appearance, along with the mesh vertex's ' motion trajectory. The surface correspondence across times benefits many downstream applications, such as dynamic texture editing, which propagates the editing under a single frame to the rest of the frames.
Our method maintains Gaussians into different times (Section 3.1). We use a differentiable Poison Solver and differen- tiable Marching Cubes method reconstruct the oriented 3D Gaussian points into meshes (Sec- tion 3.2). Our model allows the mesh-to-mesh correspondence. This is constructed by two types of correspondence, including the mesh-to-point correspondence under each time step and the point- to-canonical
of canonical 3D Gaussians and deformation networks deform the
-oint point correspondence across all time frames. We propose the Gaussian-Mesh An- choring to encourage the 1-1 correspondence between the mesh faces and the Gaussians, producing more uniformly distributed anchored Gaussians. The Cycle-Consisted points to canonical points correspondence, which includes a backward deformation network that computes the deformation from the anchored Gaussians to the canonical Gaussians (Section 3.3). Finally, we combine all the training objectives (Section 3.4).
Deformation maintains
3.1 DEFORMABLE 3D GAUSSIAN SPLATTING
3D Gaussian Splatting, Recently, 3D Gaussians Splatting (3DGS) (Kerbl et al., 2023) has adopted novel approach based on point cloud rendering, achieving optimal results in novel viewpoint syn- thesis and scene modeling. In our work, 3DGS provides a fast modeling to explicit geometry and motion, allowing a high-quality and efficient surface reconstruction from the 3D Gaussian point cloud.
We define each 3D Gaussian G by a position (mean) μ,, a full 3D covariance matrix Σ centered at point (mean) μ., and the opacity o:
G(m, E) oc exp - - 11)
During the a-blending process, each Gaussian multiplied by a. To project Gaussians onto image space for rendering, we construct covariance matrix E' through Σ' JW JWWWTTT "WTT" fT where the Jacobian of the affine approximation of the projective transformation and W viewing transformation (world-to-camera transformation matrix). To ensure positive semi-definite matrix during the optimization process, is decomposed into a scaling matrix S and a rotation matrix R: Σ RSSTT RT. In implementation, directly optimize 3D vector for scaling and a quaternion r to represent rotation during the training process. Each 3D Gaussian can now be represented G(æ;; (,,,8,))
4

---

Published as a conference paper at ICLR 2025
Figure 4: Illustration of our Gaussian- Mesh Anchoring procedure. After obtaining mesh from the DPSR and differentiable Marching Cubes, we adjust the deformed 3D Gaussians to make it more aligned with the mesh's faces. For each Gaussian point, we find its nearest neighhors. If mesh face is the nearest neighbor multiple Gaussian points at the same time, we merge these Gaussians and create a new Gaussian from them. a mesh face is not any Gaussian point's nearest neighbor, we create a new Gaussian on its center.
Deformable 3D Gaussians. 3DGS assumes static scenes and its performance drops drastically when reconstructing dynamic scenes. Several works (Wu et al., 2023a; Yang et al., 2023d; Jung al., 2023; Lin al., 2023; Zielonka al., 2023; Li et al., 2024; Yang et al., 2023c) have inves- tigated the way to extend the 3DGS to dynamic reconstruction and an efficient way is to maintain canonical 3D Gaussains set. We define transformation function F which transforms the 3D Gaussian (z (4, T, a) from one space to another by:
F(>(2), (()) (ōær, 8r, 5ax)
where y is the positional encoding function to map x and to higher dimensional Fourier features (Tancik et al., 2020) to enhance detailed reconstruction:
7 (p) - (sin(2 πр), cos(2 mp). .... sin(2" Tp), (2s(2"TT)) mp))
The deformed 3D Gaussian in the new space can be represented by:
G' + dz;; pr + ôr, + 8s, + 5(x)
3.2 MESH RECONSTRUCTION
Surface Extraction and Rendering Pipeline. To apply the isosurface algorithms (such March- ing Cubes (Lorensen & Cline, 1998) or DMTet (Shen et al., 2021)) and extract the surface geometry from the point-cloud representation of the 3DGS, indicator function x(x) is required over the 3D grid to describe the geometry. Some work (Tang et al., 2023) query a density grid using 3D Gaussian's opacity, which does not satisfy the gradient condition in the non-boundary space and is computationally inefficient. Instead, we treat 3DGS as an oriented point-cloud and perform a differ- entiable Poisson solver (DPSR) (Peng al., 2021) to obtain the (x). We then apply differentiable Marching Cubes (Wei et al., 2023) to generate watertight manifold meshes from the obtained value grid. We additionally optimize vertex color for the mesh and perform differentiable rasterization (Laine et al., 2020) to render the final image.
Laplacian Regularization. To better preserve the mesh surface tessellation, we adopt a mesh Lapla- cian regularization term to penalize local curvature changes. We use uniformly weighted differ- ential 8, of vertex v; to describe the difference between the position of vertex v, and the average position of its neighbors. The Laplacian regularization term the mean square 6; for all vertices in the mesh and n is the total number of vertices of the mesh.
1 Σ
Diop
15.11 0,
N
KEN:
where N; one-ring neighbor set of vertex Uz.
3.3 DYNAMIC MESH CORRESPONDENCE
We construct our mesh-to-mest correspondence with two separate correspondences: the mesh-to- points correspondence which maps the mesh faces to the Gaussian points in each time frame, and the point-to point-to-cann canonical-point correspondence across all time frames that helps tracking the movement of mesh time.
5

---

Published as a conference paper at ICLR 2025
Hex-Plane
ToNeuVox
DG-Mesh (Ours)
GT
Surf. Render
Mesh
Surf. Render
Mesh
Surf. Render
Mesh
T
72
T
2
Figure 5: We compare the mesh reconstruction and rendering results of method with other baselines the D-NeRF datasct. For cach object, we visualize the results under different time steps and diffcrent viewing angles. Our method oulperforms others by producing more smooth and detailed structures.
Gaussian- Anchoring. To build the mesh-to-point correspondence at each time frame, we introduce the Gaussian- Mesh Anchoring. The original 3D Gaussian densification and deformation mechanism leads uneven spreading of Gaussians in 3D space (Figure 2), which is undesirable for finding the 1-1 correspondence between mesh and the Gaussians. On the other hand, DPSR dis- cretizes the function values and differential operators in the Poisson equation by assuming the nor- mal vector field and indicator function X uniformly sampled along each dimension. However, this uniformity not guaranteed during the above training procedure. To encourage the uniformity of the 3D Gaussian distribution, we reconstructed mesh to guide the densification and pruning of the 3D Gaussian in the deformed space. Specifically, given a deformed 3D Gaussian set (G(x;)) } and the set of the face centroids from its reconstructed mesh {fy 1 and N + M. For each 3D Gaussian point T; we find its nearest neighbor in the mesh face
f==0
Ray = arg min || - fll. E 16
During the densification and pruning process, shown Figure for face fy that the near- est neighbor to multiple K Gaussian points, we prune the original K Gaussians and create new Gaussian by averaging the properties from the K Gaussians:
K
G' = K
I G(xx)
K=O
On the other hand, for face fy that is not the nearest neighbor to any Gaussian point, we create new Gaussian at the face centroid:
G' = G(},)
Besides the above Gaussian density control, also penalize the distance between Gaussian points and face centroid that has one-to-one correspondence by:
Lanchor
|x nzg 113
T
where n is the total number of faces that have a one-to-one correspondence to the 3D Gaussian points. A detailed description of the algorithm can be found in Appendix A. The obtained anchored Gaussians G' are applied in the Cycle-Consisee deformation loss Leyete which will be introduced in the next paragraph.

---

Published as a conference paper at ICLR 2025
Duck
Bird
Method
Corresp.
CD&
EMD v. PSNR
PSNRm. T
CD
EMD 4. PSNR
PSNRR
CD
EMD
PSNR
PSNR
D-NeRF
X
0,934
0.073
29.785
23.019
1.685
0.280
25.474
17.381
1.532
0.163
23,848
19,573
K-Plane
X
1.085
0.055
33.360
20.372
1.480
0.239
28.111
21.629
0.742
0.131
23.722
19.559
HexxPlae
X
2.161
0.090
32,108
27.945
1.750
0.199
26.779
22.395
4.158
0.178
22.
20.595
X
0.969
0.059
22.073
1,918
0.246
18.156
0.215
19.844
4D-GS
0.272
0.187
23.834
DeformableGGS
2.366
0.115
34.187
1,510
0.217
30,280
1.358
0.141
25,095
4DGS
5.895
069
28,008
0.895
0.340
31.138
20.426
0.277
24.568
SC-G8
X
1.306
0.097
40.825
0.897
0.177
38.402
0.897
0.106
32,575
DG-Mesh
0.782
0.047
28.120
32.757
0.297
0.164
23.437
30.865
0.510
0.125
24.362
28.085
Bengle
Tonus2sphere
Girlwalk
Method
Comesp.
CD
EMD
PSNR 1
PSNRm. f
CD
EMD 1. PSNR 7 PSNK T
CDL
EMD
1 PSNR 7 PSNR T
D-NeRF
x 1.001
0.149
34.470
24.446
1.760
0.250
24.227
13.562
0.601
0.190
28.632
21.146
K-Plane
X
0.810
0.122
38,329
24.613
1.793
0.161
31.215
15.706
0.495
0.173
32.116
23.008
HexPlane
X
0.115
29.970
0.190
0.155
31.771
24.214
ToxeuVox-B
0.874
0.129
38,972
25.773
2.115
0.203
28.756
14,985
0.568
0.184
32,806
20.207
4D-G8
X
0.644
106
42,995
2.188
0.261
28.329
0.596
0.315
33.430
Deformable-GS
X
1.154
0.161
42,530
2,210
0.248
28:274
1.103
0.183
34.157
4DGS
X
6.121
0.198
35.036
1.523
0.226
28.533
1.815
0.237
35.588
SC-G8
3.359
0.147
2.045
0.225
32.946
0,203
42.615
DG-Mesh
33.572
0.177
0.151
25.847
Table 1: Mesh reconstruction results of method compared with other baselines. We mcasure the recon- structed mesh's: s: Chamfer Distant (CD) and Earth Mover Distance (EMD) with the ground truth mesh. The unit for CD is 10-2 for Torus2sphere and 10 for the rest of the scenes. We also measure the volume ren- dering quality PSNR of the baselines as well as the PSNR,, of their mesh rendering results. We use and indicate the best, the second best and the third results. Overall, our method produces higher quality meshes among all metrics.
Cycle-Consse Cont--oonsittet Deformation. Deformable 3D Gaussians provide explicit motion and correspon- dence across times because of their point-cloud representation. During Gaussian- Anchoring, we align the reconstructed mesh faces and the 3D Gaussian points in each time frame and perform mesh-guided densification and pruning. To update the canonical space to accommodate the anchored correspondence,
Gaussians and build the toattto-cccccalapppeeee tency between the canonical and deformed spaces to ensure that the anchoring performed under the deformed space
encourage the cycle consis-
be applied back to the canonical space. We define forward transformation function Fy and a backward transformation function Fb. Given a canonical 3D Gaussian G and its new form G' under deformed space, we have:
Fy( ((G,1)=-Focc
We use the L1 loss between Fy and FF deformation allows the anchored Gaussian to be deformed back to the canonical space. Specifically, G' will be the anchored Gaussians after the Gaussian-Mesh Anchoring, as introduced previously. This coce-coons constraint will encourage adjustments to the canonical Gaussians. This pro- cess is visualized on the left side of Figure 3, where the green network is the forward deformation and the blue network is the backward deformation.
cycle consistent loss Leyele This cycle-consistent
3.4 COMBINING ALL TRAINING OBJECTIVES
We use rendering loss from both mesh rasterization ("Mesh Image" in Figure 3) and the 3DGS ("Gaussian Image" in Figure 3) to optimize the mesh geometry and appearance. Given the ground truth image Let and the Gaussian splatted image 1ge- The image loss term of 3DGS can be described as:
Lys - >ssim) 1|
Iye Assim Lssim (Tgs+ Ige)
Same for the mesh image loss Cmesh. We also compute the L1 loss on the rasterized mask Lmask-
The Laplacian loss term Liep in Section 3.2 helps preserve the mesh surface tessellation and pro- duces smoother surface. The anchoring loss term Lanchor and the cycle-consistency loss term Leyele described in Section 3.3 help the optimization of tross-frame mesh-to-mes to-mesh correspondence. The final loss term can be described as:
=Lgs + Lmmss + Lrnask
+ Liap + Lanchor + Leyele
7

---

Published as a conference paper at ICLR 2025
GT
D-NeRF
HexPlane
VoNeuVox
Deformable-GS
DG-Mesh (Ours)
Figure 6: We present the mesh reconstruction results of our method and other baselines on the DG-Mesh dataset. Our method delivers better mesh quality and geometry. For objects like the flying bird, other methods struggle recover the thin wings, while ours successfully reconstructs them.
EXPERIMENTS
We introduce the dataset we evaluate and our implementation details in Section 4.1 and Sec- tion 4.2. In Section 4.3, we evaluate our method compared with other baselines. Regarding the baselines, we primarily compare with the following works: D-NeRF (Pumarola et al., 2021), K- Plane (Fridovich- Keil al., 2023), HexPlane (Cao & Johnson, 2023), TiNeuVox (Fang et al., 2022), 4D-GS (Wu et al., 2023a), Deformable-GS (Yang et al., 2023d), 4DGS (Yang et al., 2023c), and SC-GS (Huang et al., 2024). Among these baselines, D-NeRF (Pumarola et al., 2021) and TiNeu- Vox (Fang et al., 2022) learn a deformation field to transform the dynamic scene into the canonical space, while other methods like HexPlane (Cao & Johnson, 2023) and K-Plane (Fridovich-Keil al., 2023) utilize volume factorization to learn a compact 4D feature volume. 4D-GS (Wu et al., 2023a), Deformable-GS (Yang et al., 2023d) 4DGS (Yang et al., 2023c), and SC-GS (Huang et al., 2024) use 3D Gaussians as their representations and model their dynamics over time. Note that most of these baselines do not model scene geometry explicitly. To compare the mesh quality, we extract mesh from either an implicit density field or an occupancy field queried from the 3D Gaussians, In Section 4.4, we studied several regularization factors in our method and provided quantitative results of how each factor affects the final performance.
4.1 DATASETS
We evaluate our method on the D-NeRF synthetic dataset and provide the visualization and com- parison with other baselines in Figure 5. Since the D-NeRF (Pumarola al., 2021) dataset does not provide mesh ground truth information, we rendered a synthetic dataset containing six dynamic scenes to compare the mesh reconstruction quality quantitatively. Each scene includes 200 frames of a moving object with the ground truth camera parameters and images, as well as the ground truth mesh under each time frame. Camera views are evenly distributed around the target objects on the sphere or the upper sphere. The reconstruction visualization is provided in Figure 6, and full evaluation visualizations can be found in the Appendix
For real data evaluation, we run our method on the Nerfies dataset (Park et al., 2021a). the Dycheck's dataset (Gao et al., 2022) and the Unbiased4D dataset (Johnson et al., 2023), all of which contains monocular videos of everyday deformable objects captured using handheld cameras. To demonstrate the adaptability of DG-Mesh to different dynamic capturing setups, we also test our method on the NeuralActor (Liu et al., 2021) dataset, which features multi-view videos of moving humans. Additionally, we captured several dynamic scenes using a smartphone.
4.2 IMPLEMENTATION DETAILS
We utilize the implementation from 3D Gaussian Splatting (Kerbl et al., 2023) for differential Gaus- sian rasterization. Our model was trained for a total of 50,000 iterations on single RTX 3090Ti. To better initialize deformable 3D Gaussians, we first trained the canonical Gaussians for 3k itcrations while keeping the forward and backward deformation network fixed. This helped to retain rela- tively stable positions and shapes of 3D Gaussians under the canonical space. After 5k iterations, we introduce the DPSR and differentiable Marching Cubes to extract the mesh geometry from the Gaussian points. We perform Gaussian-Mess Anchoring every 100 iteration during training. For the mesh rasterization, we use Nydiffrast. In order to query the vertex color of the deformed mesh we project the location of the deformed vertex back to the canonical space and query dependenn appearance module to obtain the color. We found optimizing the vertex color in canonical space produces better results compared with querying color under different deformed spaces. The final
8

---

Published as a conference paper at ICLR 2025
Lego
Bouncingbulls
Jumpingjacks
Hook
Method
PSNR....
SSIM
LPIPS
PSNR... 1 SSIM f LPIPS
PSNR...
S
LPIPS
PSNR...
SSIM
LPIPS
D-NeRF
20.384
0,818
0.137
23.398
0.899
0.157
22.255
0.914
0.103
20,300
0.889
0.108
K-Plane
19.523
0.828
0.127
23.307
0,935
0.109
25.240
0.937
0.068
22.503
0.900
0.094
HcxPlane
22.872
0.904
0.072
25.389
0.957
0.069
27.078
0.954
0.052
24.513
0.929
0.070
Tox-e
21.927
0.843
0.126
24.819
0.947
23.621
0.932
0,075
21.429
0.908
0.085
DG-Mesh
21.289
0.838
0.159
29.145
0.969
31,769
0.045
27.884
0.954
0.074
Mutant
Standup
Trex
Hellwarrior
Method
PSNR 1 SSIM
LPIPS
PSNR..
f SSIM 1 LPIPS
PSNR...
1
SSM
LPIPS 5
PSNR... f SSIM
LPIPS
D.NeRF
21.070
0.906
0.077
23.380
0.925
0.069
22.594
0.908
0,085
18.907
0.877
0.129
K- K-Plune
23,226
0.923
0.064
25.778
0.946
0.048
23.093
0.921
0.075
18.073
0.881
0.123
HexPlane
26.811
0.953
0.045
27.931
0.965
0.035
26.629
0.953
0.046
21,250
0.917
0.094
MiNeuVox-b
22,967
0.925
0.064
24,263
0,941
0.051
24.219
0.927
0.070
18.657
0.883
0.118
DG-Mesh
30.400
0.968
0.055
30,208
0.974
0.051
28.951
0.959
0.065
25.460
0.959
0.084
Table 2: We measure the mesh rendering PSNR, SSIM and LPIPS score of the baselines and our method on the D-NeRF datasct. Our method has the highest surface rendering quality compared with all the other baselines.
supervision comes from the rendering loss from both the splatted Gaussian images and the rendered mesh images. More implementation details can be found in Appendix B.
4.3 RESULTS AND COMPARISONS
For the D-NeRF dataset, we present our numerical results and comparisons in Table 2 and mesh visualization in Figure 5. We compare the mesh rendering PSNR, SSIM, and LPIPS scores for each object in the dataset. Since the D-NeRF dataset does not provide ground truth mesh, we do not numerically evaluate our performance of the mesh quality. In Figure 5, we highlight the part where our method outperforms the other baselines and recovers the highly detailed structures. For the DG-Mesh dataset with mesh ground truth, we provide the numerical results and comparisons in Table and visualizations in Figure 6. Regarding evaluation metrics of mesh quality, we chose two metries to measure our mesh reconstruction's 's accuracy: the Chamfer Distance (CD) and the Earth Mover Distance (EMD) to measure the displacement between the reconstructed mesh and the ground truth. As shown in Table I, our method achieves lower CD and EMD scores compared with other baselines, indicating our method reconstructs the highest quality mesh. In Figure 6, we show the visualization of our reconstructed mesh compared to other baselines in our dataset. In challeng- ing areas, such as thin structures like the wings of the bird, our method still recovers high-quality geometry and surfaces, while other methods produce floaters or oversized meshes. For real evalu- ation, provide the quantitative results the Nerfies dataset in Table 5 and qualitative results in Appendix G. Our method achieves the highest mesh query speed while delivering rendering quality comparable to the baseline. Performance other monocular datasets is detailed in the supplemen- tary materials, including the Unbiased4D dataset in Appendix J, the Dycheck dataset in Appendix H, and the self-captured iPhone dataset in Appendix I. Additionally, we present qualitative evaluations on the NeuralActor dataset in Appendix K. demonstrating the broader applicability of DG-Mesh to multiview setups.
4.4 ABLATION STUDY
Laplacian Smoothness Regularizer. We study the effect of laplacian regularizer and its different performance under different loss weights in this part. Laplacian regularization term calculates the average square Fuclidean difference between the vertex and its one-ring neighbors, which discour- ages deformation that will introduce large surface curvature change. We measure the final mesh ren- dering quality (PSRN, SSIM, LPIPS) under different laplacian loss weights. As shown in Table 3, when Whap = 1000, the network produces the highest mesh rendering quality. With appropriate laplacian regularization, method recovers smoother surface.
Gaussian-Mesh Anchoring. We study the effect of our proposed Gaussian- MMehh Anchoring proce- dure. Since the original densification and pruning mechanism of 3DGS tends to split more Gaussian in hard areas where the geometry and appearance are more complicated. The original Gaussian center distribution is not even on the object's 's surface. As discussed in Section 3.3, DPSR assumes uniformly sampled points among the space. An uneven point cloud on the surface will produce the wrong distance field and harm the optimization process. Our Gaussian- Meshh Anchoring performs densification and pruning that will result in more evenly distributed Gaussian point and rely on

---

Published as a conference paper at ICLR 2025
CD
EMD 4
PSNR f
SSIM
LPIPS
Wlap
0.972
0.089
31.947
0.971
0.079
Wtop 100
1.239
0.093
32.178
0,969
0.069
Wtep 1000
0.762
0.073
33,890
0.982
0.061
Wlap 2000
0.894
0.069
32.891
0.981
0.072
Table 3: Results of mesh reconstruction and rendering quality with different Laplacian regularizer ratios. When Wlpp 1000 our method produces the best results. During the training process, we set the Laplacian ratio be 1000.
GT
DeformableGGS
DG-Mesh (Ours)
provide the reconstructed mesh visualization and mesh rendering image our method on the Nerfies real-world datasets. Our method recovers high-quality and consistent mesh surfaces across multiple time frames, showcasing its robustness and effectiveness in handling dynamic scenes.
Figure 7:
the backward deformation network to project it back to the canonical space. As shown in Figure 2, after applying Gaussian-Mesh Anchoring. the Gaussian centers are more evenly distributed on the object's surface. We provide quantitative results of how the Gaussian- Anchoring procedure allows the network to produce better geometry in Table 4. Without Gaussian MMiih Anchoring, the reconstructed CD and EMD are both higher.
Method
CD
EMD
PSNR... 1
Method
PSNR PSNR,.. 1 FPS,.. 1
w./o. anchor
0.685
0.146
31.283
Deformable-Gr DG-Mesh
28.310
0.10
w// anchor
0.516
0.128
32.734
27.238
10.9
Table 4: Ablation study the Gaussian- An- choring. The anchoring allows mesh to be better aligned with the ground truth surface.
Table 5: Real scene evaluation the Nerfies dataset. Our method maintains high mesh querying FPS and comparable image rendering quality to the baseline.
5 CONCLUSION
We introduce Dynamic Gaussians Mesh (DG-Mesh), a framework to reconstruct high-fidelity mesh and perform motion tracking from dynamic observations. Leveraging the recent advancements in 3D Gaussian Splatting, DG-Mesh learns a deformable 3D Gaussians set and builds a novel time- consistent mesh on top of it. DG-Mesh outperforms previous methods in reconstructing intricate structures like bird wings and horse legs, which are typically difficult to capture. Moreover, DG- Mesh's 's ability provide cross-frame correspondences enables various downstream applications, such as texture editing on dynamic objects, making it versatile tool with great potential in graphics applications. We will release our code upon paper publication.
Limitations. DG-Mesh is currently applied mainly on the foreground objects in video. To make DG-Mesh work with real-world videos, we require segmentation the foreground object across the videos. While DG-Mesh handles both the correspondence and topology changes, obtaining corre- spondence becomes fundamentally challenging in the presence of significant topology changes. For instance, if an additional object object part appears in the video over time, establishing correspon- dence for the new object (or object part) with the first frame becomes difficult. This challenge stems from the nature of the problem itself, rather than limitation of our method.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported, in part, by the NSF CAREER Awards (IIS-2240014, IIS-2241600) and by the Qualcomm Embodied Al Fund.
REFERENCES
Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 130-141, 2023.
3, 8, 27, 32
Zhiqin Chen, Andrea Tagliasacchi, Thomas Funkhouser, and Hao Zhang. Neural dual contouring. ACM Transactions on Graphics (TOG), 41(4):1-13, 2022.
Ho Kei Cheng, Seoung Wug Oh, Brian Price, Alexander Schwing, and Joon-Young Lee. Tracking anything with decoupled video segmentation. In Proceedings of the IEEF/CVE International
Conference on Computer Vision, pp. 1316-1326, 2023. 20, 23
Marching Cubes. A high resolution 3d surface construction algorithm/william e. Lorensen, Harvey E. Cline-SIC 87:76, 1987. 3
Yilun Du, Yinan Zhang, Hong-Xing Yu, Joshua B Tenenbaum, and Jiajun Wu. Neural radiance flow for 4d view synthesis and video processing. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 14304-14314, IEEE Computer Society, 2021. 3
Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießnner, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIGGRAPH Asia 2022 Conference Pupers, pp. 1-9, 2022. 3, 27, 32
Sara Fridovich-Kan Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12479-12488 2023. 3, 8, 22
Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocular video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5712-5721, 2021.
Hang Gao, Ruilong Li, Shubham Tulsiani, Bryan Russell, and Angjoo Kanazawa. Monocular dy- namic view synthesis: A reality check. Advances in Neural Information Processing Systems, 35; 33768-33780, 2022. 20, 21, 23
Rana Hanocka, Gal Metzer, Raja Giryes, and Daniel Cohen-Or. Point2mesh: A self-prior for de- formable meshes. arXiv preprint arXiv:2005.11084 2020. 3
Yi-Hua Huang, Yang-Tian Sun, Ziyi Yang. Xiaoyang Lyu, Yan-Pei Cao, and Xiaojuan Qi. Sc- gs: Sparse-controlled gaussian splatting for editable dynamic scenes. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4220-4230, 2024.
Erik Johnson, Mare Habermann, Soshi Shimada, Vladislav Golyanik, and Christian Theobalt. Un- biased 4d: Monocular 4d reconstruction with neural deformation model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6597-6606, 2023. 3, 8, 24
Tao Ju, Frank Losasso, Scott Schaefer, and Joe Warren. Dual contouring of hermite data. In Pro- ceedings the 29th annual conference Computer graphics and interactive techniques, pp- 339-346, 2002.
HyunJun Jung, Nikolas Brasch, Jifei Song, Eduardo Perez-Pellitero, Yiren Zhou, Zhihao Li, Nassir Navab, and Benjamin Busam. Deformable 3d gaussian splatting for animatable human avatars. arXiv preprint arXiv:2312. 2023.

---

Published as a conference paper at ICLR 2025
Angjoo Kanazawa, Shubham Tulsiani, Alexei A Efros, and Jitendra Malik. Learning category- specific mesh reconstruction from image collections. In Proceedings of the European Conference on Computer Viston (ECCV), pp. 371-386, 2018.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Transactions on Graphics (ToG), 42(4))1-14, 2023. 2, 3, 4, 8
Samuli Laine, Janne Hellsten, Tero Karras, Ycongho Scol, Jaakko Lehtinen, and Timo Aila. Modular primitives for nigh-performance differentiable rendering. ACM Transactions on Graphics (TOG), 39, 2020. 5, 17
Xueting Li, Sifei Liu, Shalini De Mello, Kihwan Kim, Xiaolong Wang, Ming-Hsuan Yang, and Jan Kautz. Online adaptation for consistent mesh reconstruction in the wild. Advances in Neural Information Processing Systems, 33:15009-15000 2020. 3
Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. Spacetime gaussian feature splatting for real-time dynamic view synthesis. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8508-8520, 2024. 3,5
Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space- time view synthesis of dynamic scenes. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 6498-6508, 2021. 3
Yiyi Liao, Simon Donne, and Andreas Geiger. Deep marching cubes: Learning explicit surface representations. In Proceedings of the IEEE Conference Computer Vision and Pattern Recog- nition, pp. 2916-2925, 2018." 3
Youtian Lin, Zuozhuo Dai, Siyu Zhu, and Yao Yao. Gaussian-flow: 4d reconstruction with dynamic 3d gaussian particle. arXiv preprint arXiv:2312.03431, 2023. 5
Lingjie Liu, Mare Habermann, Viktor Rudnev, Kripasindhu Sarkar, Jiatao Gu, and Christian Theobalt. Neural actor: Neural free-view synthesis of human actors with pose control. ACM transactions on graphics (TOG), 40(6):1-16, 2021. 8, 26
William Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. In Seminal graphics: pioneering efforts that shaped the field, pp. 347-353. 1998. 5, 17
Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. arXiv preprint X3Xx:23308.097333 2023. 2,
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1)9991106, 2021. 3,3
Junyi Pan, Xiaoguang Han, Weikai Chen, Jiapeng Tang, and Kui Jia. Deep mesh reconstruction from single rgb images via topology modification networks. in 2019 ieee. In CVF International Conference on Computer Vision (ICCV), volume 3, 2019. 3
Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan В Goldman, Steven M Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5865-5874, 2021a. 3, 8, 20
Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, and Steven M Seitz. Hypemerf: A higher-dimensionall representation for topologically varying neural radiance fields. arXiv preprint arXiv:2106.1 13228, 2021b.
Ricardo
Songyou Peng, Chiyu Jiang, Yiyi Liao, Michael Niemeyer, Mare Pollefeys, and Andreas Geiger. Shape as points: A differentiable poisson solver. Advances in Neural Information Processing Systems, 34:13032-11044 2021. 5, 17
12

---

Published as a conference paper at ICLR 2025
Albert Pumarola, Enric Corona, Gerard Pons-Moll and Francese Moreno-Noguer D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10318-10327, 2021. 3, 27,
Edoardo Remelli, Artem Lukoianov. Stephan Richter, Benoit Guillard, Timur Bagautdinov, Pierre Baque, and Pascal Fua. Meshsdf: Differentiable iso-surface extraction. Advances in Neural Information Processing Systems, 33:22468 2222288222288, 2020. 3
Ruizhi Shao, Zerong Zheng, Hanzhang Tu, Boning Liu, Hongwen Zhang, and Yebin Liu. Ten- sor4d: Efficient neural 4d decomposition for high-fidelity dynamic reconstruction and rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 16632-16642 2023. 3
Tianchang Shen, Jun Gao, Kangxue Yin, Ming- Yu Liu, and Sanja Fidler. Deep marching tetrahedra: hybrid representation for high-resolution 3d shape synthesis. Advances in Neural Information Processing Systems, 34:6087-6100 6101, 2021. 3,5
Tianchang Shen, Jacob Munkberg, Jon Hasselgren, Kangxue Yin, Zian Wang, Wenzheng Chen, Zan Gojeie, Sanja Fidler, Nicholas Sharp, and Jun Gao. Flexible isosurface extraction for gradient- based mesh optimization. ACM Transactions on Graphics (TOG), 42(4):1-16, 2023.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich- Keil, Nithin Raghavan, Utkarsh Singhal. Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances Neural Information Process- ing Systems, 33:7537-7547, 2020. 5
Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309.1665: 2023. 5
Graham M Treece, Richard W Prager, and Andrew H Gee. Regularised marching tetrahedra: im- proved iso-surface extraction. Computers & Graphics, 23(4):583-555 1999.
Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pp. 12959-12979 2021.
Shubham Tulsiani, Nilesh Kulkarni, and Abhinav Gupta. Implicit mesh reconstruction from unan- notated image collections. arXiv preprint arXiv:2007.08500 2020. 3
Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In Proceedings of the European conference on computer vision (ECCV), pp. 52-67, 2018.
Xinyue Wei, Fanbo Xiang, Sai Bi, Anpei Chen, Kalyan Sunkavalli, Zexiang Xu, and Hao Su. Neu- manifold: Neural watertight manifold reconstruction with efficient and high-quality rendering support. arXiv preprint arXiv:2305. 17134, 2023. 5.5
Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint arXiv:2310.08528, 2023a. 2, 5,8
Shangzhe Wu, Tomas Jakab, Christian Rupprecht, and Andrea Vedaldi. Dove: Learning deformable 3d objects by watching videos. International Journal of Computer Vision, pp. 1-12, 2023b.
Wenqi Xian, Jia Bin Huang, Johannes Kopf, and Changil Kim. Space-time neural irradiance fields for free-viewpoint video. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 9421-9431, 2021.
Zhen Xu, Sida Peng. Haotong Lin, Guangzhao He, Jiaming Sun, Yujun Shen, Hujun Bao, and Xiaowei Zhou. 4k4d: Real-time 4d view synthesis at 4k resolution, 2023. 3
13

---

Published as a conference paper at ICLR 2025
Yuxuan Xue, Bharat Lal Bhatnagar, Riccardo Marin, Nikolaos Sarafianos, Yuanlu Xu, Gerard Pons- Moll, and Tony Tung. Nsf: Neural surface fields for human modeling from monocular depth. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 15049-15060, 2023. 3
Gengshan Yang, Deqing Sun, Varun Jampani, Daniel Vlasic, Forrester Cole, Huiwen Chang, Deva Ramanan, William Freeman, and Ce Liu. Lasr: Learning articulated shape reconstruction from monocular video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15980-15989, 2021a.
Gengshan Yang, Deqing Sun, Varun Jampani, Daniel Vlasic, Forrester Cole, Ce Liu, and Deva Ramanan. Viser: Video-specific surface embeddings for articulated 3d shape reconstruction.
Advances in Neural Information Processing Systems, 34: 14:11926-19933 2021b. 3
Gengshan Yang, Minh Vo, Natalia Neverova, Deva Ramanan, Andrea Vedaldi, and Hanbyul Joo. Banmo: Building animatable 3d neural models from many casual videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2863-2873, 2022.
Gengshan Yang, Shuo Yang, John Z. Zhang. Zachary Manchester, and Deva Ramanan. Ppr: Physi- cally plausible reconstruction from monocular videos. In Proceedings of the IEEE/CVF Interna- lional Conference on Computer Vision, pp. 3914-3924, 2023a. 3
Kaizhi Yang, Xiaoshuai Zhang, Zhiao Huang, Xuejin Chen, Zexiang Xu, and Hao Su. Movingparts: Motion-based 3d part discovery in dynamic radiance field. arXiv preprint arXiv:2303.05700, 2023b.
Zeyu Yang, Hongye Yang, Zijie Pan, and Li Zhang. Real-time photorealistic dynamic scene repre- sentation and rendering with 4d gaussian splatting. arXiv preprint arXiv:2310. 10642, 2023c. 3, 5,8
Ziyi Yang. Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. De- formable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. arXiv preprint arXiv:2309. 2023d. 2, 2, 21
Wojciech Zielonka, Timur Bagautdinov, Shunsuke Saito, Michael Zollhöfer, Justus Thies, and Javier Romero. Drivable 3d gaussian avatars, 2023.
14

---

Published as a conference paper at ICLR 2025
APPENDIX TABLE OF CONTENTS
Gaussian-Mesh Anchoring Details
16
Implementation Details and Network Architecture
17
C Ablations Study on the Template-based Deformation Method
18
D Ablations Study on Fine-tuning the Extracted Mesh
18
Ablation Study on the Appearance Module
18
F Ablation Study the Anchoring Frequency
19
G More Qualitative Results on the Nerfies Dataset
20
More Qualitative Results on the Dycheck Dataset
1 More Qualitative Results on the Self-Captured iPhone Dataset
23
J More Qualitative Results on the Unbiased4D Dataset
24
More Qualitative Results on the NeuralActor Dataset
26
More Qualitative Results on the D-NeRF Dataset
27
M More Qualitative Results on the DG-Mesh Dataset
32
Application Examples
36
15

---

Published as a conference paper at ICLR 2025
A GAUSSIAN-MESH ANCHORING DETAILS
We employ the Gaussian-Meet Anchoring procedure to guide the Gaussians into a more uniform distribution across the object's surface. This uniformity enhances the smoothness of the mesh recon- struction obtained from DPSR and DiftMC. A more detailed algorithm is provided in Algorithm
The algorithm begins by finding the nearest mesh face for each Gaussian point. For every mesh face, the following steps are taken:
If a mesh face is linked to multiple Gaussian points, we merge these Gaussians to create new, single Gaussian. This merging helps in reducing redundancy and promotes uniform distribution.
If a mesh face is linked to only one specific Gaussian, we adjust the position of that Gaussian by moving towards the centroid of the face. This adjustment ensures that the Gaussian is more accurately anchored to the mesh surface.
If mesh face not linked to any Gaussian point, we create new Gaussian at the face's centroid. This step fills in any gaps in the Gaussian distribution, ensuring that all areas of the mesh are adequately represented.
By following this procedure, we ensure that the Gaussians are evenly distributed over the mesh surface, which crucial for achieving high-quality mesh reconstructions. The anchoring process systematically addresses the association between Gaussians and mesh faces, leading to more ac- curate and smooth representation of the object's geometry.
Algorithm 1: Gaussian- Messh Anchoring
Input : 9 set of Gaussians in canonical space
V mesh vertices
F mesh faces
Fy (-, forward deformation function
Fb(,,t) t) backward deformation function
1 timestep
search radius
Output:
Lanchor anchoring loss
g updated set of Gaussians in canonical space
Giet 4- (garr = (g, g):
Lanchor <- 0:
foreach E F do
Cf < centroid
1/ Compate centroid for curzent face
G { ggee cyll S ra Bder E Gaer)
// Find Gaussians within radius ra of ez
if g then
Snew f merge(G));
gcanonical ggow t)
/1 Map the merged Caussian
canonical space // Renove the original Gaussians
(Fo
Edef E 9/}
Beanonical
// the Gassstan
Lanchur + Lanchue Beamonical
cyll
else
G then if
geanonical f- Fg 1),
gier E G5
Lanchor f- Lanchor Geanonical Cf
x GU@ (geanonical)
else
gnew Cf
// Create Gauasian centroid // Add the new Gausalan
Geanonical f- Fo (Bocw
end
end
end
16

---

Published as a conference paper at ICLR 2025
B IMPLEMENTATION DETAILS AND NETWORK ARCHITECTURE
The model was trained over 50,000 iterations using a single NVIDIA RTX 3090Ti GPU, which handled the computational demands effectively. To achieve more effective initialization of the deformable 3D Gaussians, we began by training the canonical Gaussians for 3,000 iterations. During this initial phase, the forward and backward deformation networks were kept fixed and not updated. This approach helped in maintaining relatively stable positions and shapes of the 3D Gaussians within the canonical space, providing a good mitiillizaiion for subsequent deformation learning.
After reaching 5,000 iterations, we introduced the Differentiable Poisson Surface Reconstruction (DPSR) (Peng et al., 2021) and differentiable Marching Cubes algorithms (Lorensen & Cline, 1998) to extract mesh geometry from the Gaussian points. This integration allowed us to transition smoothly from point-based representations to mesh-based geometry. Throughout the training pro- cess, performed Gaussian-Mesh Anchoring every 100 iterations to ensure consistency between the Gaussian and mesh representations.
For mesh rasterization, employed Nvdiffrast (Laine et al., 2020), GPU-accelerated differen- tiable renderer. To determine the vertex colors the deformed mesh, we projected the positions of the deformed vertices back into the canonical space and used time-dependent appearance mod- ule to retrieve the color information. We discovered that optimizing the vertex colors within the canonical space yielded better results compared querying colors in various deformed spaces. This is likely due to the more stable and consistent representation in the canonical space. The final su- pervision signal for model came from the rendering losses calculated from both the splatted Gaussian images and the rendered mesh images, allowing the model to learn from both point-based and mesh-based representations.
Input
Output
Linear
ReLU
Figure 8: Results comparison the DG-Mesh dataset. We show the reconstructed mesh and the mesh render- ing image. Our method reconstructs better geometry and appearance than other baselines.
Regarding the architecture of the deformation networks, both the forward and backward deformation networks are designed as multi-layer perceptrons (MLPs) with a depth of = 8 layers and hidden layers of dimensionality W = 256. The detailed architecture is illustrated in Figure 8. We apply positional encoding functions 7* (p) to map the spatial position x and the temporal label into nigher-dimension feature spaces. Specifically,
set the encoding parameter A = 10 for the positions x the 3D Gaussians and = for the time labels t, enhancing the network's ability capture high-frequency variations in both space and time. After the final hidden layer, a single linear layer without any activation function to produce the predicted offsets for Gaussian position (8x), scaling (ds), rotation (ör), and opacity (ôa). These offsett adjust the Gaussians' parameters based on the input, allowing for dynamic deformation. The appearance network, which responsible for determining the color of the vertices, shares a similar architectural design with the deformation network.
17

---

Published as a conference paper at ICLR 2025
C ABLATIONS STUDY ON THE TEMPLATE-BASED DEFORMATION METHOD
DG-Mesh reconstructs the surface a mesh across different timesteps by first deforming a set of canonical 3D Gaussians into deformed space. The surface is then recovered by leveraging the deformed Gaussian points and their corresponding normals. This method allows the model to manage flexible topology changes during deformation, distinguishing it from other approaches that maintain a canonical mesh and learn to deform it over time. We evaluated our method in an extreme scenario where the dynamic scene involves sphere transforming into a torus. In this case, the genus of the object shifts from 0 to 1. To compare, we ran a baseline experiment in which network attempted to deform a canonical mesh of a sphere into torus. The results in Figure 9 indicate that traditional mesh deformation based methods struggle to handle topology changes, whereas our approach effectively overcomes this limitation by using Gaussian points for representation, thereby avoiding the need to manage face connectivity during deformation.
11
t2
4
22
Deform canonical mesh
Ours
Figure 9: We evaluated our method in extreme scenario where a sphere deforms into torus. The traditional mesh deformation- based approach fails in this case because it cannot handle the changes face connectivity during the deformation process. In contrast, our method successfully reconstructs a high-quality mesh with a completely different topology. This is because we use 3D Gaussian points as our canonical representation, which avoids any issues related to managing mesh face connectivity throughout the deformation.
D ABLATIONS STUDY ON FINE-TUNING THE EXTRACTED MESH
Volume rendering V.S. mesh rendering PSNR. Previous baselines primarily rely on volume ren- dering, which offers advantages in view synthesis but comes with significant computational cost. This is because volume rendering integrates color from multiple points along a ray to com- pute the pixel color, whereas mesh rendering retrieves color from only single surface point. We compare the inference speed of volume rendering baselines and our mesh rendering in Table 6. Additionally, higher-quality results from volume rendering do not necessarily translate into better geometry or surface reconstruction, as highlighted by the mesh reconstruction comparisons in our paper. To ensure a fairer comparison rendering PSNR, we provide the PSNR of mesh rendering after muninuning the baselines with a mesh rendering loss over 20,000 iterations. Even after this fine-tuning, our method still outperforms the baselines terms of mesh rendering PSNR.
Methods K-Plane
Fine-tuned PSNR,, m
Training Time
Inference Speed
24.683
65 min
0.9 FPS
TiNeu Vox
24.149
~ 70 min
0.3 FPS
Deformable-GS
22.519
35 min
185 FPS
Ours
29.782
80 min
10.9 FPS
Table 6: We present the fine-tuned mesh rendering results for the baselines, alongside a comparison of training and inference times. Even after fine-tuning the mesh rendering of other baselinc methods, our approach still delivers the best results. This highlights the efficiency and effectiveness of our method, both in terms of mesh quality and overall performance.
E ABLATION STUDY ON THE APPEARANCE MODULE
We use the geometry generated by DG-Mesh and render the mesh using appearance queried from other baseline methods. The results below demonstrate that, even when the underlying geometry
18

---

Published as a conference paper at ICLR 2025
quality is comparable, DG-Mesh consistently achieves better visual quality compared other ap- proaches. The surface rendering approach DG-Mesh enables it to learn appearance directly and precisely the object's surface. In contrast, other volume rendering sradering--bsedd based methods fail to store appearance details accurately at the surface, leading to less precise results.
Methods
PSNR
SSIM
D-NeRF
26.169
0.905
HexPlane
29.013
0.964
TiNeu
28.157
0.943
DG-Mesh (Ours)
29.790
0.959
Table Comparison rendering results shows DG-Mesh achieves superior visual quality by accurately learn- ing surface appearance, unlike volume- based methods that lose surface detail.
F ABLATION STUDY ON THE ANCHORING FREQUENCY
We evaluate the impact of different anchor intervals and provide their quantitative results below. The mesh geometry quality deteriorates as the anchor intervals increase since larger intervals result in fewer anchoring steps, reducing the alignment between the Gaussians and the surface. Based on the ablation results, we the anchoring interval 100, as it represents balanced trade-off between maintaining geometry quality and achieving optimal rendering performance.
Anchor Interval
PSNR, m
CD
EMD
50
30.521
0.314
1.307
100
30.668
0.306
1.298
200
30.599
0.319
1.295
500
30.645
0.320
1.298
Table 8: Quantitative results show that increasing anchor intervals deteriorates mesh geometry quality due to fewer anchoring steps, with an interval of 100 providing the best balance hetween geometry quality and rendering performance. The unit for CD is 10 and the unit for EMD is 10
19

---

Published as a conference paper at ICLR 2025
G MORE QUALITATIVE RESULTS ON THE NERFIES DATASET
We evaluate our method on the Nerfies (Park et al., 2021a) dataset, which consists of monocular video sequences capturing everyday moving objects, such as pets. The camera poses throughout the sequences are predominantly forward-facing. To obtain the foreground mask, we utilize an off- the-shelf segmentation tool from (Cheng et al., 2023), and for the camera parameters, we follow the pipeline outlined in (Gao et al., 2022). In Figure 10, we display our mesh reconstruction and rendering results three different timesteps. Our method consistently recovers high-quality and coherent mesh surfaces over time, demonstrating its effectiveness in dynamic scenes.
GT
Mesh Rendering
Mesh
GT
Mesh Rendering
Mesh
Figure 10: Mesh reconstruction and rendering results on the Nerfics dataset. Our method recovers high-quality and consistent mesh surfaces across multiple time frames, showcasing its robustness and effectiveness in han- dling dynamic scenes.
20

---

Published as a conference paper at ICLR 2025
H MORE QUALITATIVE RESULTS ON THE DYCHECK DATASET
We evaluate our method on the DyCheck dataset (Gao et al., 2022), which consists monocu- lar videos captured using smartphones, and compare its mesh reconstruction and rendering quality against Deformable-GS (Yang et al., 2023d). The results show that our method achieves better mesh geometry reconstruction, particularly in scenarios with limited observations from monocular videos. However, we acknowledge that, similar to other dynamic Gaussian based methods, our approach faces challenges when dealing with extreme object motion or imperfect segmentation.
Deformable-GS
DG-Mesh (Ours)
Input
Novel View Rendering
Novel View Mesh
Deformable-CS
DG-Mesh (Ours)
Input
Novel View Rendering
Novel View Mesh
Figure 1l: Comparison of mesh reconstruction results with Deformable-GS on the DyCheck dataset. DG-Mesh demonstrates better reconstruction of both mesh geometry and appearance.
21

---

Published as a conference paper at ICLR 2025
Deformable-GS
DG-Mesh (Ours)
Input
Novel View Rendering
Novel View Mesh
Deformable-C5
DG-Mesh (Ours)
Input
Novel View Rendering Novel View Mesh
Figure 12: Fail Cases the DyCheck Dataset: Similar to other dynamic Gaussian-based methods, our ap- proach encounters limitations in scenarios with extreme object motion or imperfect segmentation.
22

---

Published as a conference paper at ICLR 2025
I MORE QUALITATIVE RESULTS ON THE SELF-CAPTURED IPHONE DATASET
We captured several casual, everyday videos of moving objects using smartphone and processed them using similar data processing pipeline the one proposed (Gao al., 2022). To obtain the data masks, we used an off-the-shelf segmentation tool from (Cheng et al., 2023). The self- captured videos have frame object over 8-10 seconds. In Figure 13, we present the mesh reconstruction and rendering results. Our method demonstrates the ability recover high-quality mesh surfaces from casual smartphone videos, highlighting its practicality and robustness for real-world applications.
of 60 FPS, with the camera rotating 360 degrees around the
GT
Mesh Rendering Mesh
GT
Mesh Rendering
Mesh
Figure 13: present the mesh reconstruction and rendering results on self-captured iPhone dataset. Our method successfully recovers high-quality and consistent mesh surfaces over multiple time frames, demon- strating its robustness and efficiency in processing real-world dynamic scenes captured in casual smartphone videos.
23

---

Published as a conference paper at ICLR 2025
J MORE QUALITATIVE RESULTS ON THE UNBIASED4D DATASET
e evaluate our method on real video sequence from the Unbiased4D dataset (Johnson et al., 2023) and compare with the results produced by the method proposed in their paper. The video captures a fast-moving cactus toy, making surface recovery particularly challenging due to the high-speed motion. As shown in Figure 14, our method achieves superior geometry recovery, especially in the hand area of the toy during rapid movement, whereas the Unbiased4D method produces blurrier results and incorrect geometry. Additionally, we provide our mesh reconstruction and rendering results for the sequence at different timeframes in Figure 15, showcasing the robustness of our approach in handling fast-moving objects.
FRISS
GT
Unbiased4D
Ours
GT
Unbiascd4D
Ours
Figure 14: Comparison of the mesh results with the Unbiased4D method. Our approach achieves outstanding geometry recovery, particularly in the hand area of the toy during rapid movement, while the Unbiased4D method results in blurrier outputs and incorrect geometry.
24

---

Published as a conference paper at ICLR 2025
t
MATT
AST
MET
PRESS
PRESS
PRESS
HC
AA
AC
NET
PRESS
PRESS
RESP
RESP
REST
2
WA
NUT
PRESS
BE
BE
BCC
CARIN
PRESS AC
PRESS
PRESS
H
A
GT
Mesh Rendering
Mesh
Figure 15: Mesh reconstruction and rendering results on the Unbiased4D dataset. The results indicate the robustness of our approach in handling fost-n objects.
25

---

Published as a conference paper at ICLR 2025
K MORE QUALITATIVE RESULTS ON THE NEURALACTOR DATASET
We evaluate our method on the NeuralActor dataset (Liu et al., 2021), a multi-view dynamic dataset featuring videos of moving humans captured from multiple angles. In Figure 16, we present the reconstructed meshes and corresponding mesh rendering results at different time frames. The vi- sualizations demonstrate that, with the benefit of multi-view observations, our method achieves even better surface reconstruction compared to the monocular setup, highlighting its effectiveness in leveraging additional views for more accurate and detailed mesh recovery.
Mesh Rendering
Mesh
Mesh Rendering
Figure 16: Mesh reconstruction and rendering results on the NeuralActor dataset. The result highlights the effectiveness of our approach in utilizing multiple viewpoints to produce more accurate and detailed mesh recovery in dynamic scenes.
26

---

Published as a conference paper at ICLR 2025
L MORE QUALITATIVE RESULTS ON THE D-NERF DATASET
We present the mesh reconstruction results of DG-Mesh across all eight samples from the D-NeRF dataset. For each sample, we provide visualizations of the reconstructed mesh along with the mesh surface rendering results at two different time frames, 60 and f1. We compare our visual outcomes with those from several baseline methods, including D-NeRF (Pumarola et al., 2021), HexPlane (Cao & Johnson, 2023), K-Plane (Fridovich-Ke) Keil et al., 2023), and TiNeuVox (Fang et al., 2022). The results clearly demonstrate that our method achieves superior mesh reconstruction and rendering quality compared to these baselines.
GT (img)
GT (mesh)
DoNeRF (img)
D.NeRF (mesh)
HexPlane (img)
HexPlane (mesh)
N/A.
Ours (Img)
Ours (mesh)
K-Plane (img)
K-Plane (mesh)
TiNeuVox (img)
TiNenVox (mesh)
X
GT (img)
GT (mesh)
D-NeRF fimg)
D-NeRF (mesh)
HexPlane (img)
HexPlane (mesh)
NJA.
Ours (img)
Ours (mesh)
K-Plane (img)
K-Plane (mesh)
TINeu (img)
TENeuVox (mesh)
Figure Results comparison on D-NeRF dataset. We show reconstructed mesh and mesh rendering image. Our method reconstructs better geometry and appearance than other baselines.
27

---

Published as a conference paper at ICLR 2025
GT (img)
GT (mesh)
D-NeRF (img)
D-NeRF (mesh)
HexPlanc (img)
HoxPlane (mesh)
N.A.
Ours (img)
(mesh)
K-Plane (img)
K-Plane (mesh)
(img)
TiNeuVox (mesh)
GT (img)
GT (mesh)
D-NeRF (img)
D-NeRF (mesh)
HexPlane (img)
HexPlane (mesh)
a
N.A.
Ours (img)
Ours (mesh))
K-Plane (img)
K-Plane (mesh)
Tox (img)
TiNeuVox (mesh)
GT (img)
GT (mesh)
D-NeRF (img)
D-NeRF (mesh)
HexPlane (img)
HexPlane (mesh)
0
N/A.
Ours (img)
Ours (mesh)
K-Plane (img)
K-Plane (mesh)
TiNeuVox (img)
TiNeuVox (mesh)
GT (img)
GT (mesh)
D-NeRF (img)
D-NeRF (mesh)
HexPlane (img)
HexPlane (mesh)
NJA.
(img)
(ingg)
(meshh
(img)
TiNeuVox (meshh
Figure 18: Results comparison the D-NeRF dataset. We show the reconstructed mesh and the mesh rendering image. method reconstructs better geometry appearance than other baselines.
28

---

Published as a conference paper at ICLR 2025
GT (img)
GT (mesh)
D-NeRF (img)
D-NeRF (mesh))
HoxPlane (img)
HexPlanc (mesh)
N/A.
Ours (img)
Ours (mesh)
K. Plane (img)
K. Plane (mesh)
TiNeu Vox (img)
TiNen Vox (mesh)
GT (img)
GT (mesh)
D. D.NeRF (ing)
D. NeRR (mesh)
HexPPlaee (img)
Ilane (mesh)
Ours (img)
Ours (mesh)
K-Plane (img)
K-Plane (mesh)
ToNeuVox (img)
TiNeu Vox (mesh))
GT (img)
GT (mesh)
D.NeRF (ing)
D.NeRF (mesh)
HexPlane (img)
HexPlane (mesh)
N/A.
Ours (img)
Ours (mesh)
K-Plane (ing)
K-Plane (mesh)
TiNeu (img)
TiNeu Vox (mesh)
(img)
(mesh)
D-NeRE (img)
D-NeRE
HexPlane (img)
HexPlane (mesh)
d
N/A.
Ours (img)
Ours (mesh)
K-Plane
K-Plane (mesh)
TiNeu Vox (img)
TiNeu Vox (mesh)
Figure Results comparison D-NeRF dataset. show reconstructed mesh and mesh rendering image. Our method reconstructs better geometry and appearance than other baselines.
29

---

Published as a conference paper at ICLR 2025
GT (img)
GT (mesh)
D-NeRF (img)
D.NeRF (mesh)
HexPlane (img)
HexPlane (mesh)
m
N/A.
Ours (ing)
Ours (mesh)
K-Plane (img)
K-Plane (mesh)
ToNeu (img)
Tox (mesh)
(img)
(mesh)
D-NeRF (img)
D-NeRF (mesh)
(img)
HexPlane (mesh)
d
N/A.
(ing)
K-Plane (img)
(meshh
(ingg)
TiNeu (messh)
GT (img)
GT (mesh)
D-NeRF (img)
D-NeRF (mesh)
HexPlane (img)
HexPlane (mesh)
N/A.
Ours (ing)
Ours (mesh)
K Plane (img)
K. Plane (mesh)
'iiNeuVox (img)
TiNeu (mnssh)
GT (ing)
GT (inesh)
D NeRR (ing)
D NeRF (mesh)
HeaeP (img)
HexPPlane (mesh)
N/A.
Ours (ing)
Ours (mesh)
Plala (ing)
K. Plane (mesh)
TiNeuVox (ing)
TiNeuVox (meeh)
Figure 20: Results comparison on the D-NeRF dataset. We show the reconstructed mesh and the mesh rendering image. Our method reconstructs better geometry and appearance than other baselines.
30

---

Published as a conference paper at ICLR 2025
GT (ing)
GT (mesh)
D-NeRF (img)
D-NeRF (mess)
HexPlane
HexPlane (mesh)
N.A.
Ours (img)
Ours (mesh)
K-Plane (img)
K-Plane (mesh)
ToxeuVox (img)
TiNeuVox (mesh)
GT (img)
GT (mesh)
D-NeRF (img)
D.NeRR (mesh)
HexxPlne (img)
HexPlune (mesh)
NJA.
Ours (Img)
Ours (mesh)
K-Plane (img)
K-Plane (mesh)
TiNeuVox (img)
TiNeuVox (mesh)
Figure 21: Results comparison the D-NeRF dataset. We show the reconstructed mesh and the mesh rendering image. Our method reconstructs better geometry and appearance than other baselines.
31

---

Published as a conference paper at ICLR 2025
M MORE QUALITATIVE RESULTS ON THE DG-MESH DATASET
We present the mesh reconstruction results of DG-Mesh across all six samples in the DG-Mesh dataset. Each sample consists 200 frames moving object, with ground truth camera param- eters, corresponding images, and ground truth mesh for each time frame. The camera views are uniformly distributed around the target objects, either on full sphere or an upper hemisphere. For each sample, we provide visualizations of the reconstructed mesh along with mesh surface render- ing results at different time frames, t0 and t1. We compare these visual results against several baselines, including D-NeRF (Pumarola et al., 2021), HexPlane (Cao & Johnson, 2023), K-Plane (Fridovich-Ki) Keil al., 2023), and TiNeu Vox (Fang et al., 2022). The results demonstrate that our method achieves superior mesh reconstruction and rendering quality. Notably, DG-Mesh excels in reconstructing high-quality mesh surfaces, even in challenging regions such as a bird's wings or horse's s tail, where fine details and complex motions involved.
GT (img)
GT (mesh)
D-NeRF (img)
D-NeRF (mesh)
HexPlane (img)
HexPlane (mesh)
Ours (img)
Ours (mesh)
K-Plane (img)
K-Plane (mesh)
TiNeu (img)
Vox (mesh)
(img)
(mesh)
D-NeRF (img)
D-NeRF (mesh)
HexPlanc (img)
HcxPlane (mesh)
Ours (img)
(mesh)
K-Plane (img)
K-Plane (mesh)
TiNeu (img)
TiNeu (mesh)
Figure 22: Results comparison the DG-Mesh dataset. We show the reconstructed mesh and the mesh rendering image. Our method reconstructs better geometry appearance other baselines.
32

---

Published as a conference paper at ICLR 2025
GT (img)
GT (mesh)
D-NeRF (img)
D-NeRP (mesh)
HexPlane (img)
HexPlane (mesh)
(img)
(mesh)
(img)
(mesh)
(img)
(mesh)
GT (img)
GT (mesh)
D-NeRF
D.NeRF (mesh)
HexPlane (img)
HexPlane (mesh)
Ours (img)
Ours (meshh
K- PlPaaee (img)
Ki K-Plane (mesh)
TiNeuVox (img)
TiNeuVox (mesh)
GT (img)
GT (mesh)
D-NeRF (img)
D-NeRF (mesh)
HcxPlanc (img)
HexPlane (mesh)
Ours (img)
Ours (mesh)
K-Plane (img)
K-Plane (mesh)
TiNeuVox (img)
TiNeuVox (mesh)
(img)
GT (mesh)
D-NeRF (img)
D-NeRF (mesh)
HexPlane (img)
HexPlane (mesh)
Ours (img)
Ours (mesh)
K-Plane (img)
K.Plane (mesh)
TiNewVox (img)
TiNeuVox (mesh)
Figure 23: Results comparison the DG-Mesh dataset. We show the reconstructed mesh and the mesh rendering image. Our method reconstructs better geometry and appearance other baselines.
33

---

Published as a conference paper at ICLR 2025
GT (img)
GT (mesh)
D-NeRF (img)
D-NeRE (mesh)
HexPlanc (img)
HexPlane (mesh)
Ours (img)
Ours (mesh)
K-Plane (img)
K-Plane (mesh)
Vox (img)
VoNeuVox (mesh)
GT (img)
GT (mesh)
D-NeRF (img)
D-NeRF (mesh)
HexPlane (img)
HexPlane (mesh)
Ours (img)
Ours (meshh
K-Plane (img)
K-Plane (mesh)
Vox (img)
Tox (mesh)
GT (img)
GT (mesh)
D-NeRF (img)
D-NeRI (mesh)
HexPlane (ing)
HexPlane (mesh)
10
(img)
(mesh)
(img)
(mesh)
TiNeuVox (img)
TiNeuVox (mesh)
(img)
(mesh)
D.NeRR (img)
D.NeRF (mesh)
HexPlane (img)
HexPlane (mesh)
l
Ours (img)
Ours (mesh)
PlPlaee (img)
K-Plane (mesh)
TiNenNox (img)
TiNenVox (mesh)
Figure 24:
Results comparison on the DG-Mesh dataset. We show the reconstructed mesh and the mesh rendering image. Our methad reconstructs better geometry and appearance than other baselines.
34

---

Published as a conference paper at ICLR 2025
GT (img)
GT (mesh)
D-NeRF (img)
D-NeRF (mesh)
HexPlane (img)
HexPlane (mesh)
(img)
(img)
(mesh)
ToxeuVox (img)
TiNeu (mesh)
(img)
D-NeRF (img)
D-NeRF (meal)
HoxPlane (img)
HexPlane
Ours (ing)
Ours (mesh)
K-Plane (img)
K-Plane (mesh)
TiNeuVox (img)
TiNeu Vox (mesh)
Figure 25: Results comparison the DG-Mesh dataset. We show the reconstructed mesh and the mesh rendering image. Our method reconstructs better geometry appearance other baselines.
35

---

Published as a conference paper at ICLR 2025
N APPLICATION EXAMPLES
Mesh remains the predominant representation supported by many physics simulators and rendering engines, making it versatile tool for various applications. DG-Mesh provides dense correspon- dence, which proves especially useful for downstream tasks like shape manipulation and texture editing.
We demonstrate two examples of mesh editing enabled by our method, as illustrated in Figure 26. One application involves inserting the extracted mesh into new scene, where ray-tracing can be per- formed. Another application is dynamic texture editing. Using the time-consistent mesh in the first frame, we can edit the vertex colors by painting them directly onto the surface. This modification is then propagated consistently across subsequent frames, maintaining the same color and pattern for the corresponding vertices. The entire editing process is intuitive and user-friendly, making accessible even for complex tasks.
2024
202 4
Figure 26: Two types of application can be done with time-consistent mesh: 1. Ray-tracing in the rendering engine. 2. Texture editing in dynamic object: With correspondence across time frames. we just need to edit the first frame in sequence and the change will be automatically applied to the rest of the sequence.
36


Reviews:
[{"review_id": "71kzY3zRiB", "reviewer": "Reviewer_44mR", "summary": "The proposed method uses two representations jointly to model a 3D object over time from forward-facing monocular video to produce state of the art results on publicly available datasets. The method produces a volumetric representation anchored to a mesh and can be used to track the deformation of the object over time. The tracked mesh can then be manipulated such as editing the texture. To achieve this, the proposed method uses Gaussians anchored to a canonical mesh using a forward / backward deformation field and Gaussian merging / creation rules based on mesh vertices. The method uses DPSR, DiffMC and laplacian regularization to generate intermediate meshes.", "questions": "- What's the intuition behind the improvement when using both Gaussian and mesh rendering losses?\n- How would the method handle a synthetic large textured but planar surface such as a checkerboard where there might not be enough vertices to anchor gaussians to?\n- In the attached supplementary materials, the anchored gaussians are represented as blue dots. What do these blue dots represent - is it just the mean positions?\n- How is the \"canonical\" frame chosen given that the input video sequence is monocular and the scene is not static? Can you share more details about about the (frozen) initialization of the forward / backward deformation networks when training the canonical Gaussians and the SfM initialization?\n- Are there any qualitative observations relating to decomposition of geometry / appearance information between the mesh representation and the gaussians?\n- How do you arrive at running mesh anchoring every 100 iteration? Is it possible to perform differentiable mesh anchoring?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and follows a fairly intuitive and novel approach to generating both a deformable mesh and gaussian representation in parallel leveraging the benefits of both representations in the loss functions. The method performs at interactive speeds during inference while producing high quality results both qualitatively and quantitatively for both the mesh and Gaussian representations. The paper is evaluated on a variety of both real-world and synthetic datasets and produces tracked results using the forward / backward deformation of a canonical mesh. This is likely a significant improvement in learning deformable 3D scenes.", "weaknesses": "- The paper demonstrates that using both Gaussians and meshes improves the overall quality of novel view synthesis. There are several qualitative examples of higher quality meshes but it is unclear how / if appearance improves between the baselines and the proposed method when the underlying geometry is of similar quality.\n- There is a lack of rigorous quantitative evaluation for several key decisions such as choosing the initialization, DMTet/DiffMC/FlexiCubes, mesh anchoring frequency and the training methodology and relies on qualitative observations. While these qualitative observations likely hold up well, having quantitative evaluations of various modules is desirable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wiv3WCa8Ar", "reviewer": "Reviewer_wAGq", "summary": "The paper introduces Dynamic Gaussians Mesh (DG-Mesh), a novel framework designed to reconstruct time-consistent, high-fidelity meshes from monocular video footage. Utilizing 3D Gaussian Splatting (3DGS), the framework ensures temporal consistency and superior mesh reconstruction quality. Innovations such as Gaussian-Mesh Anchoring and cycle-consistent deformation significantly enhance the distribution and stability of 3D Gaussians across frames.", "questions": "Please see the weaknesses part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The manuscript is well-structured, with fluent writing and a clear explanation of methods, making it easy for readers to understand.\n\n2. This method combines the advantages of Mesh and 3D GS and achieves remarkable improvements in surface detail and rendering quality in dynamic object reconstruction.\n\n3. The experiment is thorough, with comprehensive benchmarks against established methods, demonstrating significant enhancements in both mesh quality and image rendering.", "weaknesses": "1. The integration of 3D Gaussians with mesh surfaces is good, yet the paper does not explore the potential benefits of using 2D Gaussian Splatting for surface and image reconstruction accuracy. Could a 2D representation be more effective or efficient when Gaussians lie on the mesh surface?\n\n2. The method involves separate rasterization processes for Mesh and 3D Gaussians. However, the paper lacks a detailed explanation of how color information is harmonized between these components. How is color from the 3DGS sh presentation passed to the mesh during rasterization?\n\n3. Binding a single Gaussian to each mesh face raises questions about the distribution logic. Why was this strategy chosen? Did the authors consider alternatives, such as placing multiple Gaussians on a single mesh face or varying the number of Gaussians based on mesh face properties?\n\n4. Table 1 only presents PSNRm metrics; however, standard PSNR metrics for direct 3DGS rendering results are absent. Could the authors clarify the reason for this omission?\n\n5. The paper does not address how varying the number of mesh surfaces extracted from the same scene affects rendering accuracy and computational efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xuUFIDUyQA", "reviewer": "Reviewer_mKQh", "summary": "The paper introduces a method for using deformable Gaussian Splatting to generate dynamic meshes for each frame of a monocular video. It employs a set of canonical Gaussians, deforming them at each timestep with an MLP. These deformed Gaussians are then used to construct a mesh for each frame through Poisson Reconstruction and Marching Cubes. The mesh serves as an anchor for the Gaussians, ensuring a one-to-one correspondence between mesh faces and Gaussians. Instead of textures, vertex colors are applied to the mesh, which is rendered as an image for supervision.\n\nThe paper is clear and straightforward, though it does have some limitations. The resulting mesh is a per-frame set rather than a truly deformable mesh. Additionally, the mesh is colored using vertex colors instead of a high-resolution UV map. These constraints raise concerns about the method’s applicability.", "questions": "1. Is there a way to unify the per-frame meshes into a deformable mesh? Once unified, this mesh could be optimized with a UV map to capture rich texture details. Neural temporal textures might also be useful, as they can address inaccuracies in deformation correspondence across frames by providing time-dependent texture adjustments.\n\n2. Can DG-Mesh be used to obtain a skeleton and skinned mesh? Extracting animatable assets from dynamic videos is crucial. This could be suggested as a potential enhancement in the future work section. The paper can articulate how it advances the field toward this goal to some extent.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper sets an ambitious goal: reconstructing dynamic meshes from monocular videos, a critical challenge in graphics and vision. It begins with a baseline approach using Deformable GS to model the dynamic scene. Mesh reconstruction is then applied to each frame, incorporating an anchoring constraint on the Gaussians through the mesh. This approach helps achieve a smooth surface but can adversely affect the rendering results.", "weaknesses": "Here are the weaknesses of the paper:\n\n1. **Mesh Representation:** The method produces a set of per-frame meshes rather than a deformable mesh. Although mesh representations are known for their rich operations and memory efficiency, the paper does not demonstrate how to register each frame to a unified mesh. This per-frame approach is not memory-efficient and makes editing geometry and textures cumbersome, as changes cannot be easily propagated across frames without manual effort.\n\n2. **Detail Preservation:** The mesh optimization and anchoring mechanism do not preserve details effectively, as demonstrated in Table 1. While meshes efficiently represent low-frequency geometry, high-frequency textures should ideally be stored in a UV map. This method colors the mesh at vertices, limiting its ability to capture high-frequency details. The one-to-one mapping between mesh faces and Gaussians necessitates a high-resolution mesh to accommodate the large number of Gaussians needed for rich texture and geometry representation. However, the mesh number is limited by the resolution of Marching Cubes and is hard to increase as much as needed Gaussians. This results in inefficient memory use, as mesh faces are not effectively utilized for storing detailed textures.\n\n3. **Missing Related Works:** The paper omits several closely related works:\n   - \"SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes\" by Huang et al.\n   - \"Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis\" by Li et al.\n   - \"Neural Parametric Gaussians for Monocular Non-Rigid Object Reconstruction\" by Das et al.\n   \n   It is recommended to compare with **SC-GS** in Table 1, as it is the current state-of-the-art method in this area. It's not necessary to overwhelm its result since the focus of the two papers differs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PHUxk3527M", "reviewer": "Reviewer_9ERR", "summary": "This paper proposes a method to extract high-quality mesh from image inputs in monocular/ efficient multiview setups. The authors propose a Gaussian-Mesh anchoring method and cycle-consistent deformation loss to enhance the upper bounds of DG-Mesh representations. The authors propose many downstream tasks to demonstrate the widely applications of DG-Mesh.", "questions": "Please refer to weakness. \nOverall, I think this is a pioneer work and My main consideration is weakness2,3.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This study presents a novel framework that enables the reconstruction of high-fidelity, time-consistent meshes from monocular videos. Generating meshes for dynamic scenes is a challenging yet critical task, and the author makes a significant contribution in addressing it.\n\n2. The proposed approach includes a Gaussian-Mesh anchoring technique that produces uniformly distributed Gaussians in each frame, enhancing the effectiveness of mesh reconstruction. Cycle-consistent deformation loss adjusts the deformed Gaussians, offering a new method for densification in deformation-based 3D Gaussians.", "weaknesses": "1. **Baseline Selection:** I suggest that the authors consider incorporating additional baselines such as Spacetime-GS (Li et al., CVPR 2024), 4DGS (Yang et al., ICLR 2024), and 4D-GS (Wu et al., CVPR 2024), given the recent advancements in Dynamic Gaussian representations. It would also be helpful to include some NeRF baselines in the supplementary materials.\n\n2. **Color Prediction:** As mentioned in Section 4.2, the mesh is extracted using Nvdiffrast. How do the authors handle the color extraction for the mesh? Is the DG-Mesh representation used for this purpose? If so, why? I find it challenging to apply this approach in real-world scenarios.\n\n3. **Real-World Monocular Scenes:** In my opinion, DG-Mesh is still difficult to apply in real-world settings, as the original 4D representations struggle to maintain multi-view space-time consistency. While the results appear satisfactory in training views, they might deteriorate in novel views or the final extracted mesh. I recommend that the authors emphasize the robustness of multi-view videos rather than focusing solely on monocular videos. In terms of monocular dynamic scene reconstruction, there are several relevant works (e.g., MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffolds, Dynamic Gaussian Marbles, Shape of Motion, Dreamscene4D, MoDGS) that have yet to deliver satisfactory results. Claiming effectiveness in monocular videos might be an overstatement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)