DreamMesh4D: Video-to-4D Generation with Sparse-Controlled Gaussian-Mesh Hybrid Representation
Authors: Zhiqi Li
Yiming Chen
Peidong Liu
Venue: NeurIPS 2024 poster (2024)
Keywords: 4D generation
3D Gaussian Splatting
dynamic mesh
diffusion model
Primary area: generative_models
TL;DR: A novel 4D generation framework, with Gaussian-mesh hybrid representation and sparse-controlled geometric deformation.

Abstract: Recent advancements in 2D/3D generative techniques have facilitated the generation of dynamic 3D objects from monocular videos. Previous methods mainly rely on the implicit neural radiance fields (NeRF) or explicit Gaussian Splatting as the underlying representation, and struggle to achieve satisfactory spatial-temporal consistency and surface appearance. Drawing inspiration from modern 3D animation pipelines, we introduce DreamMesh4D, a novel framework combining mesh representation with geometric skinning technique to generate high-quality 4D object from a monocular video. Instead of utilizing classical texture map for appearance, we bind Gaussian splats to triangle face of mesh for differentiable optimization of both the texture and mesh vertices. In particular, DreamMesh4D begins with a coarse mesh obtained through an image-to-3D generation procedure. Sparse points are then uniformly sampled across the mesh surface, and are used to build a deformation graph to drive the motion of the 3D object for the sake of computational efficiency and providing additional constraint. For each step, transformations of sparse control points are predicted using a deformation network, and the mesh vertices as well as the surface Gaussians are deformed via a novel geometric skinning algorithm. The skinning algorithm is a hybrid approach combining LBS (linear blending skinning) and DQS (dual-quaternion skinning), mitigating drawbacks associated with both approaches. The static surface Gaussians and mesh vertices as well as the dynamic deformation network are learned via reference view photometric loss, score distillation loss as well as other regularization losses in a two-stage manner. Extensive experiments demonstrate superior performance of our method in terms of both rendering quality and spatial-temporal consistency. Furthermore, our method is compatible with modern graphic pipelines, showcasing its potential in the 3D gaming and film industry.

DreamMesh4D: Video-to-4D Generation with Sparse-Controlled Gaussian-Mesh Hybrid
Representation
Zhiqi Li
Yiming Chen 112
Peidong Liu
Zhejiann University Westlake University
(lizhiqi49, chenyiming, Liupeidong//rrestllkes edu. cn
Abstract
Recent advancements 2D/3D generative techniques have facilitated the gener- ation of dynamic 3D objects from monocular videos. Previous methods mainly rely on the implicit neural radiance fields (NeRF) or explicit Gaussian Splatting as the underlying representation, and struggle to achieve satisfactory spatial-temporal consistency and surface appearance. Drawing inspiration from modern 3D an- imation pipelines, we introduce Dream-Mehh4D, novel framework combining mesh representation with geometric skinning technique to generate high-quality 4D object from monocular video. Instead of utilizing classical texture map for appearance, we bind Gaussian splats to triangle face of mesh for differentiable optimization of both the texture and mesh vertices. In particular, Drca begins with a coarse mesh obtained through an image-to-33 generation procedure. Sparse points then uniformly sampled across the mesh surface, and are used to build a deformation graph to drive the motion the 3D object for the sake of computational efficiency and providing additional constraint. For each step, transformations of sparse control points are predicted using deformation network, and the mesh vertices as well the surface Gaussians are deformed via novel geo- metric skinning algorithm. The skinning algorithm is hybrid approach combining LBS (linear blending skinning) and DQS (dual-quaternion skinning), mitigating drawbacks associated with both approaches. The static surface Gaussians and mesh vertices as well the dynamic deformation network are learned via reference view photometric loss, score distillation loss as well as other regularization losses in a two-stage manner. Extensive experiments demonstrate superior performance our method in terms of both rendering quality and spatial-tempont consistency. Furthermore, our method compatible with modern graphic pipelines, showcasing its potential in the 3D gaming and film industry. The source code is available at our website: https://lizhhs wwwpzhiiii...iiihiei.e github. SrDDreammeeh4D..
Introduction
The emergence and development of Generative Artificial Intelligence (GenAl) have significantly revolutionized 3D generation techniques recent years [20]. The technology has effectively allowed the creation of static objects, including their shape, texture, and even an entire scene from a simple text prompt single image. Recently, the wave of advancement has been propelled the filed of dynamic (4D) content generation [45], which offers immense potential in fields including, but not limited to, AR/VR, filming, gaming and animation. However, it's still quite challenging efficiently
Equual Contribution;
Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Timel
Time2
Time3
Time4
Composited Scene Demo
Generated Dynamic Objects
Given monocular videos, our method is able to generate high-fidelity dynamic meshes. We also produce a composited scene demo (top bar and left side of the figure) with the generated dynamic meshes, showcasing our method's compatibility with modern 3D engines.
Figure I:
generate high-quality 4D content due to its increased spatial-tempora complexity and higher demand on algorithm design.
The promising strides in 3D generation are largely attributed to the pre-trained large 2D diffusion models [39, 41, 31J. In particular, score distillation sampling (SDS) [33] enables the 3D generation [30, 55, 18, 42, 9] from scratch by distilling 3D knowledge from a pre-trained 2D diffusion model [39]. Following works on text-to-3D [24, 57, 44, 15] and image-to-3D [26, 35, 50, 48] have further improved the performance of 3D generation tasks both in quality and stability. Inspired by the successes of SDS- 3D generation, recent works [69, 1, 38, 12, 65, 66] explore generating 4D assets by distilling prior knowledge from pre-trained video diffusion models [54, 3] or novel-view synthesis models [26, 43]. Both text-to-4D [45, 1, 25] and image-to-4D methods [69] mainly rely on pre-trained video diffusion models, which are not yet capable of generating high-quality video, thus usually struggle to generate high-quality 4D content. On the contrary, video-to-4D methods [38, 12, 66, 65] directly generate 4D assets from off- off-the-sheef monocular videos, making the results more appealing and with better spatial-tempon
consistency. Existing video-to-4D methods either rely on the implicit dynamic NeRFs [12] explicit dynamic Gaussian splatting [38, 65, 66) as the underlying representation. Nevertheless, both of them do not have tight constrains for surface, leading to redundant optimization space and impeding the learning of deformation.
Inspired by modern graphic pipelines for 3D animation, we propose DreamMesh4D, which exploits 3D triangular mesh representation and sparse-controlled geometric skinning methods [47, 16] for video-to-4D generation. To better supervise the generation with 2D signals, instead of using classic mesh with UV texture maps, we choose a hybrid representation, SuGaR [9]. which marries 3D Gaussians to mesh surface for more elaborate appearance modeling. Flat Gaussians are bound to mesh faces based on barycentric coordinates hence the rendering process of 2D images is differentiable with respect to both the position of mesh vertices and the attributes of Gaussians. For high-quality object modeling and efficient motion driving of the object, our method is designed in a static-to-dynamic optimization manner. In particular, during the static stage, an initial coarse mesh is generated utilizing existing image-to-3D generation methods [26, 68, 61]. Then we refine its both geometry and texture by jointly optimizing the mesh vertices as well as the attributes of bound surface Gaussians under the hybrid representation via both the reference image photometric loss and the SDS loss. For dynamic learning, we uniformly sample sparse control nodes from the surface of the refined mesh, to build a deformation graph. Then each timestamp, transformation associated with each control node is predicted by deformation network. The deformation of all mesh vertices and surface Gaussians are obtained from those predicted transformations via a novel geometric skinning algorithm, which benefits from both the LBS (linear blending skinning) and DQS (dual-quaternion skinning) methods. The deformation network is optimized under the supervision of photometric loss from reference video frames, novel-view SDS loss and geometric regularization terms.

---

Extensive experiments are conducted and demonstrate that our method can generate high-fidelity dynamic textured mesh from monocular video, and significantly outperforms previous works both quantitatively and qualitatively, establishing new benchmark in the field of video-to-4D generation. As shown in Fig. I, our generated assets can be directly simulated in modern 3D engines, showeasing its potential in the 3D gaming and film industries.
2 Related Work
3D Generation Since the introduction of score distillation sampling (SDS) by DreamFusion [33], subsequent works [24, 5, 57, 35, 48, 49, 23] have significantly improved the performance of optimization-basee -based 3D generation algorithms. Many works adopt a multi-stage optimization strategy [24, 5, 57, 48] to enhance generated appearance. Another line of research [26, 44, 27, 19] focuses on training multi-vien diffusion models to inject multi-view supervision into SDS loss for addressing the Janus problem. DreamGaussian [49] and GaussianDreamer [64] pioneer the usage of 3D Gaussian [8] as the underlying representation and achieve 3D content generation in minutes. Although these methods demonstrate the potential of 3D Gaussian in 3D content generation, obtaining an object with high-quality geometry is still quite challenging. In this work, we explore to employ Gaussian Gauss hybrid representation [9] in our 4D generation tasks for better modeling of both object surface geometry and texture.
4D Representations Dynamic 3D representations form the foundation of 4D reconstruction and generation tasks. Most current methods extend static NeRF [30] to dynamic scenarios. These approaches, such as deformable [32, 34, 59] and time-varying [4, 7, 8, 6] NeRF-based methods, are prevalent. There are also some works trying to model dynamic with 4D neural implicit surface [56, 14). However, these implicit representations suffer from long optimization time and low reconstruction quality due to its computationally expensive volume rendering and implicit representation. Recent interest in 4D tasks has focused on 3D Gaussian representations due to their fast rendering speed and explicit nature. Some works [63, 58, 22] train networks to predict Gaussian kernel deformations, while others model kernel motion via polynomial representation or per-frame optimization [28, 21 Besides, both SC-GS 11] and HiFi4G [13] employ sparse control points for Gaussian kernel deformation, with SC-GS using LBS and HiFi4G using DQS drive Gaussians motion, thus ensuring -papool-temporal consistency. In this work, we propose to deform the object through a novel geometric skinning method, handling the artifacts associated with both LBS and DQS.
4D Generation Although great progress has been achieved in 3D generation tasks, 4D generation is still challenging due to its requirement on additional temporal supervision. Since current pre- trained video diffusion models [3, 54] still struggle to generate high-quality video contents, it is challenging to distill useful motion knowledge via SDS optimization. Hence, the performance of existing text-to-4D [45, 1, 25] and image-to-4D methods [69] usually struggle with low appearance quality. Another line of works focus video-to-4D generation [12. 38, 65]. These methods leverage current multiview diffusion models [26, 43, 27] to calculate the SDS loss [38, 12, 60) or generate per-frame multi-view images [66, 62] as supervision signal. Among them, a concurrent work of method, SC4D [60], optimizes a set of sparse-controlled dynamic 3D Gaussians by per-frame SDS loss from Zero 23 [26] with a coarse-to-fine strategy. However, the issue of unsatisfying surface quality caused by 3D Gaussian-bart based representation still exists. In contrast, our method is grounding on a Gaussian-mesh hybrid representation [9], enhancing the reconstruction quality both texture and geometry.
3 Method
In this section, we first introduce the relevant preliminaries in section 3.1. Then we illustrate the details of DreamMesh4D which is divided into static stage and dynamic stage in section 3.2 and 3.3 respectively. The overview of our method is presented in Fig. 2.
3.1 Preliminaries
Geometric Skinning Algorithms Given a mesh with N,, vertices, M = (V,F) where V {val,, E R²), E (1,2,... N,) the set of vertices and represents the triangle faces. In geometric skinning algorithms, there are also some sparse control nodes (bones/skcletons) P.
3

---

(pilp, E R (1, where Np is the number of control nodes, For mesh vertex E V. its deformation is calculated by blending a number of neighboring control nodes A(vv) through skinning algorithms. The local deformation for control node into a deformation matrix Pp E R3x3 and a translation vector 'p E TR and the deformation matrix can be further decomposed into a rotation matrix Rp E gaxx and shear matrix Sp R using polar decomposition. The strength of the influence of node p to vertex v can be represented as ulp and ΣpEN(v) Wp 1. Linear blending skinning (LBS) [47] computes the deformation of vertex by linearly blending the influence of nodes in (vv):
(v) can be decomposed
lbs
Σ DEN(V)
"p Up(Pp++(p)
LBS is widely used due to its simple formulation and natural animations. However, it suffers from the well-known "volume loss" "candy wrapper" artifacts under complex deformation. As an enhancement, dual-quaternion skinning (DQS) [16] represents the transformation of node with a unit dual-quaternion Sp DQ(Bp, (p). Then the deformation of can be calculated with DQS by:
vdles (vŠ", where {= EpEN(V)) upCp
(2)
||EpeN(() "pSp'l'
where Ç represents the conjugate of Ç. DQS can eliminate the artifacts associated with LBS, but cannot handle non-rigid deformation since the strain effect is not considered. It also suffers from an undesirable joint-bulging artifact while blending, which requires artistic manual work to fix [40].
3D Gaussians and SuGaR Gaussian Splatting [18] represents the scene as a collection of 3D Gaussians, where each Gaussian g is characterized by its center (4g E IR$ and covariance Eg E R3x3 The covariance Eg is parameterized by scaling factor 8g IR3 and rotation matrix represented via a unit quaternion Og E _2. Additionally, each Gaussian maintains opacity 0g E IR and color features Cg E C for rendering via splatting. Typically, color features are represented using spherical harmonics to model view-dependent effects. During rendering. the 3D Gaussians are projected onto the 2D image plane as 2D Gaussians, and color values are computed through alpha composition of these 2D Gaussians in front-to-back depth order. While the vanilla Gaussian Splatting representation may not perform well in geometry modeling. SuGaR [9] introduces several regularization terms to enforce flatness and alignment of the 3D Gaussians with the object surface. This facilitates extraction of a mesh from the Gaussians through Poisson reconstruction [17]. Furthermore, SuGaR offers a hybrid representation by binding Gaussians to mesh faces. A SuGaR mesh can be represented as S = [V, F. 9} where G denotes all surface Gaussians. For triangle face E F, the Gaussians ((f) bound on f are defined with barycentric coordinates. This hybrid representation allows joint optimization of texture and geometry through backpropagation.
3.2 Static Stage
The purpose of the static stage is to generate a refined Gaussian splats bounded triangular mesh. This procedure starts with a coarse mesh generated from a reference image I* that is sampled from all video frames I. Although there exists several fast mesh generation methods [68, 61]. we refer to Zero123 22-bbedd SDS optimization for its stability. In particular, we conduct simple SDS training on a set of randomly initialized 3D) Gaussians for fixed number of steps with regular densification and pruning. After that, we stop densification and pruning, and include SuGaR's regularization terms [9] to make Gaussians aligned to object surface. Finally all Gaussians with opacity lower than threshold σ 0.5 are pruned, after which Poisson reconstruction [17] is performed to extract a coarse mesh.
On the generated coarse mesh we attach x = 6 new flat Gaussians to every triangle face. For each training step. we render RGB image i' and mask A/" under reference view to calculate RGB loss L = f - 11 and mask loss L ||ă MM 112. The SDS loss L based on Zero123 [26] is also computed under randomly sampled views. The total loss for static SuGaR optimization is:
Estatic = AsssL'sss + Xaeffies + Naass L'masko
where AGDS, Atef and Nmast are the weights for different loss terms in static stage.

---

BOW
ganmuik
amplin
Ados
Aas
image
SuGaR opimazationn
contni mides
2
lineskamp
MLP
EsDE
KNN (GeoDtee)
prndem
deforned SuGaR & mesh DG
deformation graph (DN)
defumnal onttrl makes
Cray Eyaas
Adupiive by 'rnd
Dynamic stage
skinning
Figure 2: Overview of DreamMesh4D. In static stage shown in top left part, a reference image is picked from the input video from with we generate meussian-mmshh hybrid representation through image-to-3l pipeline. As for dynamic stage, we build deformation graph between mesh vertices and sparse control nodes, and then the mesh and surface Gaussians are deformed by fusing the deformation of control nodes predicted by a MLP through a novel adaptive hybrid skinning algorithm.
3.3 Dynamic Stage
In this section, we are going to delve into the deformation procedure for the Gaussian-mest hybrid representation. First, will discuss the construction of the deformation graph the surface of the refined mesh. Then, we will explain the deformation flow, which progresses from the sparse control nodes to the mesh vertices, and ultimately to the surface Gaussians. We will break down each step to give clear picture of the entire process.
3.3.1 Deformation Graph Construction
We begin by uniformly sampling Nnode sparse points on the surface of the refined mesh to serve as control nodes. To establish connections between the mesh vertices and the sparse control nodes, instead of using simple Euclidean distance to locate the nearest nodes (KNN), we pick N neighbor nodes with the shortest geodesic distance (as indicated by GeoDist in Fig. 2) based on the mesh's topology. This ensures that inappropriate connections between disparate mesh regions are avoided. Then, for vertex v, the influence up of each neighboring control node p E N N() is calculated following [47]:
t
||v
P&l dmax
2
Wp
Σ
EpreN(() up
where up
where dmax is the distance to the (Nneighhor + nearestt node.
3.3.2 Deformation with Adaptive Hybrid Skinning
Given that the object's texture was refined in the previous static stage, we fix the Gaussians' appear- ance properties (color and opacity) and focus dynamic learning phase solely on spatial properties (positions, rotations, scalings). Initially, the local deformations of the control nodes are predicted by deformation network The updated spatial properties post-deformation are subsequently computed by integrating the local deformations of the control nodes through geometric skinning. In particular, given a control node E P and timestamp т, the deformation network predict its local deformation at following the equation below. Note that omit the subscript "7" is omitted here and in rest paragraphs for simplicity.
(Rp, SpitSpt 7p) = (p()).
(5)
5

---

where Rp. Sp Rax3 and fp E R are the rotation, shear matrix and translation respectively. Furthermore, mitigate artifacts associated with LBS and DQS, we propose an adaptive fusion of their effects to calculate the deformation of mesh vertices. Here, 1/p E IR. denotes the local rigid strength, indicating the extent to which the region around p is influenced by DQS at time т.
Deformation of Control Nodes The predicted shear matrix Sp for node is intended only for LBS, whose strength is weaken by the predicted factor 7/p- and the final shear matrix this location is computed as:
Sp = (1 - 7/p)Spp Sp + Ipp,
where E xax represents an identity matrix indicating no strain effect. Afterwards, the new position of node p at timestamp T is:
== FpP + tp = Rp
Deformation of Mesh Vertices For the deformation of a specific vertex v using hybrid skinning, the new vertex position calculated with LBS, vlbs and that with DQS, vdge, can be obtained following Eq.1 and 2 respectively. The local rigid strength at v can be computed by linearly blending that of neighboring nodes:
V/v =
Σ DEN(v)
"p 7pp
(8)
then the fused position of v after deformation is the interpolation between dbb and ydge;
V =
mv) ) + 1/ vdg.
The local rotation and strain at v are the corresponding linear blending from neighboring control nodes:
Rv = exp
Σ wp logRR)
(10)
DEN(v)
S. = Σ
Up
Sp.
(11)
PEN(v)
Deformation of Surface Gaussians Now we have obtained deformations on the level of mesh vertices, afterwards the deformation on each Gaussian kernel will be calculated. Given a Gaussian kernel g E G(f) on a triangle face f = (Va: Vb, Ve) F, its new center at timestamp 7 is computed as:
Fg = v.Va + V + V.V.
(12)
where (Tas TT, Te) is the Gaussian's 's barycentric coordinate relative to the three triangle vertices, The new rotation is calculated by applying the local rotation Dgg fused from related vertices to its original rotation Qg:
Agg = exp(TT Ta logRv, + T log + The Rog
(13)
d = -Ag 9g-
(14)
We further apply the local shear matrix AS, fused from all three vertices to the original Gaussian scaling sy obtain the new scaling:
AS = na Sv. + TbSr, + TaSvor
(15)
5g = AS
(16)
Training Losses After obtaining the deformed hybrid mesh at timestamp we render its RGB 7: and alpha M; under reference view. We then compute the reconstruction loss Lref = || - r
and mask loss Lmask = ||M; 1l2 where T: and M are the ground truth image and mask of input video at timestamp For supervision under other views, we calculate SDS loss LsDs based on Zero123 [26] under randomly sampled views. Furthermore, our mesh-based representation naturally facilitates the introduction of local rigid constraints by leveraging the topology of the mesh. Specifically, the as-rigid-as -sspsiibele (ARAP) loss [46] is computed as:
LARPP= = Σ Σ
wn (x(v))l(V - v,)) - Rv(v - -v,)l13
(17)
VEV V.E(V))

---

t2
t
22
1
f2
t
L2
Reference
DreamGaussian4D
4DGen
Consisten
STAG4D
Ours
Figure 3: Qualitative comparison with baselines. We compare method with 4 previous video- to-4D methods. The first row provides two ground truth frames for each case. For each compared method, we render each case under reference view and another novel view the two timestamps. The result demonstrates that our method is able to generate sharper 4D content with rich details, especially for the novel views. Please zoom in for more details.
where C(v) represents the one-ring neighbors of vertex v, and w,,vv (v) is the cotangent weight [29] between v and its n-th connected vertex v,. Furthermore, we also introduce the normal consistency
loss Lve provided by PyTorch3D [37] on the deformed mesh to globally constrain the mesh surface.
Hence, the overall objective for our dynamic stage is weighted combination of the above loss terms:
Ldynamic AsnsLsnn Aref + Amast Lmask + AARAP + ANcLNC,
(18)
where Àsps, Aref, Amask AARAP and ANc are strengths of different loss terms dynamic opti- mization stage.
7

---

PSNR(rel) 1
SSIM(ref) 1
LPIPS
FVD J
FID-VID
CLIP
Consistent4D
26.58
0.935
0.133
929.39
31.84
0.917
DreamGaussian4D
31.06
0.947
0.143
994.11
32.73
0.913
4DGen frames)
27.02
0,937
0.137
913.10
63.32
0,909
STAG4D
27.99
0.941
0.136
1048.10
38.77
0.905
Ours
37.04
0.980
0.126
474.96
29.14
0.938
Table I: Quantitative comparison with baselines. Our method achieves best score on all metrics.
4 Experiments
4.1 Experiment Setup
Our quantitative results are evaluated on the test dataset provided by Consistent4D [12], which contains seven multi-view videos. Each video has one input view for scene generation and four testing views for evaluation. For qualitative evaluation, we curate set of challenging videos from previous works [12] and those generated by the video diffusion model SVD [3]. Evaluation metries: The per-frame LPIPS [67] score and the CLIP-scOn [36] are computed between the testing and rendered videos, with the final scores averaged over the four testing views. These two scores serve as image-level metric assess the structural and semantic similarity between the rendered images and the ground truth. Furthermore, we compute the FID-VID [2] and FVD [52] as video-level metrics to evaluate the video temporal coherence. Note that we report PSNR and SSIM values only for the reference view, as pixel- and patch-wise similarities are too sensitive to reconstruction differences, making them unsuitable for evaluating novel views in our generation task. However, we find they are suitable for evaluating the method's ability of modeling sharp features the reference view. Baselines: We compare method with previous video-to-4D generation methods: Consistent4D [12], DreamGaussian4D [38], 4DGen [65] and STAG4D [66]. All the experiments of above baselines conducted using the code from their official GitHub repository.
Dataset:
4.2 Comparison
Qualitative Comparison Fig. shows qualitative results of our method compared to other baseline works. The results reveal that our method generates 4D objects with higher fidelity and more details under reference view. And method also outperforms other works with better spatial-temporal consistency, demonstrating the effectiveness of our method. Please zoom in for more details.
Quantitative Comparison Table demonstrates superior performance of our method against other baseline works quantitatively. Specifically, our approach notably exceeds the existing state-of-the-ar in all measured metrics. Our method excels in both PSNR and SSIM, indicating a high level of reconstruction accuracy. Furthermore, the FVD score is particularly impressive, being only half that of competing methods. We also achieve the lowest FID-VID score, suggesting significant enhancement video quality produced by our 4D generation technique. Finally, our method achieves the lowest LPIPS and highest CLIP scores, ensuring both high image realism and semantic consistency. Overall, the numerical data clearly demonstrate the superior capabilities of our method in translating video to 4D content.
4.3 Ablation Study
In this section, we conduct ablation study to analyse the impact of various components on the performance of our method. The components under consideration include: (a) the choice between Euclidean and geodesic distance (FucDist and GeoDist) when constructing deformation graph; (b) our proposed adaptive hybrid skinning algorithm; (c) the ARAP and normal consistency terms (geometric regularization terms); (d) the choice between vanilla 3D Gaussians [18] and Gaussian- mesh representation for our base 3D representation.
EucDist vs. GeoDist Fig. 4(a) provides comprehensive qualitative analysis of the choice between EucDist and GeoDist. When using EucDist, the vertices on the elephant are incorrectly connected

---

RGB
Ground Truth
DQS
Normal
TucDist
GioDist
LBS
AIIS
(n)
(b)
(c)
Figure 4: Qualitative evaluation of ablation studies on: (a) choice between GeoDist and EucDist for deformation graph (DG) construction; (b) our proposed adaptive hybrid skinning (AHS) against LBS and DQS; (c) effects of ARAP and normal consistency (NC) loss.
PSNR(ref) f SSIM(ref) f
LPIPS
FVD
FID-VID 4
CLIP
SKinning=DQS Skinning=L.BS
36.28
0.978
0.126
479.83
29.78
0,940
36.68
0.980
0.155
540.86
32.19
0.928
w/o arap
37.04
0.979
0.142
751.56
42.08
0.907
w/o normal consistency
36.86
0.980
0.147
519.49
30.90
0.932
Full method
37.04
0.980
0.126
474.96
29.14
0.938
Table 2: Quantitative evaluation of ablation study different components. During experiments, we keep all other setup unchanged compared to the full method except the tested components. The quantitative scores show that full method achieves best performance almost all metrics.
to its body, resulting in significant deformation artifacts. Conversely, GeoDist correctly links the vertices neighboring nodes, enabling smooth object motion.
Adaptive Hybrid Skinning The upper two of Table 2 presents quantitative results when replacing our adaptive hybrid skinning with DQS and LBS respectively. Almost all metrics have a decrease compared using adaptive hybrid skinning, showcasing its robustness. We also provide a qualitative comparison on a dancing robot case in Fig. 4(b). When LBS or DQS is used, there artifacts on the robot's deformed elbow along with uneven surface. In contrast, the artifacts eliminated and the surface becomes smooth when our adaptive hybrid skinning is used.
Geometric Regularization Terms The third and fourth rows of Table 2 present the scores of metrics when either the ARAP or normal consistency term, respectively. is omitted from Equation 18. Decreases are observed across all metrics compared full loss terms and drops significantly when disabling ARAP loss. This circumstance matches the qualitative analysis shown in Fig. 4(c). Without ARAP term, appears serious distortions on the object geometry. When the normal consistency term is disabled, the object surface becomes less smooth, and consequently, the texture is impaired.
3D Gaussians vs. Gaussian-mesh Hybrid In Fig. 5, we provide the qualitative comparison between 3D Gaussians and the Gaussian-mesh hybrid representation for our base 3D representation. shows that when utilizing 3D Gaussians, the texture of generated objects blurry on those parts unseen in reference image. As contrary, Gaussian-mesh hybrid representation presents clean and high-quality texture under every view, which benefits from the sharp surface provided by mesh.
4.4 Limitations
Despite the superior results achieved, our work still exist several limitations. particular, our method relies pre-trained multi-view diffusion model (Zero123) for novel view supervision through SDS, leading long optimization time and limited performance on which case Zero cannot handle well. Moreover, method is currently only designed to generate 4D contents at object level from input

---

Reference
3D Gaussians
Gaussian mesh hybrid
Figure 5: Qualitative comparison on 3D representation between 3D Gaussians and Gaussian- mesh hybrid representation. When utilizing 3D Gaussians base 3D representation, the texture is blurry on the parts unseen in reference image. As comparison, the texture is clean and of high quality under every view when employing the Gaussian-mesh hybrid representation.
videos captured under fixed viewpoint. The extension of our framework to scene-level generation or videos captured with moving camera remains an area for future exploration. Finally, due the scarcity of test data, the performance of our method on more complex task is not evaluated. These identified limitations will be addressed our future research endeavors.
Conclusion
In this work, we introduce DreamMesh4D, an innovative vidco-to-4D framework that generates dynamic meshes through static- -tatic-t--dyamiic optimization process. By employing Gaussian-mesh hybrid representation, we simultaneously refine both the geometry and texture the object. This approach allows the static object to serve the dynamic stage, we construct a deformation graph on the object's surface using geodesic distance. Thereafter, the motion of the entire mesh, as well the surface Gaussians, are driven by sparse control nodes via a novel geometric skinning algorithm named adaptive hybrid skinning. It benefits from the strengths of both Linear Blending Skinning (LBS) and Dual-Quatennion Skinning (DQS), enabling more robust deformation. Extensive experiments have demonstrated the superior performance of our method in generating high-fidelity 4D objects. It significantly surpasses previous methods in both rendering quality and spatial-temporg consistency, establishing a new benchmark for video-to-4D tasks. While our method benefits lot from the mesh-based representation, it reveals promising direction in the field of video-to-4D generation. Furthermore, our method's compatibility with modern geometric pipelines showcases its potential applicability in the 3D gaming and film industries.
excellent starting point for dynamic learning. During
Acknowledgement
This work was supported part by National Science and Technology Major Project (20227D0115102 in part by National Natural Science Foundation of China (62202389), in part by grant from the Westlake University-Muyuan Joint Research Institute, and in part by the Westlake Education Foundation.
10

---

References
[1] Sherwin Bahmani, Ivan Skorokhodov, Victor Rong, Gordon Wetzstein, Leonidas Guibas, Peter Wonka, Sergey Tulyakov, Jeong Joon Park, Andrea Tagliasacchi, and David B Lindell. 4d-fy: Text-to-4d generation using hybrid score distillation sampling. arXiv preprint 2xiv:2311.17986 2023.
[2] Yogesh Balaji. Martin Renqiang Min. Bing Bai, Rama Chellappa. and Hans Peter Graf. Conditional gan with discriminative filter generation for text-to-video synthesis. In I/CAI, volume I, page 2. 2019.
[3] Andreas Blattmann, Tim Dockhor, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, al. Stable video diffusion: Scaling latent video diffusion models to large datasets, arXiv preprint arXiv:2311.15127, 2023.
[4] Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 130-14: 1100-11, 2023.
[5] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. arXiv preprint wr.vv:2303./3872 2023.
[6] Jiemin Fang. Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIGGRAPH Asia 2022 Conference Papers, pages 9. 2022.
[7] Sara Pridovich-Kell Giacomo Mcanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12479-12488, 2023.
[8] Chen Gao, Ayush Saraf. Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocular video. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5712-5721, 2021.
[9] Antoine Guédon and Vincent Lepetit. Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and hugh-quality mesh rendering. arXiv preprint arXiv:2311.1275,.0 2023.
[10] Yuan-Chee Guo, Ying-Tias Tian Liu, Ruizhi Shao, Christian Laforte, Vikram Voleti, Guan Luo, Chia- Hao Chen, Zin Zou, Chen Wang, Yan-Pei Cao, and Song Hai Zhang. threestudio: unified framework for 3d content generation. https:/ / /gtt com/ tom/threestuun eeseeessstii--oe poudio-projectto wttofjec//trreetuiiioo 2023.
[11] Yi-Hua Huang, Yang- Tian Sun, Ziyi Yang, Xiaoyang Lyu, Yan-Pei Cao,, and Xiaojuan Qi. Se-gs: Sparse- controlled gaussian splatting for editable dynamic scenes. arXiv preprint (X.K::23/2.149333 2023.
[12] Yanqin Jiang. Li Zhang, Jin Gao. Weimin Hu, and Yao Yao. Consistent4d: Consistent 360 ('deg] dynamic object generation from monocular video. arXiv preprint arXiv:231 2023.
[13] Yuheng Jiang, Zhehao Shen, Penghao Wang, Zhuo Su, Yu Hong, Yingliang Zhang. Jingyi Yu, and Lan
Xu. Hifi4g: High-fidelity human performance rendering via compact gaussian splatting. arXiv preprint arXiv:2312.03400 2023.
[14] Erik Johnson, Mare Habermann, Soshi Shimada, Vladislav Golyanik, and Christian Theobalt. Unbiased 4d: Monocular 4d reconstruction with a neural deformation model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6598-6607, 2023.
[15] Oren Katzir, Or Patashnik, Daniel Cohen-Or, and Dani Lischinski. Noise-free score distillation. arXiv preprint wxXi:2310.115990 2023.
[16] Ladislav Kavan, Steven Collins, Jifi Zára, and Carol O'Sullivan. Geometric skinning with approximate dual quaternion blending. ACM Transactions on Graphics (TOG), 1-33 2008.
[17] Michael Kazhdan, Matthew Bolitho, and Hugues Hoppe Poisson surface reconstruction. In Proceedings of the fourth Eurographics symposium on Geometry processing, volume 7, page 0, 2006.
[18] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics (ToG), 42(4):1-14, 2023.
[19] Weiyu Li, Rui Chen, Xuelin Chen, and Ping Tan. Sweetdreamer: Aligning geometric priors in 2d diffusion for consistent text-to-3d. arXiv preprini arXiv:2310.02544 2023.
[20] Xiaoyu Li, Qi Zhang. Di Kang, Weihao Cheng. Yiming Gao, Jingbo Zhang, Zhihao Liang. Jing Liao, Yan-Pei Cao, and Ying Shan. Advances in 3d generation: A survey. arXiv preprint 4xXiv22401.78000 2024.
11

---

[21]
Zhan Li, Zhang Chen. Zhong Li, and Yi Xu. Spacetime gaussian feature splatting for real-time dynamic view synthesis. arXiv preprint arXiv:2312.16812 2023.
[22] Yiqing Liang, Numair Khan, Zhengqin Li. Thu Nguyen-Phuoe Douglas Lanman, James Tompkin, and Lei Xiao. Gaufre: Gaussian deformation fields for real-time dynamic novel view synthesis. arXiv preprint arXiv:2312.1 1458,, 2023.
[23] Yixun Liang, Xin Yang, Jiantao Lin, Haodong Li, Xiaogang Xu, and Yingcong Chen. Luciddreamer: Towards high-fidelity text-to-3d generation via interval score matching. arXiv preprint arXiv 2023.
[24] Chen-Hsuan Lin, Jun Gao, Luming Tang. Towaki Takikawa, Xiaohui Zeng. Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Y Liu, and Tsung- Yi Lin. Magic3d: High-resolttion text-to-3d content creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 300-309, 2023.
[25] Huan Ling, Seung Wook Kim, Antonio Torralba, Sanja Fidler, and Karsten Kreis. Align your gaussians: Text-to-4d with dynamic 3d gaussians and composed diffusion models. arXiv preprint ar Arvv 13763, 2023.
[26] Ruashi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey 7akharov, and Carl Vondrick. Zero- 1-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 9298-9309, 2023.
[27] Yuan Liu, Cheng Lin, Zijiao Zeng Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Generating multiview-consistent images from single-view image. arXiv preprint arXiv:2309.0345: 033453330033,..... 2023.
[28] Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. arXiv preprint arXiv:2308. 09713, 2023.
[29] Mark Meyer, Mathieu Desbrun, Peter Schröder, and Alan H Barr. Discrete pntereenttalggommett operators for triangulated 2-manifolds. In Visualization and mathematics III, pages 35-57. Springer, 2003.
[30] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65 (1):99-106, 2021.
[31] OpenAL. Dall-e 3. https//openai..oo//tllleee
[32] Keunhong Park, Utkarsh Sinha. Jomathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo -rualla Nerties: Deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5865-5874, 2021.
[33] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Drcamfusion: Text-to-3d using 2d diffusion. In ICLR, 2023.
[34] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10318 10318-10327, 2021.
[35] Guocheng Qian, Jinjie Mai. Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin- Ying Lee. Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors. arXiv preprint arXiv:2306. 17843, 2023.
[36] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMIR,
2021.
[37] Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo. Justin Johnson, and Georgia Gkioxari. Accelerating 3d deep learning with pytorch3d. arXiv preprint arXiv:2007.08501 2020.
[38] Jiawei Ren, Liang Pan, Jiaxiang Tang, Chi Zhang, Ang Cao. Gang Zeng. and Ziwei Liu. Dreamgaussian4d: Generative 4d gaussian splatting. arXiv preprini 23Xiv:2312.17144 2023.
[39] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695 2022.
12

---

[40] Nadine Abu Rumman and Marco Fratarcangeli. State of the art in skinning techniques for articulated deformable characters. In International Conference Computer Graphics Theory and Applications. volume 2. pages 200-212. SciTePress, 2016.
[41] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton. Kamyar Ghasemipour, Raphael Gontijo Lopes. Burcu Karagol Ayan, Tim Salimans, al. Photorealistic text-to- image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479-36494 2022.
[42] Tianchang Shen, Jun Gao, Kangxue Yin, Ming- Yu Liu, and Sanja Fidler. Deep marching tetrahedra: hybrid representation for high-resolution 3d shape synthesis. Advances in Neural Information Processing Systems, 34:6087-6101, 2021.
[43] Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zerol23++: a single image to consistent multi-viww diffusion base model. arXiv preprint Xiv::23300. 151 10, 2023.
[44] Yichun Shi, Peng Wang. Jianglong Ye. Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for generation. arXiv preprint arXiv:2308. 161112 2023.
[45] Uriel Singer, Shelly Sheynin, Adam Polyak. Oron Ashual, lurüi Makarov, Filippos Kokkinos, Naman Goyal, Andrea Vedaldi, Devi Parikh, Justin Johnson, et al. Text-to-4d dynamic scene generation. arXiv preprint arXiv:2301. 12200 2023.
[46] Olga Sorkine and Marc Alexa. As-rigid-as- possible surface modeling. In Symposium on Geometry processing, volume 4, pages 109-116. Citeseer, 2007.
[47] Robert W Summer, Johannes Schmid, and Mark Pauly. Embedded deformation for shape manipulation. In ACM siggruph 2007 papers. pages 80-es. 2007.
[48] Jingxiang Sun, Bo Zhang, Ruizhi Shao, Lizhen Wang, Wen Liu, Zhenda Xie, and Yebin Liu. Dreameraftdd: Hierarchical 3d generation with bootstrapped diffusion prior. arXiv preprint arXiv:2310.168/ 2023.
[49] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint urXiv:2309.1665: 2023.
[50] Junshu Tang, Tengfei Wang, Bo Zhang. Ting Zhang, Ran Yi, Lizhuang Ma. and Dong Chen. Make-it-3d: ligh-fidelity 3d creation from single image with diffusion prior. arXiv preprint vrXiv:2303.14188 2023.
[51] Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph I Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12959-12977 2021.
[52] Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: new metric & challenges. arXiv preprint arXiv:1812.01717. 2018
[53] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj. and Thomas Wolf. Diffusers: State-of-the-an diffusion models. https://github. com/ 2022.
nuggingface/difffueee
[54] Jiuniu Wang. Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang, Modelscope text-to-video technical report. arXiv preprint arXiv:2308.0653 06571, 2023.
[55] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Weaping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint 106. 10689, 2021.
[56] Yiming Wang, Qin Han, Mare Habermann, Kostas Daniilidis, Christian Theobalt, and Lingjie Liu. Neus2: Fast learning of neural implicit surfaces for multi-view reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3295-3306, 2023.
[57] Zhengyi Wang, Cheng Lu, Yikai Wang. Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: High-fidelity and diverse rext-to-3d generation with variational score distillation. arXiv preprint ar0iv:2305. (3,13,100 2023.
[58] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian. and Xinggang Wang. 4d gaussian splatting for real- time dynamic scene rendering. arXiv preprint 10.55223, 2023,
13

---

[59] Tianhao Wu, Fangcheng Zhong, Andrea Tagliasacchi, Forrester Cole. and Cengiz Oztireli. Denerf: Self-supervised decoupling of dynamic and static objects from monocular video. Advances in neural information processing systems, 35:32653-32666. 2022.
[60] Zijie Wu, Chaohui Yu, Yangin Jiang. Chenjie Cao. Fan Wang. and Xiang Bai. Sc4d: Sparse-controlled video-to-4d generation and motion transfer. arXiv preprint arXiv:2404.03730, 2024.
[61] Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang. Shenghua Gao, and Ying Shan. Instantmesh: Efficient mesh generation from single image with sparse-vies view large reconstruction models. arXiv preprint arXiv:440.007191, 2024.
[62] Zeyu Yang, Zijie Pan, Chun Gu, Li Zhang. Diffusion": Dynamic content generation via score composition of orthogonal diffusion models. arXiv preprint urXiv:2404.02145 2024.
[63] Ziyi Yang, Xinyu Gao. Wen Zhou, Shaohui Jiao, Yuqing Zhang, Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. arXiv preprint arXiv:2309.13101 2023.
[64] Taoran Yi, Jiemin Fang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Qi Tian, and Xinggang Wang. Gaussiandreamer: Fast generation from text to 3d gaussian splatting with point cloud priors. arXiv preprint arXiv:2310.08529 2023.
[65] Yuyang Yin, Dejia Xu. Zhangyang Wang, Yao Zhao, and Yunchao Wei. 4dgen: Grounded 4d content generation with spatial-temporal consistency. arXiv preprint arXiv:2312. 2023.
[66] Yifei Zeng. Yanqin Jiang, Siyu Zhu, Yuanxun Lu, Youtian Lin, Hao Zhu, Weiming Hu, Xun Cao, and Yao Yao. Stag4d: tempooal anchored generative 4d gaussians. arXiv preprint arXiv:2403. 14939, 2024.
[67] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 586 595, 2018.
[68] Ruowen Zhao, Zhengyi Wang. Yikai Wang. Zihan Zhou, and Jun Zhu. Flexidreamer: Single image-to-3d generation with flexicubes. arXiv preprint arXiv:2404. 2400004400 2024.
[69] Yuyang Zhao, Zhiwen Yan, Enze Xie, Lanqing Hong. Zhenguo Li, and Gim Hee Lee, Animate124: Animating one image to 4d dynamic scene. arXiv preprint arXiv:2311. 2023.
14

---

A Appendix supplemental material
A.1 Additional Qualitative Results
Here we provide more qualitative comparisons of our method against baseline works in Fig. 6. Please see our project page for more temporal qualitative results with video format.
2
22
22
C2
Reference
DreamGausiian4)
4DGen
Consistent4D
STAG4D
Ours
Figure Additional qualitative comparison with baselines.
15

---

A.2 Implementation Details
Static Stage In the process of both coarse mesh generation and SuGaR refinement, the loss in Equation 3 is utilized for supervision, with the strength of different terms as X = 0.1. Neer 1000 and mask 500. For the generation of coarse mesh, set of randomly initialized 3D Gaussians optimized for 3000 steps in total. In the former 1500 steps, we do densification and pruning every 100 steps. After the 1500 steps, densification and pruning are stopped, and we introduce additional opacity binary and density regularization terms described in [9] into optimization until step 3000. Finally, we prune all Gaussians with opacity less than 0.5 and extract coarse mesh through Poisson reconstruction. Afterwards, we bind new flat Gaussians to each triangle face of the coarse mesh, and do optimization for 2000 steps.
In dynamic stage, defaultly sample Noode 1024 control nodes and assign Nneighbor = nnighhoring nodes for each vertex when constructing deformation graph. For each training step, 8 frames are randomly sampled from the input video for supervision. And for each sampled timestamp, we randomly sample 2 views for the calculation of SDS loss. All images are rendered at resolution 512 x 512 with white background. The camera distance world center is fixed as 3.8 and the degree of field-of-view (FoV) is fixed as 20 As for the strenghts of different loss terms, defautly set Asns 0.1, Xref 5000, Amask 500 and ANc 10. The value of AARAP is chosen case-specifically in [1, 10] according to the motion amplitude of object. The deformation network is zero-ly initialized and totally optimized for 2000 steps with learning as 0.00032. All of our experiments are conducted on a single NVIDIA RTX 4090 GPU.
Dynamic Stage
Licenses Here we provide the URL, citations, and licenses of open-sourced assets used in this work in Table 3.
URL
Citation
License
https://githen com/ wwt.//ott/etetttee hrtpheeettuuuo-porreettt/eeeeeae
[10]
Apache-2.0 license
https://github. com/ conn//mugginngaacc//dfffusssss
[53]
Apache-2.0 license
[37]
BSD License
https:// http://caaa-oollubaa//eroooooo
[26]
MIT License
https://gi thtbb..thhhu..... com/ htt///ntttttsSss
[9]
Gaussian Gaussian-Splattig Splatting License
Table URL, citations and licenses of the open-sourced assets used in this work.
A.3
Additional Experiments
Consistent4D
DreamGaussian4D
4DGen
STAG4D
Ours
Training Time
2.0h
0.6h
3.0h
1.6h
0.8h
Memory
28GB
20GB
15GB
7GB
8GB
Table 4: Comparison of computation cost of different methods.
Computation Cost In Table 4, we report the computation cost of our method and other compared baseline methods, demonstrating the computation efficiency of our method.
#Gaussians per face
PSNR(reft
SSIM(reft
LPIPS.
FVD.
FID-VIDJ
CLIP
36.17
0.977
0.134
523.39
27.18
0,939
3
36.55
0.979
0.129
496.46
27.60
0.943
4
36.60
0.979
0.128
519.35
26.81
0.940
6
36.63
0.979
0.127
477.63
25.96
0.940
Table 5: Quantitative evaluation ablation study on the number of Gaussians per triangle face.
Ablation on Number of Face Gaussians We also conduct comparisons on different number of Gaussians per face. The qualitative and quantitative comparison results are presented in Fig. 7 and Table A.3. The results demonstrate that the more Gaussains utilized per triangle face, the more detailed appearance can be obtained (e.g., the eyes and nose of the dog). Empirically we find that 6-Gaussians-pe fartnece can already deliver satisfying performance by considering the rendering quality and training time. Hence we keep this setup for all experiment cases.
16

---

Gaussians per faee Figure 7: Qualitative comparison on the number of Gaussians face. The appearance quality of details (e.g. the eyes and nose) getting better when binding more number of Gaussians on triangle face.
Causstio per face
Goussians per face
Gaussians per face
Ablation on Number of Control Nodes Here we conduct an additional ablation study on choice of the number of control nodes, Nnode, and the number of connected nodes for each vertex, Neeighbor. We try [256, 512, 1024] for Nnode and [4, 8, 16] for Nesighhor- and the quantitative results presented in Table 6. There are significant distinct scores of different metrics under different combination of node and Nneighhors except for PSNR(ref), on which (Noodd = 1024, Neesihhbrr = 4} achieves the highest score. While PSNR(ref) is key metric revealing reconstruction quality, pick (Noode = 1024, N Nneighbor = 4} as the default setup for our method.
PSNR(ref)
SSIM(ref)
LPIPS
FVD
FID-VID
CLIP
Node 1024, Nneighbor 16
36.39
0,979
0.128
559.43
32.08
0,932
Nuode = 1024, Nueighhor = 8
36.70
0.980
0.128
521.15
30.94
0.936
Nuode 1024, Nueighhor
37.82
0.980
0.128
516.46
31.20
0.937
Nuod: 512, Nneighhor 16
36.14
0.979
0.127
542.32
30.19
0.937
Nnode 512, Nneighbor
36.28
0,979
0.127
505.86
30.34
0,935
Nnode 512, Nneightor = 4
36.42
0.979
0.127
511.78
30.59
0.936
Nnode 256, Nneighbor 8
35.92
0.978
0.129
512.25
32.21
0.935
Nnode 256, Noeighhor 4
35.97
0.979
0.127
522.46
30.55
0.935
Table 6: Quantitative results of setup on number of control nodes and mesh vertex connectivity.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We describe our proposed algorithm in section 3 in detail, and the extensive experiment results demonstrate the claimed superior performance compared to previous methods.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer section 4.4 our main paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-specfiiaatinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
18

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Our method is described in details, and we also specify all implementation details our appendix. All assets used by this work are open accessed online and they are referenced soundly in paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
19

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We have provided the link to our project page and code at the end of abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /Cooeeeooiissseeeeeee
//niss httppsccccpbccccccceececcccccccc /tteggudddeeeeoeSummiiieeee wwssssnoolicgo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Descibed section A. of appendix. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The quantitative results are not accompanied by error bars, confidence intervals, nor statistical significance tests.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
20

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Descibed section A. of appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't It make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https http://eeuuppp.eeeeee wccppublic/thiicsuuuuteuee
Answer: [Yes]
Justification: No justification.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: As described in abstract and introduction part, our method is compatible with modern geometric pipelines, showcasing its potential in the 3D gaming and film industry.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
21

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This work poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The URL, citations and licenses of open-sourced assets used in this work provided in Table 3 of appendix.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
22

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
23

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "a4XSLWNjHX", "reviewer": "Reviewer_7V8w", "summary": "This paper proposes DreamMesh4D that combines mesh representation with sparse-controlled deformation technique to generate high-quality 4D objects from a monocular video. The authors bind Gaussian splats to the surface of the triangular mesh for differentiable optimization of both the texture and mesh vertices. The method begins with a coarse mesh from single image based 3D generation. Sparse points are then uniformly sampled on surface of the mesh to build a deformation graph, which drives the motion of the 3D object. For each step, transformations of sparse control points are predicted using a deformation network. The mesh vertices and the bound Gaussians are deformed via a geometric skinning algorithm that combines LBS and DQS. Reference view photometric loss, score distillation loss, and regularization losses are used in a two-stage learning. Experiments are performed on the consistent4D dataset.", "questions": "1. One of the main idea of this paper is to use a Gaussian-mesh hybrid representation, what is the difference between the proposed method and SuGaR?\n2. In figure 1, the input is a composited scene video. Is a segmentation method needed to extract each object region to use as input to the method? There seems no mention on this in the rest of the paper. How is the mutual-occlusion between the objects solved by the proposed method for the scenario in figure 1?\n3. What is the computational cost of the proposed method?", "limitations": "Not applicable.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The methods address a challenging problem to generate dynamic objects from a monocular video. \n- A new method is proposed that generates dynamic meshes through a static-to-dynamic optimization process. By employing a Gaussian-mesh hybrid representation, the authors simultaneously refine both the geometry and texture of the object. This approach allows the static object to serve as an excellent starting point for dynamic learning. During the dynamic stage, a deformation graph is used on the object’s surface using geodesic distance.\n- Experiments show the superior performance of this method in generating high-fidelity 4D objects. \n- The method use dynamic mesh with good compatibility with modern geometric pipelines in the 3D gaming and film industries.", "weaknesses": "- One of the main idea of this paper is to use a Gaussian-mesh hybrid representation, which was originally proposed in the work of SuGaR. The technical difference that is designed to address the special setting of this paper’s task is not clearly explained.\n- In figure 1, the input is a composited scene video. But in the rest of the paper, the inputs are well segmented object video, which makes the setting of the addressed task confusing.\n- The method combines some existing methods such as SuGaR, deformation graph, LBS and DQS based skinning. It’s not clear to identify the author’s innovative contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The authors are grateful for the thoughtful feedback and valuable questions. Thanks for the support for this work! Below we will address the questions and concerns separately.\n\n### **Q1: The difference between the proposed method and SuGaR.**\n**A**: While SuGaR proposes a pipeline reconstructing static surface via a Gaussian-mesh hybrid representation from pre-trained 3D Gaussians, our work mainly exploit the proposed hybrid representation as our base representation, and focus on extending it for video-to-4D generation task, similarly as most prior works which rely on NeRF or 3D-GS as the base representation. \n\nAlthough we utilize the static 3D representation proposed by SuGaR, trivially applying it to 4D tasks brings challenges. For example: 1) how to effectively preserve the Gaussian appearance while the asset undergoes deformation (i.e. surface Gaussian attributes update during deformation); 2) how to efficiently and effectively deform the vertices of the mesh-Gaussian representation (i.e. deformation graph and hybrid skinning). \n\n### **Q2: Concerns about Figure 1.**\n**A**: The input of our method is a monocular video of the target object under a fixed view, which is consistent with the setup used in previous works. As for the composite scene video in Figure 1, it is a demo created by ourselves in Blender using our generated assets. It is to demonstrate that the generated 4D assets by our method can be directly integrated into downstream graphic engines, which further prove the advantage of our work. We will make this more clear in the final version.\n\n### **Q3: Innovative contribution of this work.**\n**A**: The innovative contribution of our work is that we for the first time demonstrate the possibility and superior advantages of extending Gaussian-mesh representation for 4D tasks, in comparison to prior works, which mainly exploit NeRF or 3D-GS as the underlying representation. As R3 also points out that it is informative to the community that SuGaR works well in a more complex 4D pipeline. Although SuGaR, deformation graph, LBS and DQS are existing techniques, trivially integrating them would not lead to optimal performance. We therefore 1) proposed to change the Gaussian attributes during deformation; 2) proposed a novel skinning algorithm suitable for such optimization-based scenarios with insufficient constrains. The experimental results demonstrate the effectiveness of our design and our method achieves superior performance in comparison to prior baseline methods.\n\n### **Q4: Computation cost of the proposed method.**\n**A**: The time and memory complexities are 0.8h and 8G for training on a single NVIDIA RTX 6000Ada graphic card, which is also superior compared to prior methods and further demonstrates the advantage of our formulation. The detailed metrics in comparison to other methods can be found in the global rebuttal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BPAop8udsG", "reviewer": "Reviewer_ow51", "summary": "DreamMesh4D is a novel framework for transforming static images into 4D dynamic meshes. It refines object geometry and texture using a Gaussian-mesh hybrid and a geodesic-based deformation graph. A new skinning algorithm combines the best of LBS and DQS for enhanced deformation. The method excels in creating high-quality 4D objects with improved rendering and consistency.", "questions": "The current article lacks interesting insights. I suggest that the author should focus more on the advantages of this representation and explain why it is important for the 4D generation task.", "limitations": "Refer to the Weaknesses and Questions section", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "1.The results show an improvement in both quantitative and qualitative metrics compared to previous methods.\n2.The use of points to control motion is an interesting approach.\n3.The writing is clear.", "weaknesses": "1.The novelty of this work is somewhat limited in comparison to previous methods. The training strategy, the structure of the loss function, and the design of deformation are quite similar. The main difference lies in the representation, which is a predictable technical combination.\n\n2.I find that the detail of the results is superior to those of previous methods. Therefore, in section 4.2, the author should elucidate which aspects of the pipeline design have contributed to the improved quality compared to the previous method's design.\n\n3.The paper argues that previous works have significant time and memory complexity, but there is a lack of comparison regarding time and GPU usage costs. It would be beneficial to include such a comparison to demonstrate the efficiency of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The authors are grateful for the valuable and in-depth feedback of the reviewer. Below we address the mentioned concerns separately.\n\n### **Q1: Limited novelty compared to previous methods.**\n**A**: Previous methods for monocular video-to-4D generation include Consistent4D, DreamGaussian4D, 4DGen and STAG4D. All of them directly employ existing dynamic neural radiance fields or 3D Gaussian Splatting as their base representations, and predict query points' deformation with a MLP network. In contrast, we are the first to exploit mesh as the underlying representation for the problem and demonstrate superior effectiveness compared to prior methods. Although it might be predictable, no prior work has really tried and demonstrated the possibility of the idea for this challenging task. As both reviewer 1 and reviewer 3 also agree on, the method to generate dynamic meshes is new and informative. The authors humbly think the proposed method would provide valuable insight for following works. \n\n### **Q2: Which aspects of the pipeline have contributed to the quality improvement compared to previous methods?**\n**A**: The main aspect contributing to the quality improvement is the employment of mesh representation. 3D mesh models surface explicitly and is crucial for appearance modeling and animation. To validate the claim, we further conduct a thorough ablation study by replacing the mesh representation with no surface bound 3D Gaussian Splatting, and leave other parts unchanged. The experimental results demonstrate the mesh representation achieves a PSNR 36.73 dB compared to 28.71 dB with 3D-GS. It demonstrates the superior advantage by exploiting surface bound Gaussian Splatting for this challenging less constrained optimization problem. Detailed quantitative and qualitative evaluation results can be found in the rebuttal PDF (i.e. Table 2 and Figure 1).  \n\n### **Q3: Concerns about computation cost.**\n**A**: The detailed comparisons against prior works in terms of time and GPU memory usage can be found in the global rebuttal section. We evaluate all the methods on a single NVIDIA RTX 6000Ada graphic card since Consistent4D cannot run on a 24G GPU. Our method requires 0.8h and 8G memory for training, which is superior compared to Consistent4D, DreamGaussian4D, 4DGen and STAG4D. For example, Consistent4D requires 2h and 28G memory for training, STAG4D requires 1.6h and 7G memory for training. \n\n### **Q4: Insights of this work.**\n**A**: The 4D generation task is inherently not a well constrained problem, usually only a sequence of monocular video is given. Additional constrains should thus be enforced to achieve better performance. Prior works usually rely on a pre-trained image diffusion model to provide extra constrain in the form of score distillation loss. Although they can deliver impressive performance, the quality of the generated 4D asset still has room for improvement, e.g. the asset usually exhibits blurry appearance or unsatisfactory geometry as shown in our experimental results. \n\nThe authors suspect the performance gap might be caused by the used loosely constrained 4D representation, i.e. NeRF or 3D-GS with an additional deformation network for dynamic modeling, which is commonly used for multi-view 4D reconstruction problem. Since multi-view cues are usually sufficient to constrain the problem, prior commonly used 4D representations would thus still be able to deliver good performance. However, it is not the case for 4D generation from a monocular video, which provides much less multi-view cues. \n\nTherefore, the authors thought additional constraint should be enforced to better constrain this more challenging problem, which motivates us to explore the usage of the Gaussian-mesh hybrid representation. Gaussian-mesh representation was firstly proposed in SuGaR and used for triangular mesh extraction from a pretrained 3D-GS model. It enforces 3D Gaussians to bind on the object surface and degenerate towards 2D Gaussians. This kind of enforcement reduces the parameter searching space of the optimization problem, and thus could deliver better performance. The experimental results also prove our hypothesis and demonstrate superior performance compared to prior works. \n\nHowever, it is not trivial to simply integrate Gaussian-mesh representation into a 4D pipeline. For example, 1) how to efficiently and effectively deform the mesh-Gaussian representation for 4D dynamics? 2) how to make sure the surface bound Gaussians are still able to render high-quality textures under deformation? It is those challenges motivating us: 1) to propose the new skinning algorithm combines the best of LBS and DQS for this particular problem; 2) to propose the deformation of surface Gaussian attributes; 3) the usage of deformation graph to provide additional geometric deformation constraints. It is all those factors and insights leading to the final formulation and superior performance of our method. \n\nFurthermore, mesh based representation is already very mature for modern CG pipelines. Generated 4D assets with mesh representation and skinning method can be conveniently exported to existing commercial pipelines, which further motivates us to exploit this representation. The authors humbly think our work would be valuable to the community and provide good insight for following works in relevant fields.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ILg1XQSRK6", "reviewer": "Reviewer_2HE6", "summary": "The paper proposes a framework to generate animated surfaces from input videos. It has a static step to generate a base triangle mesh, and textures represented by 3D Gaussians. A subsequent dynamic step changes the positions of the mesh vertices and the gaussians. The framework leverages on Zero123 and SuGaR to generate the mesh and gaussians, integrating loss terms from those approaches in its training mechanism. Tha dynamic step uses geodesic distances to create a deformation graph and include As-Rigid-As-Possible and Normal Consistency loss terms.", "questions": "(1) I would like to know more about the temporal coherence in the proposed approach, since a video is not available. Even though the paper show metrics in that respect I would also like to see qualitative results.\n\n(2) Even though the results are probably ready for a CG pipeline, I believe the training and inference must be computationally expensive. I would like to know the training and inference times, and the hardware used.\n\n(3) Does only changing positions, rotations, and scaling of the gaussians in the deformation step result in accurate view-dependent effects? In theory the spherical harmonics should also be recalculated.\n\n(4) Why 6 gaussians per triangle in the static stage?\n\n(5) Equation 10 should come with a small explanation to give intuition. Mentioning that the logarithmic rotation is a mapping to the Lie Algebra and that the exp is a conversion back to the rotation matrix form would at least give a direction for a reader that is not a specialist.\n \n(6) I believe the references in line 38 [29, 54, 17, 41, 9] are not associated directly with score distillation sampling. Is that a typo?", "limitations": "Yes, the paper has a specific section describing limitations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) This paper is very informative and well contextualized. I am a specialist in geometry but not in Diffusion models, but I was able to use the Introduction and Related Work sections to familiarize myself with the theme. That was very fun. There are lots of the references from very recent papers, so good job on keeping track of the field!\n\n(2) The results are probably ready for current Computer Graphics pipelines, since it uses meshes and skinning.\n\n(3) I liked the mathematical modeling. Explaining everything in terms of the losses made a good paper structure. I have reservations about the specific mathematical notation (see Weaknesses), but reading was good overall.\n\n(4) I liked the proposed method. All losses terms make sense and are intuitive. Knowing that the SuGaR SDS would work well in a more complex pipeline was informative.\n\n(5) I liked the use of geodesic distance to compute the deformation graph. It is more robust than euclidean distance indeed.", "weaknesses": "(1) I missed a video in the supplementary material. As a 4D approach, the paper would benefit greatly from showing one. It shows metrics about temporal coherence, but a video would avoid any possible doubt.\n\n(2) I'm missing references about 4D neural implicit surfaces in the 4D Representations paragraph in the Related Work section. They also deal with dynamic surfaces in neural pipelines. I believe a sentence including all of them would be sufficient.\n\nNovello, Tiago, et al. \"Neural Implicit Surface Evolution.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\nGuandao Yang, Serge Belongie, Bharath Hariharan, and Vladlen Koltun. Geometry processing with neural fields. Advances in Neural Information Processing Systems, 34, 2021.\n\nMehta, Ishit, Manmohan Chandraker, and Ravi Ramamoorthi. \"A level set theory for neural implicit evolution under explicit flows.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n\n(3) The mathematical notation is a little bit polluted, impacting reading in my opinion. Subscripts and superscripts are used too much. The text would be cleaner by using more symbol letters instead. A good rule of thumb is to avoid using subscripts and superscripts unless they represent indices. \n\nThe same letters are also used for very different contexts. N is used for number of vertices and control point set, for example. Even though different fonts are used in each context, the cognition when reading is to remember N and forget the font used. It would be cleaner to use different symbols.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The authors are grateful for the insightful feedback and support from the reviewer. Below we address the mentioned concerns separately.\n\n### **Q1: Video qualitative results.**\n**A**: In the first paragraph of appendix, we have provided the link to our anonymous project page, which contains the input video and our rendered video results as well as our blender demo. Please check it.\n\n### **Q4: Concerns about computation cost of training and inference**\n**A**: For inference, which is the process of computing the deformed 3D object under a given timestamp, we report the time consuming of different methods as below: \n||Consistent4D|DreamGaussian4D|4DGen|STAG4D|Ours|\n|:---:|:---:|:---:|:---:|:---:|:---:|\n|Inference Time|51ms|3ms|3ms|3ms|3ms(MLP)+11ms(Skinning)|\n\nAmong these methods, Consistent4D utilizes implicit representation and volume rendering, leading to much more query points for the deformation network and hence the longest inference time (51ms). Compared to those Gaussian-based methods, DreamGaussian4D, 4DGen and STAG4D, our method has a little bit slower inference time (14ms). The extra computation overhead comes from the skinning calculation, which is implemented with pure PyTorch. This overhead can be eliminated through some engineering effort of implementing it as a CUDA extension, and we leave it for future works.\n\nAs for the computation cost of training, we have reported the overall training time and GPU memory usage in our global rebuttal. Compared to other methods, our method is the most computational efficient considering both the time and memory consuming (0.8h and 8GB).\n\n### **Q5: Concerns on updating spherical harmonics.**\n**A**: Following previous 3D generation works on 3D Gaussians, we use 0-degree spherical harmonics which is actually equivalent to RGB color without view-dependent effects. Hence we do not compute the deformation of spherical harmonics. We would explore to use higher level SH-coefficients for view-dependent effects in future. \n\n### **Q6: Why 6 Gaussians per triangle in the static stage?**\n**A**: By compromising on the appearance representation capability and computational cost, we choose 6 Gaussians per triangle. More Gaussians would enable a better capability to represent the surface appearance, with an expense of more computational cost. As shown in Figure 2 and Table 3 of the supplemented PDF rebuttal, the experimental results demonstrate 6-Gaussians can already deliver satisfying performance by considering the rendering quality and training time. \nIt is worth mentioning that, even with only 1 Gaussians per face, our method still largely outperforms previous methods with the fastest training time. \n\n### **Q2\\&Q3\\&Q7\\&Q8: Concerns about writing.**\n**A**: We really appreciate your advice on the paper writing, and we will revise the manuscript according to the constructive suggestions. We will include the works on 4D neural implicit surfaces in the 4D representations paragraph in the Related Work section. All mathematical notations in the paper will be carefully simplified for easier reading. And we will supplement a proper explanation about Equation 10 as you mentioned in Question (5). Finally, the references in line 38 are indeed a typo, and the words before these references should be \"differentiable 3D representations\" rather than \"3D generation\".", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers for their valuable feedback and agree that: 
* we address a challenging problem to generate dynamic objects from a monocular video;
* the method to generate dynamic meshes through a static-to-dynamic optimization process is new, and the losses terms make sense and are intuitive; It was also informative that the SuGaR SDS would work well in a complex pipeline of 4D task;
* the usage of mesh and skinning has good compatibility with modern computer graphic pipelines in the 3D gaming and film industries; 
* the experimental results show the superior performance of our method in generating high-fidelity 4D objects compared to previous methods; 
* the paper is clearly written, very informative, well contextualized, in good structure and has a good track of the prior methods.

In the following, we will first report both the time and memory complexities of our method against prior works requested by all reviewers, and then address individual reviewer's concerns respectively. All the responses will be incorporated into the final version.   

## Comparisons on time and memory complexities.

||Consistent4D|DreamGaussian4D|4DGen|STAG4D|Ours|
|:---:|:---:|:---:|:---:|:---:|:---:|
|Training Time|2.0h|0.6h|3.0h|1.6h|0.8h|
|Memory|28GB|20GB|15GB|7GB|8GB|

The above table shows the computation cost of our method compared to prior works. Our method is efficient in terms of both training time and memory consumption. In particular, our method requires 0.8h and 8G memory during training, which is more efficient than Consistent4D and 4DGen. Although DreamGaussian4D requires 0.6h to train the model, they require much more memory (i.e. 20G). STAG4D requires similar level of memory consumption, however they requires two times more training time than ours. Since Consistent4D cannot run on GPUs with 24GB of memory, we train all methods on a single NVIDIA RTX 6000Ada graphic card with 48GB of memory for fair complexity comparisons. The results further demonstrate the advantage of our method compared to previous works.

Meta-review: This work introduces a novel framework for transforming static images into 4D dynamic meshes, DreamMesh4D. It refines object geometry and texture using a Gaussian-mesh hybrid and a geodesic-based deformation graph. A new skinning algorithm combines the best of LBS and DQS for enhanced deformation. The method excels in creating high-quality 4D objects with improved rendering and consistency.

The work addresses a challenging problem to generate dynamic objects from a monocular video; the method to generate dynamic meshes through a static-to-dynamic optimization process is novel, and the losses terms make sense and are intuitive; It was also informative that the SuGaR SDS would work well in a complex pipeline of 4D task; the usage of mesh and skinning has good compatibility with modern computer graphic pipelines in the 3D gaming and film industries; the experimental results show the superior performance of the proposed method in generating high-fidelity 4D objects compared to previous methods;
the paper is clearly written, very informative, well contextualized, in good structure and has a good track of the prior methods. The rebuttal addresses all the reviewers' concerns. 

Overall, the Gaussian-mesh representation for 4D tasks is inspiring and will lead further developments along this direction.
Decision: Accept (poster)