DreamMesh4D: Video-to-4D Generation with Sparse-Controlled Gaussian-Mesh Hybrid Representation

Authors: zhiqi  li
yiming  chen
peidong  liu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 10:30:46

Abstract: Recent advancements in 2D/3D generative techniques have facilitated the generation of dynamic 3D objects from monocular videos. Previous methods mainly rely on the implicit neural radiance fields (NeRF) or explicit Gaussian Splatting as the underlying representation, and struggle to achieve satisfactory spatial-temporal consistency and surface appearance. Drawing inspiration from modern 3D animation pipelines, we introduce DreamMesh4D, a novel framework combining mesh representation with geometric skinning technique to generate high-quality 4D object from a monocular video. Instead of utilizing classical texture map for appearance, we bind Gaussian splats to triangle face of mesh for differentiable optimization of both the texture and mesh vertices. In particular, DreamMesh4D begins with a coarse mesh obtained through an image-to-3D generation procedure. Sparse points are then uniformly sampled across the mesh surface, and are used to build a deformation graph to drive the motion of the 3D object for the sake of computational efficiency and providing additional constraint. For each step, transformations of sparse control points are predicted using a deformation network, and the mesh vertices as well as the surface Gaussians are deformed via a novel geometric skinning algorithm. The skinning algorithm is a hybrid approach combining LBS (linear blending skinning) and DQS (dual-quaternion skinning), mitigating drawbacks associated with both approaches. The static surface Gaussians and mesh vertices as well as the dynamic deformation network are learned via reference view photometric loss, score distillation loss as well as other regularization losses in a two-stage manner. Extensive experiments demonstrate superior performance of our method in terms of both rendering quality and spatial-temporal consistency. Furthermore, our method is compatible with modern graphic pipelines, showcasing its potential in the 3D gaming and film industry.

Main paper: 
 introduction: 
 The emergence and development of Generative Artificial Intelligence (GenAI) have significantly revolutionized 3D generation techniques in recent years [20] . The technology has effectively allowed the creation of static objects, including their shape, texture, and even an entire scene from a simple text prompt or a single image. Recently, the wave of advancement has been propelled to the filed of dynamic (4D) content generation [45] , which offers immense potential in fields including, but not limited to, AR/VR, filming, gaming and animation. However, it's still quite challenging to efficiently Figure 1 : Given monocular videos, our method is able to generate high-fidelity dynamic meshes. We also produce a composited scene demo (top bar and left side of the figure) with the generated dynamic meshes, showcasing our method's compatibility with modern 3D engines.generate high-quality 4D content due to its increased spatial-temporal complexity and higher demand on algorithm design.The promising strides in 3D generation are largely attributed to the pre-trained large 2D diffusion models [39, 41, 31] . In particular, score distillation sampling (SDS) [33] enables the 3D generation [30, 55, 18, 42, 9] from scratch by distilling 3D knowledge from a pre-trained 2D diffusion model [39] . Following works on text-to-3D [24, 57, 44, 15] and image-to-3D [26, 35, 50, 48] have further improved the performance of 3D generation tasks both in quality and stability. Inspired by the successes of SDS-based 3D generation, recent works [69, 1, 38, 12, 65, 66] explore generating 4D assets by distilling prior knowledge from pre-trained video diffusion models [54, 3] or novel-view synthesis models [26, 43] . Both text-to-4D [45, 1, 25] and image-to-4D methods [69] mainly rely on pre-trained video diffusion models, which are not yet capable of generating high-quality video, thus usually struggle to generate high-quality 4D content. On the contrary, video-to-4D methods [38, 12, 66, 65] directly generate 4D assets from off-the-shelf monocular videos, making the results more appealing and with better spatial-temporal consistency. Existing video-to-4D methods either rely on the implicit dynamic NeRFs [12] or explicit dynamic Gaussian splatting [38, 65, 66] as the underlying representation. Nevertheless, both of them do not have tight constrains for surface, leading to redundant optimization space and impeding the learning of deformation.Inspired by modern graphic pipelines for 3D animation, we propose DreamMesh4D, which exploits 3D triangular mesh representation and sparse-controlled geometric skinning methods [47, 16] for video-to-4D generation. To better supervise the generation with 2D signals, instead of using classic mesh with UV texture maps, we choose a hybrid representation, SuGaR [9] , which marries 3D Gaussians to mesh surface for more elaborate appearance modeling. Flat Gaussians are bound to mesh faces based on barycentric coordinates hence the rendering process of 2D images is differentiable with respect to both the position of mesh vertices and the attributes of Gaussians. For high-quality object modeling and efficient motion driving of the object, our method is designed in a static-to-dynamic optimization manner. In particular, during the static stage, an initial coarse mesh is generated utilizing existing image-to-3D generation methods [26, 68, 61] . Then we refine its both geometry and texture by jointly optimizing the mesh vertices as well as the attributes of bound surface Gaussians under the hybrid representation via both the reference image photometric loss and the SDS loss. For dynamic learning, we uniformly sample sparse control nodes from the surface of the refined mesh, to build a deformation graph. Then at each timestamp, transformation associated with each control node is predicted by a deformation network. The deformation of all mesh vertices and surface Gaussians are obtained from those predicted transformations via a novel geometric skinning algorithm, which benefits from both the LBS (linear blending skinning) and DQS (dual-quaternion skinning) methods. The deformation network is optimized under the supervision of photometric loss from reference video frames, novel-view SDS loss and geometric regularization terms.Extensive experiments are conducted and demonstrate that our method can generate high-fidelity dynamic textured mesh from monocular video, and significantly outperforms previous works both quantitatively and qualitatively, establishing new benchmark in the field of video-to-4D generation. As shown in Fig. 1 , our generated assets can be directly simulated in modern 3D engines, showcasing its potential in the 3D gaming and film industries. 
related work: 
 3D Generation Since the introduction of score distillation sampling (SDS) by DreamFusion [33] , subsequent works [24, 5, 57, 35, 48, 49, 23] have significantly improved the performance of optimization-based 3D generation algorithms. Many works adopt a multi-stage optimization strategy [24, 5, 57, 48] to enhance generated appearance. Another line of research [26, 44, 27, 19] focuses on training multi-view diffusion models to inject multi-view supervision into SDS loss for addressing the Janus problem. DreamGaussian [49] and GaussianDreamer [64] pioneer the usage of 3D Gaussian [18] as the underlying representation and achieve 3D content generation in minutes.Although these methods demonstrate the potential of 3D Gaussian in 3D content generation, obtaining an object with high-quality geometry is still quite challenging. In this work, we explore to employ a Gaussian-mesh hybrid representation [9] in our 4D generation tasks for better modeling of both object surface geometry and texture.4D Representations Dynamic 3D representations form the foundation of 4D reconstruction and generation tasks. Most current methods extend static NeRF [30] to dynamic scenarios. These approaches, such as deformable [32, 34, 51, 59 ] and time-varying [4, 7, 8, 6] NeRF-based methods, are prevalent. There are also some works trying to model dynamic with 4D neural implicit surface [56, 14] . However, these implicit representations suffer from long optimization time and low reconstruction quality due to its computationally expensive volume rendering and implicit representation. Recent interest in 4D tasks has focused on 3D Gaussian representations due to their fast rendering speed and explicit nature. Some works [63, 58, 22] train networks to predict Gaussian kernel deformations, while others model kernel motion via polynomial representation or per-frame optimization [28, 21] . Besides, both SC-GS [11] and HiFi4G [13] employ sparse control points for Gaussian kernel deformation, with SC-GS using LBS and HiFi4G using DQS to drive Gaussians motion, thus ensuring spatial-temporal consistency. In this work, we propose to deform the object through a novel geometric skinning method, handling the artifacts associated with both LBS and DQS.4D Generation Although great progress has been achieved in 3D generation tasks, 4D generation is still challenging due to its requirement on additional temporal supervision. Since current pre-trained video diffusion models [3, 54] still struggle to generate high-quality video contents, it is challenging to distill useful motion knowledge via SDS optimization. Hence, the performance of existing text-to-4D [45, 1, 25] and image-to-4D methods [69] usually struggle with low appearance quality. Another line of works focus on video-to-4D generation [12, 38, 65] . These methods leverage current multiview diffusion models [26, 43, 27] to calculate the SDS loss [38, 12, 60] or generate per-frame multi-view images [66, 62] as supervision signal. Among them, a concurrent work of our method, SC4D [60] , optimizes a set of sparse-controlled dynamic 3D Gaussians by per-frame SDS loss from Zero123 [26] with a coarse-to-fine strategy. However, the issue of unsatisfying surface quality caused by 3D Gaussian-based representation still exists. In contrast, our method is grounding on a Gaussian-mesh hybrid representation [9] , enhancing the reconstruction quality both in texture and geometry.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Please refer to section 4.4 in our main paper.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 In this section, we first introduce the relevant preliminaries in section 3.1. Then we illustrate the details of DreamMesh4D which is divided into static stage and dynamic stage in section 3.2 and 3.3 respectively. The overview of our method is presented in Fig. 2 .Geometric Skinning Algorithms Given a mesh with N v vertices, M = {V, F} where V = {v i |v i ∈ R 3 }, i ∈ {1, 2, ..., N v } is the set of vertices and F represents the triangle faces. In geometric skinning algorithms, there are also some sparse control nodes (bones/skeletons) P = {p i |p i ∈ R 3 }, i ∈ {1, 2, ..., N p }, where N p is the number of control nodes. For a mesh vertex v ∈ V, its deformation is calculated by blending a number of neighboring control nodes N (v) through skinning algorithms. The local deformation for a control node p ∈ N (v) can be decomposed into a deformation matrix F p ∈ R 3×3 and a translation vector t p ∈ R 3 , and the deformation matrix can be further decomposed into a rotation matrix R p ∈ R 3×3 and a shear matrix S p ∈ R 3×3 using polar decomposition. The strength of the influence of node p to vertex v can be represented as w p and p∈N (v) w p = 1. Linear blending skinning (LBS) [47] computes the deformation of vertex v by linearly blending the influence of nodes in N (v):EQUATIONLBS is widely used due to its simple formulation and natural animations. However, it suffers from the well-known "volume loss" or "candy wrapper" artifacts under complex deformation. As an enhancement, dual-quaternion skinning (DQS) [16] represents the transformation of node p with a unit dual-quaternion ζ p = DQ(R p , t p ). Then the deformation of v can be calculated with DQS by:EQUATIONwhere ζ * represents the conjugate of ζ. DQS can eliminate the artifacts associated with LBS, but cannot handle non-rigid deformation since the strain effect is not considered. It also suffers from an undesirable joint-bulging artifact while blending, which requires artistic manual work to fix [40] .3D Gaussians and SuGaR Gaussian Splatting [18] represents the scene as a collection of 3D Gaussians, where each Gaussian g is characterized by its center µ g ∈ R 3 and covarianceΣ g ∈ R 3×3 .The covariance Σ g is parameterized by a scaling factor s g ∈ R 3 and a rotation matrix represented via a unit quaternion q g ∈ R 4 . Additionally, each Gaussian maintains opacity α g ∈ R and color features c g ∈ R C for rendering via splatting. Typically, color features are represented using spherical harmonics to model view-dependent effects. During rendering, the 3D Gaussians are projected onto the 2D image plane as 2D Gaussians, and color values are computed through alpha composition of these 2D Gaussians in front-to-back depth order. While the vanilla Gaussian Splatting representation may not perform well in geometry modeling, SuGaR [9] introduces several regularization terms to enforce flatness and alignment of the 3D Gaussians with the object surface. This facilitates extraction of a mesh from the Gaussians through Poisson reconstruction [17] . Furthermore, SuGaR offers a hybrid representation by binding Gaussians to mesh faces. A SuGaR mesh can be represented as S = {V, F, G} where G denotes all surface Gaussians. For a triangle face f ∈ F, the Gaussians G(f ) bound on f are defined with barycentric coordinates. This hybrid representation allows joint optimization of texture and geometry through backpropagation.The purpose of the static stage is to generate a refined Gaussian splats bounded triangular mesh. This procedure starts with a coarse mesh generated from a reference image I * that is sampled from all video frames I. Although there exists several fast mesh generation methods [68, 61] , we refer to Zero123-based SDS optimization for its stability. In particular, we conduct simple SDS training on a set of randomly initialized 3D Gaussians for a fixed number of steps with regular densification and pruning. After that, we stop densification and pruning, and include SuGaR's regularization terms [9] to make Gaussians aligned to object surface. Finally all Gaussians with opacity lower than a threshold σ = 0.5 are pruned, after which Poisson reconstruction [17] is performed to extract a coarse mesh.On the generated coarse mesh, we attach x = 6 new flat Gaussians to every triangle face. For each training step, we render RGB image Î * and mask M * under reference view to calculate RGB lossL s ref = || Î * -I * || 2 2 and mask loss L s mask = || M * -M * || 2 2 .The SDS loss L s SDS based on Zero123 [26] is also computed under randomly sampled views. The total loss for static SuGaR optimization is:EQUATIONwhere λ s SDS , λ s ref and λ s mask are the weights for different loss terms in static stage. In static stage shown in top left part, a reference image is picked from the input video from with we generate a Gaussian-mesh hybrid representation through a image-to-3D pipeline. As for dynamic stage, we build a deformation graph between mesh vertices and sparse control nodes, and then the mesh and surface Gaussians are deformed by fusing the deformation of control nodes predicted by a MLP through a novel adaptive hybrid skinning algorithm.In this section, we are going to delve into the deformation procedure for the Gaussian-mesh hybrid representation. First, we will discuss the construction of the deformation graph on the surface of the refined mesh. Then, we will explain the deformation flow, which progresses from the sparse control nodes to the mesh vertices, and ultimately to the surface Gaussians. We will break down each step to give a clear picture of the entire process.We begin by uniformly sampling N node sparse points on the surface of the refined mesh to serve as control nodes. To establish connections between the mesh vertices and the sparse control nodes, instead of using simple Euclidean distance to locate the nearest nodes (KNN), we pick N neighbor nodes with the shortest geodesic distance (as indicated by GeoDist in Fig. 2 ) based on the mesh's topology. This ensures that inappropriate connections between disparate mesh regions are avoided. Then, for a vertex v, the influence w p of each neighboring control node p ∈ N (v) is calculated following [47] :EQUATIONwhere d max is the distance to the (N neighbor + 1)-nearest node.Given that the object's texture was refined in the previous static stage, we fix the Gaussians' appearance properties (color and opacity) and focus the dynamic learning phase solely on spatial properties (positions, rotations, scalings). Initially, the local deformations of the control nodes are predicted by a deformation network Ψ. The updated spatial properties post-deformation are subsequently computed by integrating the local deformations of the control nodes through geometric skinning. In particular, given a control node p ∈ P and timestamp τ , the deformation network predict its local deformation at τ following the equation below. Note that we omit the subscript "τ " is omitted here and in rest paragraphs for simplicity.EQUATIONwhere R p , S p ∈ R 3×3 and t p ∈ R 3 are the rotation, shear matrix and translation respectively. Furthermore, to mitigate artifacts associated with LBS and DQS, we propose an adaptive fusion of their effects to calculate the deformation of mesh vertices. Here, η p ∈ R denotes the local rigid strength, indicating the extent to which the region around p is influenced by DQS at time τ .The predicted shear matrix S p for node p is intended only for LBS, whose strength is weaken by the predicted factor η p , and the final shear matrix at this location is computed as:EQUATIONwhere I ∈ R 3×3 represents an identity matrix indicating no strain effect. Afterwards, the new position of node p at timestamp τ is:EQUATIONDeformation of Mesh Vertices For the deformation of a specific vertex v using hybrid skinning, the new vertex position calculated with LBS, ṽlbs , and that with DQS, ṽdqs , can be obtained following Eq.1 and Eq.2 respectively. The local rigid strength at v can be computed by linearly blending that of neighboring nodes:EQUATIONthen the fused position of v after deformation is the interpolation between ṽlbs and ṽdqs :EQUATIONThe local rotation and strain at v are the corresponding linear blending from neighboring control nodes:EQUATIONEQUATIONDeformation of Surface Gaussians Now we have obtained deformations on the level of mesh vertices, afterwards the deformation on each Gaussian kernel will be calculated. Given a Gaussian kernel g ∈ G(f ) on a triangle face f = (v a , v b , v c ) ∈ F, its new center at timestamp τ is computed as:EQUATIONwhere (π a , π b , π c ) is the Gaussian's barycentric coordinate relative to the three triangle vertices. The new rotation is calculated by applying the local rotation ∆q g fused from related vertices to its original rotation q g :EQUATIONqg = ∆q g • q g . ( 14) We further apply the local shear matrix ∆S g fused from all three vertices to the original Gaussian scaling s g to obtain the new scaling:EQUATIONEQUATION)Training Losses After obtaining the deformed hybrid mesh at timestamp τ , we render its RGB Î * τ and alpha M * τ under reference view. We then compute the reconstruction lossL ref = || Î * τ -I * τ || 2 2and mask lossL mask = || M * τ -M * τ || 2 2, where I * τ and M * τ are the ground truth image and mask of input video at timestamp τ . For supervision under other views, we calculate SDS loss L SDS based on Zero123 [26] under randomly sampled views. Furthermore, our mesh-based representation naturally facilitates the introduction of local rigid constraints by leveraging the topology of the mesh. Specifically, the as-rigid-as-possible (ARAP) loss [46] is computed as:EQUATIONFigure 3 : Qualitative comparison with baselines. We compare our method with 4 previous videoto-4D methods. The first row provides two ground truth frames for each case. For each compared method, we render each case under reference view and another novel view at the two timestamps. The result demonstrates that our method is able to generate sharper 4D content with rich details, especially for the novel views. Please zoom in for more details.where C(v) represents the one-ring neighbors of vertex v, and ω n (v) is the cotangent weight [29] between v and its n-th connected vertex v n . Furthermore, we also introduce the normal consistency loss L N C provided by PyTorch3D [37] on the deformed mesh to globally constrain the mesh surface. Hence, the overall objective for our dynamic stage is a weighted combination of the above loss terms:EQUATIONwhere λ SDS , λ ref , λ mask , λ ARAP and λ N C are strengths of different loss terms in dynamic optimization stage. 4 ExperimentsQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification: We have provided the link to our project page and code at the end of abstract.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiment setup: 
 Dataset: Our quantitative results are evaluated on the test dataset provided by Consistent4D [12] , which contains seven multi-view videos. Each video has one input view for scene generation and four testing views for evaluation. For qualitative evaluation, we curate a set of challenging videos from previous works [12] and those generated by the video diffusion model SVD [3] . Evaluation metrics:The per-frame LPIPS [67] score and the CLIP-score [36] are computed between the testing and rendered videos, with the final scores averaged over the four testing views. These two scores serve as image-level metric to assess the structural and semantic similarity between the rendered images and the ground truth. Furthermore, we compute the FID-VID [2] and FVD [52] as video-level metrics to evaluate the video temporal coherence. Note that we report PSNR and SSIM values only for the reference view, as pixel-and patch-wise similarities are too sensitive to reconstruction differences, making them unsuitable for evaluating novel views in our generation task. However, we find they are suitable for evaluating the method's ability of modeling sharp features in the reference view.Baselines: We compare our method with previous video-to-4D generation methods: Consistent4D [12] , DreamGaussian4D [38] , 4DGen [65] and STAG4D [66] . All the experiments of above baselines are conducted using the code from their official GitHub repository.Qualitative Comparison Fig. 3 shows qualitative results of our method compared to other baseline works. The results reveal that our method generates 4D objects with higher fidelity and more details under reference view. And our method also outperforms other works with better spatial-temporal consistency, demonstrating the effectiveness of our method. Please zoom in for more details.Quantitative Comparison Table 1 demonstrates superior performance of our method against other baseline works quantitatively. Specifically, our approach notably exceeds the existing state-of-the-art in all measured metrics. Our method excels in both PSNR and SSIM, indicating a high level of reconstruction accuracy. Furthermore, the FVD score is particularly impressive, being only half that of competing methods. We also achieve the lowest FID-VID score, suggesting a significant enhancement in video quality produced by our 4D generation technique. Finally, our method achieves the lowest LPIPS and highest CLIP scores, ensuring both high image realism and semantic consistency. Overall, the numerical data clearly demonstrate the superior capabilities of our method in translating video to 4D content.In this section, we conduct ablation study to analyse the impact of various components on the performance of our method. The components under consideration include: (a) the choice between Euclidean and geodesic distance (EucDist and GeoDist) when constructing deformation graph; (b) our proposed adaptive hybrid skinning algorithm; (c) the ARAP and normal consistency terms (geometric regularization terms); (d) the choice between vanilla 3D Gaussians [18] and Gaussianmesh representation for our base 3D representation.EucDist vs. GeoDist Fig. 4 (a) provides a comprehensive qualitative analysis of the choice between EucDist and GeoDist. When using EucDist, the vertices on the elephant are incorrectly connected to its body, resulting in significant deformation artifacts. Conversely, GeoDist correctly links the vertices to neighboring nodes, enabling smooth object motion.Adaptive Hybrid Skinning The upper two rows of Table 2 presents quantitative results when replacing our adaptive hybrid skinning with DQS and LBS respectively. Almost all metrics have a decrease compared to using our adaptive hybrid skinning, showcasing its robustness. We also provide a qualitative comparison on a dancing robot case in Fig. 4 (b). When LBS or DQS is used, there are artifacts on the robot's deformed elbow along with uneven surface. In contrast, the artifacts are eliminated and the surface becomes smooth when our adaptive hybrid skinning is used.The third and fourth rows of Table 2 present the scores of metrics when either the ARAP or normal consistency term, respectively, is omitted from Equation 18.Decreases are observed across all metrics compared to full loss terms and it drops significantly when disabling ARAP loss. This circumstance matches the qualitative analysis shown in Fig. 4(c ). Without ARAP term, it appears serious distortions on the object geometry. When the normal consistency term is disabled, the object surface becomes less smooth, and consequently, the texture is impaired.3D Gaussians vs. Gaussian-mesh Hybrid In Fig. 5 , we provide the qualitative comparison between 3D Gaussians and the Gaussian-mesh hybrid representation for our base 3D representation. It shows that when utilizing 3D Gaussians, the texture of generated objects is blurry on those parts unseen in reference image. As a contrary, Gaussian-mesh hybrid representation presents clean and high-quality texture under every view, which benefits from the sharp surface provided by mesh.Despite the superior results achieved, our work still exist several limitations. In particular, our method relies on a pre-trained multi-view diffusion model (Zero123) for novel view supervision through SDS, leading to long optimization time and limited performance on which case Zero123 cannot handle well. Moreover, our method is currently only designed to generate 4D contents at object level from input Figure 5 : Qualitative comparison on 3D representation between 3D Gaussians and Gaussianmesh hybrid representation. When utilizing 3D Gaussians as our base 3D representation, the texture is blurry on the parts unseen in reference image. As a comparison, the texture is clean and of high quality under every view when employing the Gaussian-mesh hybrid representation.videos captured under fixed viewpoint. The extension of our framework to scene-level generation or to videos captured with a moving camera remains an area for future exploration. Finally, due to the scarcity of test data, the performance of our method on more complex task is not evaluated. These identified limitations will be addressed in our future research endeavors. 
conclusion: 
 In this work, we introduce DreamMesh4D, an innovative video-to-4D framework that generates dynamic meshes through a static-to-dynamic optimization process. By employing a Gaussian-mesh hybrid representation, we simultaneously refine both the geometry and texture of the object. This approach allows the static object to serve as an excellent starting point for dynamic learning. During the dynamic stage, we construct a deformation graph on the object's surface using geodesic distance.Thereafter, the motion of the entire mesh, as well as the surface Gaussians, are driven by sparse control nodes via a novel geometric skinning algorithm named adaptive hybrid skinning. It benefits from the strengths of both Linear Blending Skinning (LBS) and Dual-Quaternion Skinning (DQS), enabling more robust deformation. Extensive experiments have demonstrated the superior performance of our method in generating high-fidelity 4D objects. It significantly surpasses previous methods in both rendering quality and spatial-temporal consistency, establishing a new benchmark for video-to-4D tasks. While our method benefits a lot from the mesh-based representation, it reveals a promising direction in the field of video-to-4D generation. Furthermore, our method's compatibility with modern geometric pipelines showcases its potential applicability in the 3D gaming and film industries. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Descibed in section A.1 of appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The quantitative results are not accompanied by error bars, confidence intervals, nor statistical significance tests.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: Descibed in section A.1 of appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: As described in abstract and introduction part, our method is compatible with modern geometric pipelines, showcasing its potential in the 3D gaming and film industry.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This work poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The URL, citations and licenses of open-sourced assets used in this work are provided in Table 3 of appendix.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Static In the process of both coarse mesh generation and SuGaR refinement, the loss in Equation 3 is utilized for supervision, with the strength of different terms as λ s SDS = 0.1, λ s ref = 1000 and λ s mask = 500. For the generation of coarse mesh, a set of randomly initialized 3D Gaussians are optimized for 3000 steps in total. In the former 1500 steps, we do densification and pruning every 100 steps. After the 1500 steps, densification and pruning are stopped, and we introduce additional opacity binary and density regularization terms as described in [9] into optimization until step 3000. Finally, we prune all Gaussians with opacity less than 0.5 and extract coarse mesh through Poisson reconstruction. Afterwards, we bind x = 6 new flat Gaussians to each triangle face of the coarse mesh, and do optimization for 2000 steps.In dynamic stage, we defaultly sample N node = 1024 control nodes and assign N neighbor = 4 neighboring nodes for each vertex when constructing deformation graph. For each training step, 8 frames are randomly sampled from the input video for supervision. And for each sampled timestamp, we randomly sample 2 views for the calculation of SDS loss. All images are rendered at resolution 512 × 512 with white background. The camera distance to world center is fixed as 3.8 and the degree of field-of-view (FoV) is fixed as 20 • . As for the strenghts of different loss terms, we defautly set λ SDS = 0.1, λ ref = 5000, λ mask = 500 and λ N C = 10. The value of λ ARAP is chosen case-specifically in [1, 10] according to the motion amplitude of object. The deformation network is zero-ly initialized and totally optimized for 2000 steps with learning rate as 0.00032. All of our experiments are conducted on a single NVIDIA RTX 4090 GPU.Licenses Here we provide the URL, citations, and licenses of open-sourced assets used in this work in Table 3 . NeurIPS Paper Checklist 1.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: We describe our proposed algorithm in section 3 in detail, and the extensive experiment results demonstrate the claimed superior performance compared to previous methods.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: No justification.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: This work introduces a novel framework for transforming static images into 4D dynamic meshes, DreamMesh4D. It refines object geometry and texture using a Gaussian-mesh hybrid and a geodesic-based deformation graph. A new skinning algorithm combines the best of LBS and DQS for enhanced deformation. The method excels in creating high-quality 4D objects with improved rendering and consistency.

The work addresses a challenging problem to generate dynamic objects from a monocular video; the method to generate dynamic meshes through a static-to-dynamic optimization process is novel, and the losses terms make sense and are intuitive; It was also informative that the SuGaR SDS would work well in a complex pipeline of 4D task; the usage of mesh and skinning has good compatibility with modern computer graphic pipelines in the 3D gaming and film industries; the experimental results show the superior performance of the proposed method in generating high-fidelity 4D objects compared to previous methods;
the paper is clearly written, very informative, well contextualized, in good structure and has a good track of the prior methods. The rebuttal addresses all the reviewers' concerns. 

Overall, the Gaussian-mesh representation for 4D tasks is inspiring and will lead further developments along this direction.

Reviews: review: paper_summary: paper_summary: This paper proposes DreamMesh4D that combines mesh representation with sparse-controlled deformation technique to generate high-quality 4D objects from a monocular video. The authors bind Gaussian splats to the surface of the triangular mesh for differentiable optimization of both the texture and mesh vertices. The method begins with a coarse mesh from single image based 3D generation. Sparse points are then uniformly sampled on surface of the mesh to build a deformation graph, which drives the motion of the 3D object. For each step, transformations of sparse control points are predicted using a deformation network. The mesh vertices and the bound Gaussians are deformed via a geometric skinning algorithm that combines LBS and DQS. Reference view photometric loss, score distillation loss, and regularization losses are used in a two-stage learning. Experiments are performed on the consistent4D dataset.
strength_weakness: strength_weakness: - One of the main idea of this paper is to use a Gaussian-mesh hybrid representation, which was originally proposed in the work of SuGaR. The technical difference that is designed to address the special setting of this paper’s task is not clearly explained.
- In figure 1, the input is a composited scene video. But in the rest of the paper, the inputs are well segmented object video, which makes the setting of the addressed task confusing.
- The method combines some existing methods such as SuGaR, deformation graph, LBS and DQS based skinning. It’s not clear to identify the author’s innovative contribution.
questions: questions: 1. One of the main idea of this paper is to use a Gaussian-mesh hybrid representation, what is the difference between the proposed method and SuGaR?
2. In figure 1, the input is a composited scene video. Is a segmentation method needed to extract each object region to use as input to the method? There seems no mention on this in the rest of the paper. How is the mutual-occlusion between the objects solved by the proposed method for the scenario in figure 1?
3. What is the computational cost of the proposed method?
limitations: limitations: Not applicable.
paper_summary: paper_summary: DreamMesh4D is a novel framework for transforming static images into 4D dynamic meshes. It refines object geometry and texture using a Gaussian-mesh hybrid and a geodesic-based deformation graph. A new skinning algorithm combines the best of LBS and DQS for enhanced deformation. The method excels in creating high-quality 4D objects with improved rendering and consistency.
strength_weakness: strength_weakness: 1.The novelty of this work is somewhat limited in comparison to previous methods. The training strategy, the structure of the loss function, and the design of deformation are quite similar. The main difference lies in the representation, which is a predictable technical combination.

2.I find that the detail of the results is superior to those of previous methods. Therefore, in section 4.2, the author should elucidate which aspects of the pipeline design have contributed to the improved quality compared to the previous method's design.

3.The paper argues that previous works have significant time and memory complexity, but there is a lack of comparison regarding time and GPU usage costs. It would be beneficial to include such a comparison to demonstrate the efficiency of the proposed method.
questions: questions: The current article lacks interesting insights. I suggest that the author should focus more on the advantages of this representation and explain why it is important for the 4D generation task.
limitations: limitations: Refer to the Weaknesses and Questions section
paper_summary: paper_summary: The paper proposes a framework to generate animated surfaces from input videos. It has a static step to generate a base triangle mesh, and textures represented by 3D Gaussians. A subsequent dynamic step changes the positions of the mesh vertices and the gaussians. The framework leverages on Zero123 and SuGaR to generate the mesh and gaussians, integrating loss terms from those approaches in its training mechanism. Tha dynamic step uses geodesic distances to create a deformation graph and include As-Rigid-As-Possible and Normal Consistency loss terms.
strength_weakness: strength_weakness: (1) I missed a video in the supplementary material. As a 4D approach, the paper would benefit greatly from showing one. It shows metrics about temporal coherence, but a video would avoid any possible doubt.

(2) I'm missing references about 4D neural implicit surfaces in the 4D Representations paragraph in the Related Work section. They also deal with dynamic surfaces in neural pipelines. I believe a sentence including all of them would be sufficient.

Novello, Tiago, et al. "Neural Implicit Surface Evolution." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

Guandao Yang, Serge Belongie, Bharath Hariharan, and Vladlen Koltun. Geometry processing with neural fields. Advances in Neural Information Processing Systems, 34, 2021.

Mehta, Ishit, Manmohan Chandraker, and Ravi Ramamoorthi. "A level set theory for neural implicit evolution under explicit flows." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

(3) The mathematical notation is a little bit polluted, impacting reading in my opinion. Subscripts and superscripts are used too much. The text would be cleaner by using more symbol letters instead. A good rule of thumb is to avoid using subscripts and superscripts unless they represent indices. 

The same letters are also used for very different contexts. N is used for number of vertices and control point set, for example. Even though different fonts are used in each context, the cognition when reading is to remember N and forget the font used. It would be cleaner to use different symbols.
questions: questions: (1) I would like to know more about the temporal coherence in the proposed approach, since a video is not available. Even though the paper show metrics in that respect I would also like to see qualitative results.

(2) Even though the results are probably ready for a CG pipeline, I believe the training and inference must be computationally expensive. I would like to know the training and inference times, and the hardware used.

(3) Does only changing positions, rotations, and scaling of the gaussians in the deformation step result in accurate view-dependent effects? In theory the spherical harmonics should also be recalculated.

(4) Why 6 gaussians per triangle in the static stage?

(5) Equation 10 should come with a small explanation to give intuition. Mentioning that the logarithmic rotation is a mapping to the Lie Algebra and that the exp is a conversion back to the rotation matrix form would at least give a direction for a reader that is not a specialist.
 
(6) I believe the references in line 38 [29, 54, 17, 41, 9] are not associated directly with score distillation sampling. Is that a typo?
limitations: limitations: Yes, the paper has a specific section describing limitations.

Comments: comment: We are pleased to hear that our rebuttal addressed your concerns well, and we are grateful for your efforts and time in reviewing. If you have any further concerns, questions, or suggestions, please do not hesitate to let us know.
Thank you for your response and support! We sincerely appreciate your efforts and time in reviewing and providing incisive comments on our paper.
I appreciate the authors for providing the detailed explanations. For my question 2 for Figure 1, as I guessed, it is a demo created by the authors using their generated assets. I think this should be clearly explained in final version of the paper, because as the current Figure 1, it is easy to misunderstand that the composited scene demo is the input video. For my concerns about the innovation of the paper, the authors explained what are new compared to a straight-forward combination of SuGaR, deformation graph, LBS and DQS. The clarifications are reasonable and are meaningful to work with a pipeline to yield dynamic mesh with good compatibility with modern graphics. In regards of the above, I'd tend to keep my previous positive rating.
Thanks you for the clear answers and I am happy to increase to a positive rating. Based on the novelty and impact of the technology, I can only give it a borderline accept.
Thanks for the reviewer's efforts and valuable comments. 

### **Q: Missing orbital view videos**

**A**: We originally built the anonymous project page following a prior work, i.e. Consistent4D (ICLR2024), that shows the reference view for the demonstration of reconsturction quality and a novel view for that of spatial consistency. And we admit that it would be better to present more novel views in a nice page arrangement. We will improve it.

As a supplement, we have also provided the video of our composite scene demo in the same project page (i.e. at bottom). The demo video contains surrounding views of the generated dynamic objects, demenstrating their high spatial consistency, as also agreed by reviewer *2HE6*.

### **Q: Similar with 4DGS (CVPR2024) on Gaussian attributes deformation**

**A**: We carefully checked the implementation of 4DGS (CVPR2024). It takes a different approach as ours. In particular, they use a deformation network to directly predict the delta update of each Gaussian's attributes (i.e. position, rotation, scaling etc.). In contrast, we predict the deformation gradients (i.e. transformation matrix) of the control nodes in the deformation graph via an MLP network, the Gaussian attributes are then computed from the predicted node deformation via skinning algorithm in an analytical manner. 

### **Q: Utilizing of existing 3D representation technology**

**A**: The authors humbly think the main contributions of the work can be summarized to: 1) the analysis and findings of which prevent existing methods from delivering better performance, as discussed in the reply for "Q4: Insights of this work"; 2) based on these analysis and insights, we for the first time demonstrate the possibility and superior advantages of extending Gaussian-mesh representation for 4D tasks, in comparison to prior methods. 

Although Gaussian-mesh hybrid representation is an existing technology for static scene, no one has really tried to extend it for 4D representation. There is a fact should not be ignored that almost all 4D representations are developed from existing 3D representations. The authors humbly think *a new use of an old method can be novel if nobody ever thought to use it this way for a new problem, and it really can deliver superior performance*.
1. I also missed the video results like Reviewer 2HE6，because I previously believed that links were not allowed. Though I think the video details are good. However, I am concerned about why the orbital view videos are not shown, as I can't judge the spatial consistency.

2.My concerns are quite similar to those of Reviewer 7V8W. In Q3: Innovative Contribution of this work under 7V8W, I find the author summarized two contributions. Regarding the changing of Gaussian attributes during deformation, I think it is a basic strategy. In 4DGS (CVPR 2024), the positions, rotations, and scalings are also deformed. Regarding the new skinning algorithm, in the ablation study in the main paper, the evaluation metric did not significantly exceed DQS and LBS. So it is not the key. In the rebuttal PDF, I think the ablation about GS and hyper is important.I admit that the representation's blur problems are mitigated, and I believe the improvement contributes to the changes in representation, but the technology is existing. Therefore, I think it is a not bad paper, but not an excellent one. So I do not rate it above "boardline accept."
The authors are grateful for your support! We will further polish our method in the final version.
Thank you for the responses. I missed the link for the result videos at the appendix, and I believe they are good enough. There are minor artifacts in the normals, but my concerns about temporal coherence are addressed. I think the paper is very well written and deserves publication, specially for the good contextualization, clean presentation of the theory, and the coherent mathematical treatment of the problem. It would be a very good reference for someone not so familiar with the field.