Sync4D: Video Guided Controllable Dynamics for Physics-Based 4D Generation

## Abstract

Abstract In this work, we introduce a novel approach for creating controllable dynamics in 3D-generated Gaussians using casually captured reference videos. Our method transfers the motion of objects from reference videos to a variety of generated 3D Gaussians across different categories, ensuring precise and customizable motion transfer. We achieve this by employing blend skinning-based non-parametric shape reconstruction to extract the shape and motion of reference objects. This process involves segmenting the reference objects into motion-related parts based on skinning weights and establishing shape correspondences with generated target shapes. To address shape and temporal inconsistencies prevalent in existing methods, we integrate physical simulation, driving the target shapes with matched motion. This integration is optimized through a displacement loss to ensure reliable and genuine dynamics. Our approach supports diverse reference inputs, including humans, quadrupeds, and articulated objects, and can generate dynamics of arbitrary length, providing enhanced fidelity and applicability. Unlike methods heavily reliant on diffusion video generation models, our technique offers specific and high-quality motion transfer, maintaining both shape integrity and temporal consistency.

## 1 Introduction

The introduction of large-scale [, ] diffusion-based generative models has sparked a revolution in creative and high-quality image synthesis, which has been successfully extended to 3D generation [, , , , , , , , , ] and further evolved into video generation [, , ], laying the groundwork for dynamic 3D content or 4D generation. This technological convergence enhances various applications, from virtual reality to simulation training, by significantly boosting the realism and interactivity of virtual environments.

However, despite these technological strides, existing methodologies still face significant limitations. Current implementations, utilizing Score Distillation Sampling (SDS) [] as seen in [, , , , ], aim to distill motion priors from video diffusion models to facilitate dynamic 3D creation. However, this often leads to inaccurate motion representations. Alternatively, methods like those documented in [, ] directly use the per-frame outputs from video diffusion models as references. While faster and more straightforward, this approach still fails to adequately address issues of movement irrationality and shape incoherence in the generated outputs. The effectiveness of both approaches is inherently limited by the capabilities of the pretrained video diffusion models they adopted. Therefore, the generation quality of the dynamic and geometry quality frequently suffers from inconsistencies and poor geometric integrity. Moreover, these methods lack precise motion control, typically relying on vague text prompts to guide motions, which further compromises the fidelity and applicability of the generated content.

Significant advancements have also been made in dynamics representation, particularly in integrating physical properties into dynamic models. The introduction of PhysGaussian [], which utilizes a novel style of 3D Gaussians representation from Kerbl et al.[], has facilitated high-quality motion synthesis. Zhang et al.[] pioneered the integration of dynamic generation model with physical simulation techniques [, ], marking a crucial step forward in this domain. Incorporating physical simulation produces more reliable and genuine dynamics on 3D Gaussian representations. However, these methods require hand-crafted input motions, which are also limited to a narrow range of actions and relatively simple scenarios.

In this work, we introduce a novel approach for creating controllable dynamics in generated 3D Gaussians guided by casually captured reference videos. As shown in Figure 1, our method transfers the motion of an object from the reference video to various generated 3D Gaussians across different categories. To achieve this, we first apply blend skinning-based non-parametric shape reconstruction to extract the shape and motion of the reference object from the video. This process allows the decomposition of the reference object into motion-related parts based on skinning weights. Next, we establish shape correspondences between the reference shape and the generated target shapes utilizing pretrained 2D diffusion models and 3D point cloud models. Finally, we map the motion-related parts to the corresponding target shapes, enabling the matched parts in the target shapes to inherit the motion from the reference object parts.

To tackle the shape and temporal inconsistency issue that widely appears in existing works, instead of the commonly used point-wise deformation, we drive the target shapes with the matched motion using Material Point Method (MPM) physical simulation [, , ]. However, due to the shape variation in target objects, directly providing the reference motion as input on each part to the physical simulation model may not produce the desired outputs and may suffer from cumulative errors. Therefore, we model a delta velocity field to adjust the input motion adopted from the reference, which is optimized by a displacement loss between two object spaces.

In summary, our contributions are as follows:

- •
We introduce a novel method that transfers motion from casually captured videos to various 3D-generated Gaussians, ensuring precise and customizable dynamics across different categories.
- •
Our technique employs shape reconstruction to extract shape and motion from reference objects. We segment the reference objects into motion-related parts based on skinning weights and map the parts to generated target shapes by establishing shape correspondences.
- •
We integrate physical simulation to drive target shapes with matched motion to ensure shape integrity and temporal consistency. Our approach further ensures reliable and genuine dynamics by introducing a displacement loss to optimize physical signals, avoiding cumulative errors.
- •
Our method supports diverse reference inputs, including humans, quadrupeds, and articulated objects. Unlike existing methods reliant on diffusion video generation models, our approach generates dynamics specific to the reference input and can be of arbitrary length.

## 2 Related Works

### 2.1 4D Generation

Dynamic generation seeks to create robust and persistent 3D representations that excel in virtual environments like gaming, animation, and virtual reality. Initiatives commonly begin with a text prompt specifying the 3D object and its motions [, , ]. Zhao et al. [] adopt a different strategy, using an image prompt, which offers greater versatility over the 3D object’s representation. Meanwhile, Yin et al. [] and Ren et al. [] utilize videos generated from video diffusion models as direct references, indicating that controlling motions through video input holds promise. However, these approaches face challenges, including constrained motion expression, discrepancies between the input text and the resulting motions, and poor generation results.

### 2.2 Shape and Motion Reconstruction from Videos

Dynamics reconstruction from video footage is a prolonged and challenging endeavor, and reconstructing from monocular video poses an even greater difficulty. A commonly employed approach [, , , , , , , ] involves utilizing a deformation field [] to enhance the neural radiance field [] while concurrently implementing various techniques to ensure high-quality reconstruction. While these works mostly rely on multi-view datasets, Yang et al.[, , , ] focus on reconstructing shapes from casual videos, achieving remarkable progress in the area. As 3D Gaussian Splatting proved to be an efficient and effective approach for reconstructing tasks, several works [, , , , , , ] are adapted to dynamics reconstruction, achieving promising results.

### 2.3 Motion Transfer

A common perspective on attaining reliable motion is to derive it from a real video and transfer it to another object. This can be achieved by estimating poses frame-by-frame and subsequently transferring these poses. However, these works [, , , ] fundamentally rely on correspondences between the same category of objects. An alternative approach [, ] to motion transfer based on the diffusion model has garnered popularity in the video domain. These methods can transfer motions between different types of objects. However, the quality of the results significantly falls short of the requirements for 3D and 4D generation, considering the inconsistency and vagueness of the video.

Figure: Figure 2: Overview of Sync4D: Sync4D processes a reference video to derive a canonical shape and a bone-based motion sequence through reconstruction techniques. Meanwhile, given a text prompt or image prompt, we generate a 3D Gaussian object through diffusion models. The framework matches motion-related parts from the reconstructed shape to the generated shape and transfers the motion. This motion information is then initialized into the velocity physical signals. We employ a triplane representation to produce a delta velocity field to adjust physical signals. The velocity field for each part of the target is optimized using the differentiable Material Point Method (MPM) simulation. To ensure fidelity to the original, a displacement loss is designed to reduce cumulative errors and ensure plausible motions.
Refer to caption: x2.png

## 3 Method

We propose a framework capable of transferring motion from casually captured videos to generated static 3D objects, as illustrated in Figure 2. We begin by reconstructing the shape of the captured object from a video and extracting the motion information. In the subsequent stage, the reconstructed object will be matched with the target 3D Gaussian representation to achieve regional correspondence. Finally, we transfer the original motion to the corresponding target regions and utilize physics simulation to animate the 3D object. We optimize the velocity field in physics simulation by minimizing spatial displacement differences to enhance motion correctness, thereby achieving superior visual fidelity.

### 3.1 Preliminaries

Material Point Method (MPM) is a computational technique for simulating the behavior of continua. It uses a dual representation where material properties and state variables are stored on particles while computations and interactions are handled on a background computational grid. Following PhysGaussian [], we employ MPM simulation directly on Gaussian particles, discretizing the entire scene into a set of Lagrangian particles. At timestep $t$, each particle $p$ maintains its state variables, which include spatial position $\boldsymbol{x}^{t}_{p}$, velocity $\boldsymbol{v}^{t}_{p}$ and its material properties, including mass $\boldsymbol{m}^{t}_{p}$, deformation gradient $\boldsymbol{F}^{t}_{p}$, Kirchhoff stress $\boldsymbol{\tau}^{t}_{p}$, affine momentum $\boldsymbol{C}^{t}_{p}$.

MPM simulation process transfers data between particles and grid nodes at each simulation period $\boldsymbol{\Delta t}$, which can be delineated into three distinct steps. Firstly, we apply particle-to-grid to transfer momentum as follows:

$$ $\displaystyle\boldsymbol{m}^{t}_{i}$ $\displaystyle=\sum_{p}{N(\boldsymbol{x}_{i}-\boldsymbol{x}_{p}^{t})\boldsymbol {m}_{p}},$ (1) $\displaystyle\boldsymbol{m}^{t}_{i}\boldsymbol{v}^{t}_{i}$ $\displaystyle=\sum_{p}{N(\boldsymbol{x}_{i}-\boldsymbol{x}_{p}^{t})\boldsymbol {m}_{p}}(\boldsymbol{v}^{t}_{p}+\boldsymbol{C}^{t}_{p}(\boldsymbol{x}_{i}- \boldsymbol{x}_{p}^{t})).$ (2) $$

Here $\sum_{p}{N(\boldsymbol{x}_{i}-\boldsymbol{x}_{p}^{t})}$ is the B-spline kernel, and $\boldsymbol{v}^{t}_{i}$ is the updated velocity on grid node. Then we use grid transfer to get the next state grid velocity $\boldsymbol{v}^{t+1}_{i}$ as

$$ $\displaystyle\boldsymbol{v}^{t+1}_{i}=\boldsymbol{v}^{t}_{i}-\frac{\boldsymbol {\Delta t}}{\boldsymbol{m_{i}}}(\sum_{\boldsymbol{p}}{N(\boldsymbol{x}_{i}- \boldsymbol{x}_{p}^{t})\frac{4}{r^{2}}V_{p}^{0}\frac{\partial\psi}{\partial \boldsymbol{F}}\boldsymbol{F^{t}_{p}}(\boldsymbol{x}_{i}-\boldsymbol{x}_{p}^{t })}+g^{t}_{i}),$ (3) $$

where $r$ is the grid resolution, $V_{p}^{0}$ is the initial representing volume, $\psi$ is a strain energy density function related to Kirchhoff stress $\boldsymbol{\tau}^{t}_{p}$, $g^{t}_{i}$ is a possible external force.
Finally, we convert the grid velocity to particle velocity at timestep $t+1$, alongside transferring of particle positions:

$$ $\displaystyle\boldsymbol{v}_{p}^{t+1}=\sum_{\boldsymbol{i}}{N(\boldsymbol{x}_{ i}-\boldsymbol{x}_{p}^{t})}\boldsymbol{v}_{i}^{t+1},\quad\boldsymbol{x}_{p}^{t +1}=\boldsymbol{x}_{p}^{t}+\boldsymbol{\Delta t}\boldsymbol{v}_{p}^{t+1}.$ (4) $$

Since our work mainly focus on optimizing velocity field $\boldsymbol{v}(p,t)$, material properties $\boldsymbol{F}_{p}^{t}$, $\boldsymbol{\tau^{t}_{p}}$, and $\boldsymbol{C}_{p}^{t}$ update are not listed here. Please refer to Section A.1 for more information on the MPM simulation process.

### 3.2 Extracting Shape and Motion from Videos

To extract the shapes and motions of arbitrary objects from casual videos, we model the object with bones and neural blend skinning [] following several existing non-parametric reconstruction methods [, , , , ]. For a point $\mathbf{x}^{t}$ in three-dimensional space at time $t$, we aim to determine its equivalent point $\mathbf{x}^{*}$ within a canonical space. The model achieves the transition between $\mathbf{x}^{t}$ and $\mathbf{x}^{*}$ by incorporating the rigid transformations linked to the coordinates of bones in 3D. We define ${\bf G}^{t}\in SE(3)$ as the global transformation mapping the entire structure from the fixed frame to time $t$. We initialize the canonical bone center coordinates ${\bf B}^{*}\in\mathbb{R}^{B\times 3}$ and let ${\bf J}^{t}_{b}\in SE(3)$ indicate the relative rigid transformation adapting the $b$-th bone from its initial position ${\bf B}^{*}_{b}$ to its transformed state ${\bf B}^{t}_{b}$ at time $t$. These transformations can be described by the following relations:

$$ $\displaystyle\mathbf{x}^{t}$ $\displaystyle=\mathcal{W}^{t,\rightarrow}(\mathbf{x}^{*})=\mathbf{G}^{t} \mathbf{J}^{t,\rightarrow}\mathbf{x}^{*},$ (5) $\displaystyle\mathbf{x}^{*}$ $\displaystyle=\mathcal{W}^{t,\leftarrow}(\mathbf{x}^{t})=\mathbf{J}^{t, \leftarrow}({\mathbf{G}^{t}})^{-1}\mathbf{x}^{t},$ (6) $$

where $\mathcal{W}^{t,\rightarrow}$ and $\mathcal{W}^{t,\leftarrow}$ indicate forward and backward warping, $\mathbf{J}^{t,\rightarrow}$ and $\mathbf{J}^{t,\leftarrow}$ represent the weighted averages of $B$ rigid transformations $\{{\bf J}^{t}_{b}\}_{b\in\{1,\dots,B\}}$, mapping the bones from their default positions to their current configurations at time $t$. Since the primary aim of the reconstruction is to offer motion cues for the target objects, we configure the number of bones $B$, to be the minimum count of articulated segments required to accurately model the reference shape.

The skinning weights are defined as $\mathbf{W}=\{w_{1},...,w_{B}\}\in\mathbb{R}^{B}$. For any 3D point $\mathbf{x}$, the skinning weights are calculated using the Mahalanobis distance $d_{M}(\mathbf{x},{\bf B}^{t})$ between the point and the Gaussian-shaped bones under pose ${\bf B}^{t}$, as indicated in the equation:

$$ $\mathbf{W}=\mathrm{softmax}(d_{M}(\mathbf{x},{\bf B}^{t})+\mathbf{W}_{\Delta}).$ (7) $$

where $\mathbf{W}_{\Delta}$ is produced by a coordinate MLP to enhance the details. We optimize all the parameters following the framework of BANMo [].

### 3.3 Part Mapping with Shape Correspondence

To transfer the motion, we map the articulated parts from the reference shape to the target shape. We first extract the surface meshes of the shapes. We abuse the notation to define the vertices of the reference mesh and target mesh as ${\bf X}^{ref}\in\mathbb{R}^{N_{ref}\times 3}$ and ${\bf X}^{tar}\in\mathbb{R}^{N_{tar}\times 3}$. Inspired by Diff3F [], we utilize pretrained 2D diffusion models to obtain the 2D semantic features on multi-view renderings and back-project to 3D vertices to get $f_{diff}\in\mathbb{R}^{N\times 1024}$. However, solely using semantic features may not provide enough information, for example, it cannot distinguish the different limbs of humans and quadrupeds. Therefore, we adopt another geometry based pretrained 3D correspondence network [] to extract additional features $f_{geo}\in\mathbb{R}^{N\times 128}$, the resulting features on mesh surfaces are given by:

$$ $f^{ref}=f^{ref}_{diff}\|f^{ref}_{geo},\quad f^{tar}=f^{tar}_{diff}\|f^{tar}_{geo}$ (8) $$

Where $\|$ denotes concatenation. We segment the reference objects into $B$ articulated parts based on the optimized skinning weights. The part labels are noted as ${\bf Y}^{ref}\in\mathbb{R}^{N_{ref}}$, the label for vertex $n$ is obtained:

$$ $y^{ref}_{n}=\arg\max(\mathbf{W}(\mathbf{X}_{n}))$ (9) $$

Then, we calculated the mean feature for each part of the reference object:

$$ $\bar{f}^{ref}_{b}=\frac{1}{N_{b}}\sum_{n:y^{ref}_{n}=b}f_{n}^{ref}$ (10) $$

We derive the correspondence between each vertex in the target mesh and the reference part as:

$$ $y^{tar}_{n}=\arg\max_{b\in B}(\frac{\bar{f}^{ref}_{b}\cdot f^{tar}_{n}}{\|\bar {f}^{ref}_{b}\|\|f^{tar}_{n}\|})$ (11) $$

We further perform an outlier removal based on the distance to part centroids to get $\hat{y}^{tar}_{n}$. From the mapped surface points $\hat{y}^{tar}_{n}$, we can draw bounding boxes for each part and assign all the Gaussian points in the bounding boxes to the corresponding part. The relative motion for $b$-th part can be approximated as $\Delta\mathbf{B}^{t}_{b}=\mathbf{B}^{t+1}_{b}-\mathbf{B}^{t}_{b}$.

### 3.4 Physics-Integrated Motion Transfer

The process of motion transfer commences with the utilization of the reconstructed prior alongside the identified corresponding matching. This is achieved through the initialization of $\boldsymbol{v}$ at the onset of each simulation, guided by the motion sequence observed in reference space, broadly indicating the velocity direction. The initialized velocity for $b$-th part of target should be:

$$ $\displaystyle v_{0}^{t}=\hat{\upsilon^{t}}=\frac{\hat{\delta}^{t}}{\boldsymbol {N\Delta t}},\;\;\hat{\delta}^{t}=\mathbf{b}^{t+1}-\mathbf{b}^{t},$ (12) $$

where $\mathbf{b}$ represents $\mathbf{B}_{b}$. In this section, we drop $b$ in every notation for simplicity.

To better control the simulated motion and avoid cumulative errors, we employ a triplane representation [] accompanied by a three-layer MLP to adjust the velocity field. The network shares the same spatial information as the physics field, generating particle-level $\Delta v$ for each part of the object. The velocity field before simulation can then be set to:

$$ $\displaystyle\boldsymbol{v}^{t}\leftarrow v_{0}^{t}+\Delta v^{t}.$ (13) $$

Based on the given velocity states and other physics properties, we animate the 3D static generation with a differentiable MLS-MPM [] simulator. This process should be done between adjacent two frames, estimating one motion sequence, which can be formulated as follows:

$$ $\displaystyle\boldsymbol{x^{t+1},v^{t+1}=S(x^{t},v^{t},\theta,\Delta t,N),}$ (14) $$

where $\boldsymbol{x}^{t}$ denotes particle positions of $b$-th part at time $t$, and similarly $\boldsymbol{v}^{t}$ denotes the velocities of corresponding particles at time $t$. $\boldsymbol{\theta}$ denotes the collection of the physical properties of all particles: deformation gradient $\boldsymbol{F}^{t}$, gradient of local velocity fields $\boldsymbol{C}^{t}$, mass $\boldsymbol{m}$, Young’s modulus $\boldsymbol{E}$, Poisson’s ratio $\boldsymbol{\nu}$, and volume $\boldsymbol{V}$. $\Delta t$ is the simulation step size, and $N$ is the number of steps.

While the modification goal is to ensure that the resulting pose closely matches the reconstructed one, one approach to addressing this issue is to approximate the displacement in the target space to be consistent with the displacement in the reference space, considering the respective part sizes. With this as a reference, we optimize velocity field $\boldsymbol{v}$ for all parts by a per-frame loss function:

$$ $\displaystyle L^{t}_{x}=\sum_{b}{L_{1}(\delta^{t}_{b}-\frac{s_{t}}{s_{o}}\hat{ \delta^{t}_{b}})},$ (15) $$

where $s_{t},s_{o}$ is the coverage ratio for target space and reference space, respectively. To calculate the displacement $\delta$, we determine the positional difference between the part mass centroid of the initial state and the simulated end state, which is slightly divergent from the initialization of velocity.

Furthermore, we employ total variation regularization across all spatial planes to promote spatial continuity. Denoting $u$ as one of the 2D spatial planes and $u_{j,k}$ as a feature vector on the 2D plane, the total variation regularization term is formulated as:

$$ $\displaystyle L^{t}_{tv}=\sum_{j,k}\|u_{j+1,k}-u_{j,k}\|_{2}^{2}+\|u_{j,k+1}-u _{j,k}\|_{2}^{2}$ (16) $$

Rather than directly training the complete video motion, we utilize the motion between two frames as the training phase. Subsequently, after sufficient training in this phase, we advance to the next motion phase. This training methodology ensures that the dynamics’ posture is as accurate as possible after each motion sequence. After training the relative motion, we apply the global transformation ${\bf G}^{t}$ on the entire 3D Gaussians for each frame to get the final rendering.

## 4 Experiments

In this section, we demonstrate the versatility of our framework for generalized data and substantiate the reliability of the resulting motions.

### 4.1 Experimental Settings

Implementation details. For text-to-3D generation, we choose LucidDreamer [] as our model, while for image-to-3D generation, we choose LGM [] as our model. Our reconstruction model is implemented based on Lab4D [, ]. We set the number of bones $B=11$ for human, $B=13$ for quadrupeds and $B=2$ for laptops. For humans and quadrupeds, we provide an average initial bone center coordinates for faster training. For laptops, the bones are all initialized from the origin.
The Gaussian objects from two generative models are viewed as our simulation area, which has 1.5 to 2 million particles for LucidDreamer generation and 20 to 50 thousand
particles for LGM. Considering simulation consumption, we use a $41^{3}$ resolution grid to downsample LucidDreamer output, ensuring consistency with the LGM output by order of magnitude. We take the average coordinate of all particles within the same grid as our control point, where physical simulations are applied. Upon completion of the simulation, particles within the same grid point will share the same velocity field properties, ensuring the rigid body motion of the object.

For the optimization process, we utilize a triplane [, ] followed by a three-layer MLP, similar to PhysDreamer []. Although we did not optimize the material properties, in our experiments, they retain physical significance and are adjustable. Users can select Young’s modulus $E$ between $1\times 10^{3}$ and $1\times 10^{5}$, and the Poisson’s ratio $\nu$ between 0.1 and 0.5, based on the desired visual effects. A higher $E$ results in a more resilient object, while a higher $\nu$ leads to a stiffer object.

We train our task on a single NVIDIA RTX 6000 Ada machine. Our training process requires 7-8 NVIDIA RTX 6000 Ada GPU minutes per frame, with an approximate memory consumption of 24 GB.

Metrics. Our framework focuses on the realism and similarity between input video motion and generated motion. For evaluation, we conduct a user study listing our results and the other experimental results as a pair. Three questions are set for better evaluation: the overall generation quality of the dynamic scene, the motion similarity of the input video and the 4D generation, and the shape consistency of results. We conduct the evaluation on three pairs and recruit 34 participants to join the evaluation, getting a high score for all of the questions. Detailed experimental results can be referred at Section A.2

Figure: Figure 3: Comparative Analysis between Sync4D and Other Frameworks. On the left, the reference video alongside the edited video from DMT is displayed. The upper example shows a successful adaptation, whereas the lower example is deemed a failure due to continual alterations in shape and appearance across frames. On the right, the Sync4D outputs are highlighted, showcasing superior motion and shape consistency relative to other frameworks.
Refer to caption: x3.png

### 4.2 Results

We compare our proposed method with one framework: video motion transfer (DMT) [] combined with DreamGaussian4D []. The compared approach involves generating a motion-transferred video from the input casual video. This process begins by applying the DMT model to the initial video, effectively transferring the motion patterns to a new text-prompt object. Subsequently, the motion-transferred video is utilized in the DreamGaussian4D framework to generate the corresponding dynamics.

However, we observe in some complicated cases, the edited video from the DMT model has low quality and inconsistency. To tackle this problem, we employ ChatGPT [] to extract the description of the original video and convert the subject term to our target object. Then, we input the description to DreamGaussian4D to obtain corresponding dynamics.

As Figure 3 illustrated, for both experiments, our results outperform in both motion similarity and shape consistency. We also present the qualitative results of our generated 3D dynamics in comparison with reference video frames In Figure 4. Our method effectively captures the reference motion while preserving both the integrity of the shape and the temporal consistency of the dynamics. Please refer to the supplementary materials for video results.

Figure: Figure 4: We present the qualitative results of our generated 3D dynamics with reference video frames. Our method generates dynamics that align with the reference motion while retaining the shape integrity and temporal consistency. Please check the video results in the supplementary materials for a more intuitive illustration.
Refer to caption: x4.png

Figure: Figure 5: Ablation study on the number of bones in reconstruction to segment motion-related parts. Upper Row: number of bones $B=25$. Bottom Row: number of bones $B=13$, indicating the minimum articulated parts. Color black indicates removed outliers.
Refer to caption: x5.png

### 4.3 Ablation Studies

Number of Motion-related Parts. As illustrated in Figure 5, the upper row presents the matching and simulation results with the number of bones $B=23$, close to the conventional settings in the SMPL [] and SMAL []. We observe that some parts might be redundant in modeling the motions, for example, the circled part near the creaking nest, which results in stiffness in the target motion. In the bottom row, we set the number of bones to $B=13$, indicating the minimum articulated parts, which produces better dynamics in the target shapes.

Optimization Process. We choose not to optimize the velocity field in the simulation for the ablation study. Since the initialized velocity $v_{0}^{t}$ is a unit vector, resulting in an unobvious observation, we manually scale the initialized velocity to a certain numerical number $\alpha$. In this case, we prepare the velocity field with the scaled velocity by parts, as $\boldsymbol{v^{t}}\leftarrow\alpha v_{0}^{t}$. On the other side, we set up the full experiment with the same velocity field and get both of the generated motions illustrated in Figure 6. It is noticed that without optimization, relative errors are accumulated for the motion, affecting the simulation to ill-posed states.

Figure: Figure 6: Ablation study on optimization process. Upper Row: manually set up the initial velocity field. Bottom Row: with optimization to the initial velocity field.
Refer to caption: x6.png

## 5 Conclusion

This paper introduces Sync4D, a cutting-edge approach to 4D generation guided by casually captured video, which ensures exceptional motion realism and shape integrity. Our framework enhances general 3D generation by transferring motion with precise guidance from video sequences. Moreover, we incorporate physical simulations into the generation of 4D dynamics, optimizing the velocity field appropriately. Experimental results confirm the efficacy of Sync4D. This method not only facilitates intuitive control over 4D generation but also produces physically plausible dynamics, making it highly suitable for integration into various applications such as game engines and virtual reality environments.

Limitations.
Although Sync4D is capable of generating diverse dynamics across various shapes, it encounters difficulties when transferring motion to objects with different topologies. We view this limitation as indicative of the need for deeper knowledge in 3D scene comprehension and 4D motion understanding, areas that could benefit from advancements in more robust video diffusion models in the future. Additionally, our framework has a constraint concerning the initial pose of the reference video and the generated 3D representation; they cannot be substantially different. This limitation arises because our model focuses on learning relative motion rather than replicating each specific pose within the frames.

Broader Impacts. Positive Societal Impact: Video-guided 4D asset generation can significantly improve educational engagement and effectiveness through interactive, realistic simulations. Negative Societal Impact: The technology could be misused to create deceptive content, potentially violating privacy and spreading misinformation.

## Appendix A Appendix

### A.1 MPM Material Field

Despite particle position $\boldsymbol{x}$ and velocity $\boldsymbol{v}$ being tracked in MPM simulation, particle material properties are also sufficiently needed for updating. Firstly, we go through how material property $\boldsymbol{F}$, $\boldsymbol{C}$, $\boldsymbol{\nu}$, and $\boldsymbol{E}$ can influence the deformation of the object. Our Gaussian model is viewed as a continuum mechanics model, who utilize a deformation map $\phi(\mathbb{X},t)$ to record deformed space from base space $\mathbb{X}$. For numerical calculation, $\boldsymbol{F}$ is introduced to store the deformation gradient of $\phi$, know as the Jacobian of the map:

$$ $\displaystyle\boldsymbol{F}=\nabla_{\mathbb{X}}\phi(\mathbb{X},t)$ (17) $$

$\boldsymbol{F}$ measures the local rotation and strain of the deformation and helps formulate the stress-strain relationship.

Another two physics parameters noted are Shear modulus $\mu$ and Lamé modulus $\lambda$, which are related to Young’s modulus $\boldsymbol{E}$ and Poisson’s ratio $\boldsymbol{\nu}$:

$$ $\displaystyle\mu=\frac{\boldsymbol{E}}{2(1+\boldsymbol{\nu})},\quad\lambda= \frac{\boldsymbol{E}\boldsymbol{\nu}}{(1+\boldsymbol{\nu})(1-2\boldsymbol{\nu} )}.$ (18) $$

These two parameters help formulate Kirchhoff stress $\boldsymbol{\tau}$, which can be adapted to different elasticity and plasticity models. We utilize the fixed corotated elasticity model, whose Kirchhoff stress $\boldsymbol{\tau}$ is defined as:

$$ $\displaystyle\boldsymbol{\tau}=2\mu(\boldsymbol{F}^{E}-R)\boldsymbol{F}^{E^{T} }+\lambda(J-1)J,$ (19) $$

where $\boldsymbol{F}=\boldsymbol{F}^{E}\boldsymbol{F}^{P}$ is multiplicative decomposition on $\boldsymbol{F}$, while $R=UV^{T}$ is a matrix from Singular Value Decomposition on $\boldsymbol{F}$ as $\boldsymbol{F}=U\Sigma V^{T}$. $J$ is the determinant of $\boldsymbol{F}^{E}$.

In the process of MPM simulation, $\boldsymbol{F}$, $\boldsymbol{C}$, and $\boldsymbol{\tau}$ are also updated in P2G, G2P process, which can be denoted as:

$$ $\displaystyle\boldsymbol{C}^{t+1}_{p}$ $\displaystyle=\frac{4}{r^{2}}\sum_{i}N(\boldsymbol{x}_{i}-\boldsymbol{x}^{t}_{ p})\boldsymbol{v}_{i}^{t+1},$ (20) $\displaystyle\boldsymbol{F}^{t+1}_{p}$ $\displaystyle=(I+\boldsymbol{\Delta t}\boldsymbol{C}^{t+1}_{p})\boldsymbol{F}_ {p}^{t},$ (21) $\displaystyle\boldsymbol{\tau}^{t+1}_{p}$ $\displaystyle=\boldsymbol{\tau}(\boldsymbol{F}^{E,t+1}_{p}).$ (22) $$

This is just one case application for MPM simulator and for more details, please refer to [, , ]

**Table 1: Human study on Sync4D (Ours) over DMT generated video and DreamGaussian4D dynamics generation.**
| Overall Visual Quality | human-to-cross | laptop-to-shell | human-to-monkey |
| --- | --- | --- | --- |
| Ours over DMT | 82.4% | 100% | 94.1% |
| Ours over DreamGaussian4D | 100% | 94.1% | 100% |
| Motion similarity |  |  |  |
| Ours over DMT | 97.1% | 94.1% | 100% |
| Ours over DreamGaussian4D | 94.1% | 97.1% | 100% |
| Shape consistency |  |  |  |
| Ours over DMT | 88.2% | 100% | 94.1% |
| Ours over DreamGaussian4D | 88.2% | 94.1% | 97.1% |

### A.2 User Study Results

We conduct the user study on three sets of experiments, which are from human to cross, from laptop to sea shell, and from human to monkey toy. Participants are asked to choose between renderings from Sync4D and competitor’s generation forcibly. The three evaluation metrics are Overall visual quality, Motion similarity, and shape consistency. We render our dynamics in a fixed view, comparing it to video motion transfer output and renderings of DreamGaussian4D. Table 1 shows the remarkable advantage of Sync4D over other methods.

### A.3 Matching Results

In Figure 7, we present the results of articulated part matching between the reference and target shapes. Black color indicates the outliers that have been removed from the correspondence matching. As shown in row 2, for the human-cross pair, our method allows for reasonable matching even between pairs that are topologically different.

Figure: Figure 7: We showcase the articulated part matching between the reference and target shapes.
Refer to caption: x7.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: Q1: Can you provide quantitative metrics to evaluate the performance of your method against baseline models? This would strengthen the validation of your approach.

Q2: Could you elaborate on the implementation details of the triplane representation and the optimization process for the velocity field?

Q3: Beyond the brief mention, could you provide a deeper analysis of the limitations of your method, particularly regarding spinning motions and initial pose alignment? Are there potential solutions or future work directions to address these issues?
soundness: 3
strengths: - Approach: This method uniquely transfers motion from casual videos to 3D objects, bridging 2D video inputs with 4D dynamic generation. Using the Material Point Method (MPM) for simulation, it heightens realism and physical accuracy. The framework supports diverse object categories and motions—humans, animals, and articulated objects—showcasing robustness. A displacement loss optimizes the velocity field, reducing cumulative errors and improving temporal and shape consistency.
- Extensive Experimentation: Experiments cover qualitative results, comparisons, ablations, and user studies, validating the approach's effectiveness.
summary: Sync4D generates controllable 4D dynamics in 3D Gaussian representations, guided by casually captured reference videos. This approach uses blend-skinning-based non-parametric shape reconstruction to extract shape and motion from reference videos. It transfers motion to generated 3D Gaussian objects across categories by establishing shape correspondences and mapping motion-related parts. To tackle shape and motion inconsistencies, it incorporates physical simulation optimized with a displacement loss, ensuring reliable and realistic dynamics. Sync4D supports diverse inputs—humans, quadrupeds, and articulated objects—and claims superior performance over diffusion-based video generation methods.
weaknesses: - Related Works: The paper fails to reference several works in the field, such as Transfer4D, SC4D (Sparse-Controlled Video-to-4D Generation), and MagicPose4D. These works are directly related to the topic and have introduced techniques in motion transfer and 4D generation that are relevant to the proposed method. The lack of discussion about these works limits the contextualization of the contributions and may overlook existing solutions to some of the challenges addressed.
- Clarity: The paper's structure could be improved for better readability. Some sections, particularly the methodology, are dense and may be challenging for readers unfamiliar with the background concepts.
- Lack of Quantitative Eval: While qualitative results and user studies are provided, the paper lacks quantitative metrics (e.g., numerical comparisons, statistical significance) to objectively assess performance against baseline methods.
- Limited Discussion on Limitations: The limitations section briefly mentions challenges with spinning motions and initial pose alignment but lacks an in-depth analysis or potential solutions.
- Comparative Analysis: The comparisons with existing methods seem somewhat limited. For instance, the failure cases of other methods are highlighted, but it would be beneficial to include more successful cases for a balanced evaluation.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - Is it possible to infer material properties like elasticity from the reference video?
- Is it possible to retarget to real-world objects in addition to synthetic objects?
- Could you clarify the training methodology around how the different phases of the pipeline are trained and used to produce the final output?
soundness: 3
strengths: The paper is written well and method's originality is utilizing the MPM framework to retarget motion which makes this method temporally consistent and physically based. The proposed method is evaluated on real-world reference videos to drive synthetic objects and is able to transfer motion between very dissimilar topologies while maintaining temporal consistency with the use of a triplane velocity field to account for accumulated errors.
summary: The paper proposes a novel method to transfer motion from casually captured monocular videos to generated 3D models implicit representations and bones with skinning weights. A mapping is computed using deep features in 2D and 3D between the casually captured 3D object and the generated 3D object. Temporal consistency is achieved using a physically based MPM simulation framework used commonly in computer graphics.
weaknesses: - Various key decisions of the technique are not thoroughly evaluated. For example, with/without the Triplane velocity field, effects of various loss components, etc.
- The method is qualitatively evaluated on a limited dataset of reference videos and generated set of 3D objects. Expanding this evaluation to some publicly available datasets would improve quality of evaluation.
- There's no information around approximate training and inference speed for the method.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: **Video motion**

It seems the author uses BANMo to extract source object motions and geometry. However, BANMO might not suitable for all casually captured videos. I wonder what is the bottleneck in such pipeline and could the extracted motion well reflect the real object motions. For a video with both camera motion and object motion, will the transfer be success?
soundness: 3
strengths: **Interesting application**

The application is quite interesting and the generated results are good. Free-form motion transfer for 4D generation is useful in content creation community. 


**Motion transfer with physics integrated**

It is good to integrate MPM into motion transfer, with such we could generate deformations that typical articulated models can't achieve (like parametric human and animal models). The proposed velocity field optimization achieved good visual results.


**Applicable to different objects**

In the video the author shows good generalization to different types of objects.
summary: The author proposes a controllable 4D generation pipeline by transferring a source video motion to a target 3D object. To achieve this goal, the author proposes a a part mapping pipeline to match shape between source and target object,  and further integrate with MPM-guided physical simulation to synthesize the deformation. Experimental results show good motion transfer on various types of objects.
weaknesses: **Shape correspondence**

The matching only relies on shape correspondence. I wonder how could the matched parts guarantee to have good motion similarities with only rely on static features. Some more results on articulated objects and different geometry should be included to demonstrate the effectiveness, it will be also good to report some quantitative metric like matching success rate.




**MPM VS point-wise deformation**

The author mentions point-wide deformation would cause temporal inconsistency, however the author doesn't do such ablations to support the claim. For MPM, similar to what Zhang et al 2024 did, the author doesn't optimize the Young’s modulus and other physical parameters, so it is hard to say the later MPM-based method actually follows the real physics motion and thus weaken the built deformation. I am not fully convinced why MPM would work generally better than point or skeleton based approaches. Besides, MPM would be very hard to maintain rigidity for rigid objects and will also not work well for articulated objects and human with clear skeleton geometry. 



**Missing quantitative results**

As the author claim to do motion retargeting, the author should report some common metrics in motion transfer for ablations (like vertex-to-vertex distance and others) of different design choices and compare  with different type of objects (rigid, articulated, soft etc).

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 1
questions: Please refer to the weakness.
soundness: 2
strengths: - This work is systematic and covers aspects from video-based geometry/motion reconstruction, 3D Gaussian optimization from diffusion prior, correspondence determination, to material point method-based motion transfer.
summary: This paper introduces an approach for dynamic 3D Gaussian generation from casual videos. Firstly, the shape and motion are extracted from the reference videos. The reconstructed objects are then segmented into motion-related parts based on skinning weights and mapped to the generated 3D Gaussians. Additional physical simulation and displacement loss are applied to drive the 3D Gaussians to maintain integrity and accuracy.
weaknesses: - The method seems to simply correspond the Gaussian points in the target shape to each part of the reference model, but it does not perform geometric analysis on the target shape or define a corresponding skeleton. The motion of the Gaussian points primarily relies on the movement of bones in the reference model and the solution of the material point method (MPM). This makes it difficult to ensure the reasonableness of motion under the target shape. For example, in the bottom left corner of Fig 5, the box shows significant distortion in the supplementary video (00:22s).
- The rendering quality of the generated dynamic 3D Gaussians is relatively poor. The advantage of Gaussians lies in rendering scenes with difficult surfaces, such as fuzzy geometry, while MPM excels at representing physical properties in motion, such as elasticity. However, the examples presented in this work do not demonstrate these strengths; all the dynamics are merely simple hard surface geometric deformations.
- Although Fig 9 mentions that many generated geometric rigs cannot be handled well, for some examples shown in this paper, such as the humanoid bear and giraffe, directly generating a 3D mesh, coupled with geometric analysis to extract the skeleton and skinning weights and then applying texture mapping, could yield better results.