Vidu4D: Single Generated Video to High-Fidelity 4D Reconstruction with Dynamic Gaussian Surfels
Authors: Yikai Wang
Xinzhou Wang
Zilong Chen
Zhengyi Wang
Fuchun Sun
Jun Zhu
Venue: NeurIPS 2024 poster (2024)
Keywords: generative models
4D reconstruction
Gaussian splatting
Primary area: generative_models
TL;DR: Vidu4D, a reconstruction model that excels in accurately reconstructing dynamic gaussian surfels from single generated videos.

Abstract: Video generative models are receiving particular attention given their ability to generate realistic and imaginative frames. Besides, these models are also observed to exhibit strong 3D consistency, significantly enhancing their potential to act as world simulators.  In this work, we present Vidu4D, a novel reconstruction model that excels in accurately reconstructing 4D (i.e., sequential 3D) representations from single generated videos, addressing challenges associated with non-rigidity and frame distortion. This capability is pivotal for creating high-fidelity virtual contents that maintain both spatial and temporal coherence. At the core of Vidu4D is our proposed Dynamic Gaussian Surfels (DGS) technique. DGS optimizes time-varying warping functions to transform Gaussian surfels (surface elements) from a static state to a dynamically warped state. This transformation enables a precise depiction of motion and deformation over time. To preserve the structural integrity of surface-aligned Gaussian surfels, we design the warped-state geometric regularization based on continuous warping fields for estimating normals. Additionally, we learn refinements on rotation and scaling parameters of Gaussian surfels, which greatly alleviates texture flickering during the warping process and enhances the capture of fine-grained appearance details. Vidu4D also contains a novel initialization state that provides a proper start for the warping fields in DGS. Equipping Vidu4D with an existing video generative model, the overall framework demonstrates high-fidelity text-to-4D generation in both appearance and geometry.

Vidu4D: Single Generated Video to High-Fidelity 4D Reconstruction with Dynamic Gaussian Surfels
Yikai Wang"l, Xinzhou Wang 1.2,3 Zilong Chen 1.2 Zhengyi Wang!.2 Fuchun Sun', Jun Zhu
Department of Computer Science and Technology. BNRist Center, Tsinghua University
ShunggShu College of Electronic and Information Engineering, Tongji University yikai com, wangxi hingxinzhou@tonggi edu..cd.....ooo dcs? scszjotsinght edu.
Abstract
Video generative models are receiving particular attention given their ability to gen- erate realistic and imaginative frames. Besides, these models are also observed exhibit strong 3D consistency, significantly enhancing their potential to act as world simulators. In this work, we present Vidu4D, novel reconstruction model that excels in accurately reconstructing 4D (i.e., sequential 3D) representations from single generated videos, addressing challenges associated with non-rigidity and frame distortion. This capability is pivotal for creating high-fidelity virtual contents that maintain both spatial and temporal coherence. At the core of Vidu4D our pro- posed Dynamic Gaussian Surfels (DGS) technique. DGS optimizes fime-varying warping functions to transform Gaussian surfels (surface elements) from static state to dynamically warped state. This transformation enables a precise depiction motion and deformation over time. To preserve the structural integrity of surface- aligned Gaussian surfels, design the warped-state geometrie regularization based on continuous warping fields for estimating normals. Additionally, we learn refinements rotation and scaling parameters of Gaussian surfels, which greatly alleviates texture flickering during the warping process and enhances the capture of fine- ine-grained appearance details. Vidu4D also contains novel initialization state that provides proper start for the warping fields in DGS. Equipping Vidu4D with existing video generative model, the overall framework demonstrates high- fidelity text-to-4D generation in both appearance and geometry. Project page: https: htt///idddddddoo wtt.ggg...tt...u..... withub.....
Introduction
The field of multimodal generation exhibits significant advancements and holds great promise for various applications. Recently, video generative models have garnered attention for their remarkable capability to craft immersive and lifelike frames [4, 8]. These models produce visually stunning content while also exhibiting strong 3D consistency [15, 81], largely increasing their potential to simulate realistic environments.
Parallel to these developments. high-quality 4D reconstruction has made great strides [19, 58, 63, 94, 100]. This technique involves capturing and rendering detailed spatial and temporal information. When integrated with generative video technologies, 4D reconstruction potentially enables the creation of models that capture static scenes and dynamic sequences over time. This synthesis provides more holistic representation of reality, which is crucial for applications such virtual reality, scientific visualization, and embodied artificial intelligence.
Equall contribution.
'The corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a) Prompt: A portrait captures the dignified presence of orange cal with striking blue eyes. The cat wears single pearl carring. Her head tilts in contemplation, reminiscent of Dutch cap.
(b) Prompt: dragon with its hair blown by strong wind. Devil enters the soul with ethereal landscapes.
(c) Prompt: Light painting photo of a cheetah, cinematic.
(d) Prompt: A goldfish seemingly swimming through the air.
(e) Prompt: small, fluffy with appearance reminiscent mythical being. The creature's fur texture is rendered high detail. The monster's large eyes and open mouth express wonder and curiosity.
(f) Prompt: An isolated coloured abstract sculpture with dali shape.
Figure 1: Text-(to-video)--o-4D samples generated by equipping Vidu4D with a pretrained video diffusion model [4]. For each sample, we exhibit per-frame 3D rendering for novel-view color, normal, and surfel feature. We observe that Vidu4D can reconstruct precisely detailed and photo-realistic 4D representation. See our accompanying videos in our project page for better visual quality.

---

However, achieving high-fidelity 4D reconstruction from generated videos poses great challenges. Non-rigidity and frame distortion are prevalent issues that can undermine the temporal and spatial coherence of the reconstructed content, thus complicating the creation of seamless and coherent depiction of dynamic subjects.
In this work, we introduce Vidu4D, a novel reconstruction pipeline designed to accurately recon- struct 4D representations from single generated videos, facilitating the creation of 4D content with high precision in spatial and temporal coherence. Vidu4D contains two novel stages, namely, the initialization of non-rigid warping fields and Dynamic Gaussian Surfels (DGS), together enabling the reconstruction of high-fidelity 4D content with detailed appearance and accurate geometry.
Specifically, the proposed DGS optimizes non-rigid warping functions that transform Gaussian surfels from static to dynamically warped states. This dynamic transformation accurately represents motion and deformation over time, crucial for capturing realistic 4D representations. Besides, DGS demonstrates superior 4D reconstruction performance due to two other key aspects. Firstly, in terms of geometry, DGS adheres to Gaussian surfels principles [16, 28] achieve precise geometric repre- sentation. Unlike existing methods, DGS incorporates warped-state normal consistency regularization to align surfels with actual surfaces with learnable continuous fields (w.r.t spatial coordinate and time) to ensure smooth warping when estimating normals. Secondly, for appearance, DGS learns additional refinements on the rotation and scaling parameters of Gaussian surfels by dual branch structure. This refinement reduces the flickering artifacts during warping and allows for the precise rendering of appearance details, resulting in high-quality reconstructed 4D representations.
By integrating Vidu4D) with an existing powerful video generative model named Vidu [4], the overall framework demonstrates exceptional capabilities in text-to-4D generation. We provide 4D visualization results in Fig. 1. Extensive experiments based on the generated videos verify the effectiveness of our method compared current state-of-the-art methods.
2 Related works
3D representation. Transforming 2D images into 3D representations has long been a central chal- lenge in the field. Initially, triangle meshes were favored for their compactness and compatibility with rendering pipelines [9, 17, 67, 78, 82, 93]. However, the transition to more sophisticated volu- metric methods was inevitable due to the limitations of surface-based approaches. Early volumetric representations included voxel grids [35, 48, 61, 72] and multi-plane images [20, 54, 74, 75, 80, 105], which, despite their straightforrwrddesss, demanded intricate optimization strategies. The introduction of neural radiance fields (NeRF) [55] marked a great advancement, offering an implicit volumetric neural representation that could store and query the density and color of each point, leading to highly realistic reconstructions. The NeRF paradigm has since been improved upon in terms of reconstruction quality [5, 6. 33, 53, 92] and rendering [12, 23, 25, 27, 41, 45, 49, 64-66, 85, 102]. To address the limitations of NeRF, such as rendering speed and memory usage, recent work dubbed 3D Gaussian splatting (3DGS) [33] has proposed anisotropic Gaussian representations with GPU- optimized tile-based rasterization. This has opened up new avenues for surface extraction [24, 28], generation [14, 77, 96], and large-scale scene reconstruction [34, 46, 70], with 3DGS emerging as a universal representation for 3D scenes and objects. Gaussian surfels methods [16, 28] further exhibit advantages in modeling accurate geometry. While these methods have significantly advanced the field of static 3D representation, capturing the dynamic aspects of real-world scenes with non-rigid motion and deformation introduces a distinct set of challenges that demand innovative solutions.
Dynamic reconstruction and generation. The dynamic reconstruction of scenes from video captures presents more complex challenge than static reconstruction, necessitating the capture of non-rigid motion and deformation over time [30, 37, 60, 76, 88]. Traditional methods have explored dynamic reconstruction using synchronized multi-view videos [1. 3. 11, 36, 48, 59, 73, 83, 84, 86, 89] or have focused on specific dynamic elements like humans or animals. More recently, there has been a shift towards reconstructing non-rigid objects from monocular videos, which is a more practical yet challenging scenario. One approach involves incorporating time as an additional input to the neural radiance field [11, 38, 68, 98], allowing for explicit querying of spatiotemporal information. Another line of research decomposes the spatiotemporal radiance field into canonical space and a deformation field, representing spatial attributes and their temporal variations [18, 19, 19, 21, 22, 31, 39, 44, 47, 57, 58, 63, 69, 79, 95, 104]. With advancements in 3DGS, deformable-GG [100]
3

---

Field initialization stage
DGS stage
Initialization
Forward warping (for sampled points)
Forward warping
Statie state
Warped state
Static
(for Gaussian surfels)
Warped state
Inversion
state
Backward warping (for sampled points)
Sampled point
Gaussian surfel
Figure 2: Illustration of the pipeline of Vidu4D, including the initialization stage and the DGS stage.
and 4DGS [94] have been developed, utilizing neural deformation fields with multi- nulti-layer perception (MLP) and triplane, respectively. SCGS [29] and dynamic 3D Gaussians [52] also advance the field by modeling time-varying scenes. Building on these advances, our work introduces dynamic Gaussian surfels, a novel extension of Gaussian representations that enhances the quality of both appearance and surface reconstruction under dynamic scenarios. A concurrent work DGM [43] builds time-consistent meshes from monocular video with 3D Gaussian Splatting. In the realm of 3D or 4D generation, our approach diverges from recent progress in optimization-bannt feed-forward [26, 91, 106], and multi-view reconstruction methods [15, 50, 51] by leveraging video generative model to achieve generation capabilities. Our primary focus is on preserving high-quality appearance and geometrical integrity from generated videos. This results in generation process that not only captures the nuances motion and deformation but also maintains the high standards of realism and detail that essential for creating immersive and lifelike virtual 3D representations.
[2. 13, 14, 40, 42, 62, 71, 88, 90],
3 Method
We start by introducing the problem definition for 4D reconstruction in Sec. 3.1. Following that, introduce our Vidu4D which encompasses two novel stages. The first stage is designed to learn Dynamic Gaussian Surfels (DGS), ensuring precise representation of both visual appearance and geometric structure during the non-rigid reconstruction process, as detailed in Sec. 3.2. The second stage focuses on establishing the initial non-rigid warping fields of DGS, as detailed in Sec. 3.3.
3.1 Problem Definition
When given single sequence of RGB video with T frames, the goal of 4D reconstruction is to determine a sequential 3D representation that could be rendered to fit each video frame as much as possible. Specifically, suppose the 3D representation for the (-th frame (termed time () is parameterized by 0,, where t = 1. T. Given a differentiable rendering mapping g, we could obtain the rendered color at the frame pixel x IR? We choose volume rendering commonly adopted in NeRF [55], Gaussian Splatting [33], and Gaussian Surfels [16, 28]. The optimization of 4D reconstruction can be implemented by minimizing the empirical loss
T
Σ MΣΣ c(c(x') t=1 x
= g(0,, (x/T)=,.....))) (() c(x)))
min T
where x E I3' is the i-th 3D point sampled or intersected with Gaussian primitives along the ray that emanates from the frame pixel x"; N the number of sampled or intersected points per ray; c(x') and c(x') are the rendered color and the observed color at x', respectively.
In the following, we detail the proposed Vidu4D, reconstruction pipeline comprising two key stages as illustrated in Fig. 2, including a field initialization stage and DGS stage.
3.2 Dynamic Gaussian Surfels
By optimizing Eq. (1), essentially our goal is to build a sequential 3D representation that could deform
to be consistent with each 2D frame. We first start by considering an ideal video exhibiting different views of the same static object without object deformation, movement, video distortion. To model the 3D representation with high appearance fidelity and geometry accuracy, we follow the method of using differentiable 2D Gaussian primitives proposed by recent Gaussian Surfels advances [16, 28].

---

prompt sriking New eyyss...
Video diffiusion model +
Field init.
Warped-state normal
with
DGS
regularization
R
P
Method details of DGS
D
Warping cond. on 2 and u [R 11
Joini training with stared centers and warping
Warped state
A. CARC
Refinement AR. AS;
R +
Rasterization
Figure 3: Illustration of the overall framework and our DGS in detail. For DGS, Gaussian surfels in the static state are transformed the warped state by learning non-rigid warping functions conditioned on time and coordinate u. We incorporate warped-state normal regularization for accurate geometry. and refined rotation and scaling matrices of Gaussian surfels for detailed appearance. Both branches in the warped state, including with and without refinement, share the same centers of Gaussian surfels and the same warping functions. "Field init." stands for field initialization as introduced in Sec. 3.3.
Specifically, the k-th Gaussian surfel (of the total K) is characterized by central point P E IR' and a local coordinate system centered at P with two principal tangential vectors t, E R t, E 33x1 and scaling factors S IR, 8 E RR. Here, static state. A Gaussian surfel is computed as a 2D Gaussian defined in a local tangent plane in the world space. Following [28], for any point (u, v) located on the coordinate system centered at pk. its coordinate in the world space, denoted as Px (u) E g is computed by
use the notation "*" represent parameters in the
(u) = Pp + S S where Rt - [t t x t+] E SO(3) denotes the rotation matrix, and the diagonal matrix Sj. diag(s;, S 0) R333 denotes the scaling matrix.
s
PEl (u, v, 1)
(2)
In this work, our focus is on 4D reconstruction from a single generated video, which may exhibit large non-rigidity, distortion, or illumination changes. We introduce Dynamic Gaussian Surfels (DGS), method designed to achieve precise 4D reconstruction while accommodating non-rigidity and other time-varying effects.
Motivated by recent advancements in non-rigid reconstruction methods [57, 88, 98], we aim to ensure that the target object maintains consistent static state across different frames, thereby mitigating non-rigidity and distortion effects. To achieve this, we employ warping techniques on each Gaussian surfel represented by P (u), transforming them into corresponding Gaussian surfel (u) at time I, which is centered at Pk E I with rotation matrix R E SO(3) and a scaling matrix S E Rax3
Non-rigid warping for Gaussian surfels. We now build the warping process from the static state to the warped state. We leverage a non-rigid warping function with bones as key points to ease the training of deformation. In the static state, the b-th bone is represented by 3D Gaussian ellipsoids [97], with more details provided the Appendix. We let JJ E SE(3) represent a rigid transformation that moves the b-th bone from its static state to the warped state at time t. In effect, J is achieved by non-linear mappings using multi-layer perception (MLP) with SE(3) guaranteed, as will be given later in Eq. (5). The non-rigid warping function be written the weighted combination of J where we apply dual quaternion blend skinning (DQB) [32] to ensure valid SE(3) after combination,
B
J== R Σ o (
where w/ is the b-th element of the skinning weight vector w² E BR as detailed in the Appendix; Q and R denote the quaternion process and the inverse quaternion process, respectively. In this case, there is J E SE(3).
We therefore rewrite the warping as J = R T with the rotation R E SO(3) and translation T E R and apply the corresponding transformation to Eq. (2) by
(u) J' (u) R R'p T 1,
(4)
5

---

Note that Eq. (4) holds for any given point (T (u) including the center point of the k-th Gaussian surfel (i.e., P when u 0). By deriving Eq. (4), we enable connection of the warping function w.E. to any point u = (u, v) on the local coordinate system centered at P which is needed later in Eq. (8) where u an intersection with Gaussian surfels and a ray that emanates from the frame pixel.
Warped-state normal regularization. To accurately capture the geometric representation. we follow similar methods in Gaussian Surfels [16, 28] to add normal consistency regularization which encourages all Gaussian surfels to be locally aligned with the actual surfaces. Differently, unlike 3D reconstruction for static scenes, 4D reconstruction commonly faces non-rigidity and distortion. Thus simply performing regularization to promote surface-aligned Gaussian surfels like previous methods harms the structural integrity due to the non-rigid warping.
We therefore design a warped-state normal regularization. As mentioned, each point (u( in the warped state at time is transformed from its corresponding static point (? based on the warping function in Eq. (4), namely, (u) = P (u) with J composed by J To maintain the structural integrity large extent when regularizing normal, we design Jj as continuous field that takes both the point P (u) (or equivalently, u in the local coordinate system) and the time t as conditions. By this setting, J is expected to change continuously with the change of or t. We implement the continuous field by using a NeRF-style MLP which directly outputs a 6-dimensional dual quaternion, and rely on the inverse quaternion process R to guarantee SE(3), i.e..
(5) where V is learnable latent code for encoding the b-th bone at time f; both u and f are sent to the MLP as conditions to obtain J. Thus J' is also expected to be continuous w.r.f. u and t.
J = R (MLP t)):
Based on the above design, the normal consistency loss at time t is obtained similar to [28].
Ln Σ 2ua(1- N² = V x V '' k
(6)
V.P' yp'
where l: indexes over intersected surfels along the ray that emanates from the frame pixel x; We ag Gu (u(x)) I a, Gy(u(x)) denotes the blending weight of the intersection point; ng represents the normal of the surfel that is oriented towards the camera; N computed with finite differences, is the surface normal estimated by the nearby depth point p' at warped state time t.
In summary, by learning a continuous warping field and aligning the surfel normal with the estimated surface normal in the warped state, we ensure that all Gaussian surfels locally approximate the actual object surface without being noticeably impaired by the non-rigid warping.
Dual branch structure with refinement. To further achieve line-graince appearance and reduce the texture flickering during warping, we propose to learn refinement terms for adjusting the rotation matrices R₁ and scaling matrices S (defined in Eq. (2)) in the static state. We suppose the refinement terms are AR; E SO(3) and AS; E R3x3, respectively. Note that the third-axis of AS; is no longer necessarily O. During refinement, we remain the center points P and the warping J' (i.e., including both R and T') to be unchanged. The new warped process is formulated as,
Pt (u) = [[) + AS;) R'p + T'] (u, T.
During the training of DGS, we maintain two branches including one with refinement and one without. In the warped state, both branches are jointly trained with shared warping functions and centers of Gaussian primitives?. Due to the involvement of AR; and AS; both branches have different rotation and scaling matrices of Gaussian primitives.
Rasterization. Given frame pixel x and camera ray that emanates from x, following the static- state methods to calculate intersection coordinates with Gaussian primitives along the ray [28, 33], we could obtain warped-state intersection coordinates based on Eq. (4) and Eq. (7). We then perform the volume rendering process [28] that integrates dipha-weighted appearance along the ray by
k-1
c(x) Σ cs an Ge (u(x) II (1 aj (u(x))))
(8)
'Here. since the -xis of the refined scaling matrix is not necessarily 0, we adopt "Gaussian primitive" for commonly referring to both Gaussian surfel and the refined Gaussian.

---

where k indexes over intersected Gaussian primitives along the ray that emanates from the frame pixel x; ag and Cą denote the opacity and view-dependent appearance parameterized with spherical harmonics of the k-th Gaussian surfel, respectively; G% (u(x)) = exp "²² 2
corresponds to the k-th intersection point u(x) which could be directly calculated when given (u PK (u) and the corresponding local coordinate system. During implementation, Gy (x())) is further applied a low-pass filter following [7. 28].
detailed architecture of DGS is depicted in Fig. Important symbols are summarized in our Appendix.
3.3 Field Initialization
Given that the camera trajectory of generated videos is unknown, SfM methods like COLMAP struggle to converge due to rigidity violations. Additionally, since the background of generated videos appears to exhibit soft deformation flickering colors, proper estimation of camera/body poses through background SfM is hindered. These challenges often result in very few successful registrations, demonstrated in previous monocular 4D reconstruction tasks [98].
To address this, we design an implicit field before performing DGS initialize the camera poses and establish the continuous warping field in Eq. (5). In this part, we propose the field initialization as another key component our pipeline to initialize the continuous warping field of DGS for fast and stable convergence, as detailed below.
Initially, we train a neural Signed Distance Function (SDF) model [87], leveraging the same warping structure with bones as utilized in DGS. While DGS transforms Gaussian surfels from the static state to the warped state for rasterization, the neural SDF reverses this process, mapping points along camera rays from the warped state back to the static state. For the neural SDF component, we optimize the reverse warping process and deduce the forward warping as inverse by minimizing a cycle loss, inspired by [10, 98]. Subsequently, we initialize MLP(.) in Eq. (5) to obtain warping functions J by the network weights learned by the neural SDF part.
During the rendering of the neural SDF, we perform backward warping on the warped-state sampling points to the static state.
J(. = R R( ( of w()) ='=J'-1--
(9)
which is an inversion of Eq. (3). By querying the SDF with a sample point X in the static state, we render RGB and compute the photometric loss to optimize the SDF and the warping field defined in Eq. (5).
Nevertheless, there are two discrepancies between the neural SDF warping and DGS warping. Firstly, sampling points of the neural SDF are distributed in the frustum of the camera, while sampling points of DGS are distributed on the object surface. Additionally, we train the inversion of during initialization, while we utilize the non-inverse ones DGS. To resolve the distribution gap and ensure that faithfully models the forward warping, we add cycle loss,
Ceye =
(x')) x/ 1|:
(10)
where X could be either a mesh surface point or a sample point on the camera ray.
After initialization, we extract the canonical space mesh using marching cubes and initialize Gaussian surfels on We set the spherical harmonic in 0-th order to the RGB value of the nearest vertices. The warping field and learned camera poses are retained.
With the field initialization before DGS, our Vidu4D is capable of performing a text-(to-video)-oo44D generation task with the integration of existing video diffusion models.
Experiment
In this section, we provide extensive evaluation of our method DGS (Sec. 3.2) with the initialization in Sec. 3.3, comparing both appearance and geometry against previous -f methods. Additionally, we analyze the contributions of each proposed component in detail.
7

---

BANMo
NeRt
Def Four fiel Eei.
SC GSS HoR Ini:
BANM
Defermble
SE
Deformel--GS
loss dell
SC GS hou Celd Insl.
Spr
Ki=
Novdl
Novd
Nuemal
Novel
Niemal
Novd
Figure 4: Novel-view qualitative evaluation compared with SOTA methods including NeRF-hased methods (BANMo [98] and D-NeRF [63]) and Gaussian splatting-bass methods (Deformable- GS [100] and SCGS [29]). We also provide our learned camera poses to baseline approaches for fair comparison. These variants are denoted as "w. Poses", Best view in color and zoom in.
4.1 Implementation
For all qualitative and quantitative experiments, follow the standard pipeline for dynamic recon- struction [58], to construct our evaluation setup by selecting every fourth frame as training frame and designating the middle frame between each pair of training frames validation frame.
Our model configuration involves several key parameters to balance reconstruction and regularization losses. For the field initialization stage, rendering as in NeRF [55], and initialize MLP for predicting SDF as an approximate unit sphere [101] We obtain neural SDF, warping field, and camera poses after this stage. For the DGS stage, initialize centers of the Gaussian surfels with the sampled surface points extracted from the neural SDF, and initialize the warping field by the forward field from the first stage. The dimension of the latent code embedding 7/ is set as 128. Following BANMo [98], we adopt 25 bones to optimize skinning weights. For each reconstruction, the overall training takes I hour A800 GPU.
similar architecture with 8 layers for volume
4.2 Qualitative Evaluation
In the qualitative evaluation, we visually compare the novel-view reconstructions produced by our DGS against those generated by other state-of-the-an models, as illustrated in Fig. Our evaluation focuses several key aspects including detail preservation, texture quality, and geometric accuracy. Compared to methods based on implicit fields, the integration of Gaussian our approach facilitates the rendering of highly detailed textures. Additionally, benefiting from more geometry-aware representation, our method produces superior normal maps compared to those purely Gaussian-bace methods. This also enhances the robustness of our method against artifacts of the generated videos

---

Table 1: Novel-view quantitative results on generated videos. Evaluation metrics are PSNR, SSIM, and LPIPS. We report results on three single videos and the averaged results over 30 single videos.
Deagon
Ascrage
PSNR
SSIM
LPISS
PSNR
SSIM
LPPS
PSNR:
SSIM
LPPPS
PSNR
SSIM 7
LPIPS
BANMa 195]
0.9921
15.48
0.6423
0.3500
13.62
2.09
0.6153
0.3758
D-NeKF (63]
15.15
0.6337
0.2657
13.21
0.599
0.3344
16.53
0,6489
013522
21.01
2386
0,5519 à 0U0737
0.1522 00034
Defomable-Gr [100]
24.19
0.9100
000992
13.22
SCGS (29)
19.46
0.7867
0.2405
20.87
0,8123
0.1919
24,03
0.9083
0.1009
21.17
2,69
0.8547
1 0.0991
0.1594 0.0737
Defomable-GS
22:41
26.15
0,9218
22.63
SCGS our field init.
25.25
0.8288
0.1574
23.70
0,8338
0.1497
25.40
0.9375
0,0686
24.75
2.11
0.3680
1. 0.0640
0.1201 0.0399
Rendered color
Rendered nonmal
Surface nommal
Rendered solor
Rendered soceaa
(3) Ground truth
(b) Our full model
w/o, relinement branch
w/o, warped-state nurnal regultizztion
Figure 5: Ablation studies on the geometric regularization and refinement strategy. For our full model shown in (b), we provide our rendered color, rendered normal, and surface normal (estimated from comparison, we visualize the rendered color for
the depth points for regularization). Additionally, the case without refinements in (c) and the rendered normal for the case without warped-state normal regularization (d), respectively. We showcase our model's fidelity with close-ups.
like occlusions. For instance, the third clip the series, which features a dragon shrouded in fog, both SCGS and Deformable-GS methods tend to overfit and subsequently show decline in performance. In contrast, our method consistently delivers superior results.
4.3 Quantitative Evaluation
We provide the quantitative evaluation comparing our method with state-of-the-art works in Table 1. Metrics include Peak Signal-to-Noise Ratio (PSNR) evaluate the fidelity of the reconstructed tex- tures, Structural Similarity Index (SSIM) for the quality evaluation, and LPIPS [103] as perceptual metric. Our method exhibits superiority over all baseline methods, even with our learned poses, e.g., 225 PSNR increase over SCGS with poses for the averaged results.
4.4 Ablations
To understand the contributions of each component in Vidu4D, especially DGS, we conduct ablation studies in this section. We remove or alter specific elements of our model and observe the resulting performance changes in both appearance and geometry reconstruction.
Geometric regularization. We evaluate the impact of warped-state normal regularization by disabling during training. From Fig. 5(b)(d), observe that when removing the regularization, there is an obvious degradation in the structural integrity of surface-aligned Gaussian surfels, leading to noticeable inconsistency in the reconstructed 4D models.
Refinement strategy. We examine the effect of omitting refinements by keeping one branch (the concept of branches could be better visualized in Fig. 3) during training, shown in Fig. 5(b) The performance indicates that removing refinements increases the loss of fine- fine-grained appearance details. Additionally, also find that refinements are crucial for mitigating the texture flickering issue.
Additional ablations. Please refer to the Appendix for additional ablation studies that detail the effectiveness of our refinement strategy and field initialization.

---

Conclusion
We introduce Vidu4D as novel reconstruction model to achieve high-fidelity 4D representations from single generated videos. Vidu4D is powerful with our proposed DGS which builds the non-rigid warping field to transform Gaussian surfels, ensuring precise capture of motion and deformation over time. DGS also introduces key innovations that greatly enhance the accuracy and fidelity of 4D reconstruction, including dual branch refinement and warped-state geometric regularization. Our experiments demonstrate that Vidu4D outperforms existing methods in both quantitative and qualitative evaluations, highlighting its superiority in generating realistic and immersive 4D content.
Limitations and broader impact. While Vidu4D with DGS presents significant performance in 4D reconstruction, currently there are still limitations such as the reliance on video quality, scalability challenges for large scenes, and computational difficulties in real-time applications. Additionally, when equipping Vidu4D with generative models, as with any generative technology, there is risk of producing deceptive content which needs more caution.
6 Acknowledgement
This work was partly supported by the NSFC Projects (Nos. 62350080, 62306163, 62276149, 92370124, 92248303, U2341228, 62061136001 62076147), BNRist (BNR2022RC01006 Tsinghua Institute for Guo Qiang, and the High Performance Computing Center, Tsinghua University. J. Zhu was also supported by the XPlorer Prize. Y.K. Wang was also supported by the China National Postdoctoral Program (No. 2023M741951) and Major Project of the New Generation Artificial Intelligence (No. 2018AAA0102900),
References
[1] Attal, B., Huang, J.B., Richardt, C., Zollhoefer, M., Kopf, J., O'Toole, M., Kim, C.: Hyperreel: High- fidelity 6-dof video with ray condiiiit sampling. arXiv preprint arXiv:2301.0222 .223032002288 (2023)
[2] Bahmani. S., Skorokhodov, I., Rong. V.. Wetzstein, G., Guibas. L., Wonka, P., Tulyakov. S., Park, J.J. Tagliasacchi. A., Lindell, D.B.: 4d-fy: Text-to-4d generation using hybrid score distillation sampling. In: CVPR) (2024)
[3] Bansal, A., Vo, M., Sheikh, Y., Ramanan, D., Narasimhan, S.: 4d visualization of dynamic events from unconstrained multi-view vidcos. In: CVPR (2020)
[4] Bao, F., Xiang, C., Yue, G., He, G., Zhu, H., Zheng, K., Zhao, M., Liu, S., Wang, Y., Zhu, Vidu: a highly consistent, dynamic and skilled texi-to-video generator with diffusion models. arXiv preprint arXiv:2405.0423 (2024)
[5] Barron, J.T., Mildenhall, B., Tancik, M., Hedman, P., Martin-Brualla, R., Srinivasan, P.P.: Mip-nerf: multiscale representation for anti-aliasing neural radiance fields. In: ICCV (2021)
[6] Barron, J.T., Mildenhall, Verbin, D., Srinivasan, P.P., Hedman, P.: Zip-nerf: Anti-aliased grid-hased neural radiance fields. arXiv preprint a7X1v (2023)
[7] Botsch, M., Hornung, A.. Zwicker. M., Kobbelt, L.: High-quality surface splatting on today's gpus. In: Proceedings Eurographies/IEE VGTC Symposium Point-Based Graphics (2005)
[8] Brooks, T., Peebles, B., Holmes, C., DePue, W., Guo, Y., Jing, L., Schnurr, D., Taylor, J., Luhman, T.
Luhman, E., Ng. C., Wang, R., Ramesh, A.: Video generation models as world simulators (2024), ht attps: / //openai... com researsseec htt./vvddeoeeeeee gen.ratteeeeeraateeeeome
model s-asss worlddd sil-ddiinee www.smmmmtoooooommmmmmmm
[9] Buchler, C., Bosse, M., McMillan, I.., Gortler, S., Cohen, M.: Unstructured lumigraph rendering. In: Proceedings of the 28th annual conference on Computer graphics and interactive techniques (2001)
[10] Cai, H., Feng, W. Feng, X., Wang. Y., Zhang, J.: Neural surface reconstruction of dynamic with monocular RGB-D camera. NeurIPS (2022)
[11] Cao, A., Johnson, J.: Hexplane: fast representation for dynamic scenes. arXiv preprint arXiv:2301.00633 (2023)
[12] Cao, J.. Wang, H., Chemerys, P., Shakhrai, V.,
Hu, J., Fu, Y.. Makoviichuk, D., Tulyakov. S., Ren, J.: Real-time neural light field on mobile devices. arXiv preprint arXiv:2212 (2022)
[13] Chen, R., Chen, Y. Jiao, N., Jia, K.: Fantasia3d: Disentangling geometry and appearance for high-quality text-to-34 3d content creation. In: ICCV (2023)
[14] Chen, Z., Wang, F., Wang, Y., Liu, H.: Text-to-3d using gaussian splatting, In: CVPR (2024)
10

---

[15] Chen, Z., Wang. Y., Wang, F. Wang. Z., Liu. H.: V3d: Video diffusion models are effective 3d generators. arXiv preprint arXiv:2403.06738 (2024)
[16] Dai, P., Xu, J., Xie, W., Liu, X., Wang, H., Xu, W. High-quality surface reconstruction using gaussian surfels. In: SIGGRAPH (2024)
[17] Dehevec, P.F., Taylor, C.J., Malik, J.: Modeling and rendering architecture from photographs: A hybrid
geometry-and image-bases approach. In: Proceedings of the 23rd annual conference on Computer graphics and interactive techniques (1996)
[18] Du, Y., Zhang, Y. Yu, H.X., Tenenbaum, J.B., Wu, Neural radiance flow for 4d view synthesis and video processing. In: ICCV (2021)
[19] Fang, J., Yi, T., Wang, X., Xie, I.,, Zhang. X., Liu, W., Nießner, M., Tian, Q.: Fast dynamic radiance fields with time-aware neural voxels. In: SIGGRAPH Asia (2022)
[20] Flynn, J., Broxton, M., Debevec, P., DuVall, M.. Fyffe, G., Overbeck, R., Snavely, N., Tucker, R.: Deepview: View synthesis with learned gradient descent. In: CVPR (2019)
[21] Gao, C., Saraf, A., Kopf, J., Huang, J.B.: Dynamic view synthesis from dynamic monocular video. In:
ICCV (2021)
[22]
Gao, H., Li, R., Tulsiani, S., Russell, B., Kanazawa, A.: Dynamic novel-view synthesis: reality check.
In: NeurlPS (2022)
[23] Garbin, S.J., Kowalski, M.. Johnson, M., Shotton, J., Valentin, J.: Fastnerf: High-fidelity neural rendering 200fps. In: ICCV (2021)
[24] Guédon, A., Lepetit, V:: Sugar: Surface-aligned gaussian splating for efficient 3d mesh reconstruction and high-quality mesh rendering. arXiv preprint arXiv:2311.12775 (2023)
[25] Hedman, P., Srinivasan, P.P., Mildenhall, B., Barron, J.T., Dehevec, P:: Baking neural radiance fields for saal-time view synthesis. ICCV (2021)
[26] Hong, Y., Zhang, K., Gu, J., Bi, S., Zhou, Y.. Liu, D., Liu, F., Sunkavalli, K., Bui. T., Tan, Lrm: Large reconstruction model for single image to 3d. arXiv preprint arXiv:2311 11.-233111444000 (2023)
[27] Hu, T., Liu, S., Chen, Y., Shen, T., Jia, J.: Efficientnerf efficient neural radiance fields. In: CVPR (2022)
[28] Huang, B., Yu, Z., Chen, A., Geiger, A., Gao, S.: 2d gaussian splatting for geometrically accurate radiance
fields. In: SIGGRAPH. Association for Computing Machinery (2024)
[29] Huang, Y.H., Sun, Y.T., Yang, Z., Lyu, X., Cao, Y.P., Qi, X.: Sc-gs: Sparse-controlled gaussian splatting for editable dynamic scenes. arXiv preprint arXiv:2312.1492: (2023)
[30] Jiakai, Z., Xinhang, L.., Xinyi, Y., Fuqiang, Z., Yanshun, Z., Minye, W., Yingliang, Z., Lan, X., Jingyi, Y::
Editable free-viewpoint video using layered neural representation. In: SIGGRAPH (2021)
[31] Jiang, Y., Hedman, P., Mildenhall, B., Xu, D., Barron, J.T., Wang, 7., Xue, T.: Alignerf: High-fidelity neural radiance fields via alignment-aware training. arXiv preprint arXiv:2211 100:2211.00982 (2022)
[32] Kavan, L., Collins, S., Zára, J., O' O'Sullivan, C.: Skinning with dual quaternions. In: SI3D (2007)
[33] Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G.: gaussian splatting for real-time radiance field rendering. ACM Trans. Graph. (2023)
[34] Kerhl, B., Meuleman, A., Kopanas, G., Wimmer, M., Lanvin, A., Drettakis, G.: A hierarchical 3d gaussian
representation for real-time rendering of very large datasets. ACM Trans. Graph, (2024)
[35] Kutulakos, K.N., Scitz, S.M.: A theory of shape by space carving. IJCV (2000)
[36] Li, L.. Shen, Z., Wang, Z., Shen, L., Tan, P.: Streaming radiance fields for 3d video synthesis. arXiv preprint arxiv:2210. 14831 (2022)
[37] Li, R., Tanke, J., Vo, M., Zollhofer, M., Gall, J., Kanazawa, A., Lassner, C.: Tava: Template-free animatable volumetric actors (2022)
[38] Li, T., Slaveheva, M., Zollhoefer, M., Green, S., Lassner, C., Kim, C., Schmidt, T., Lovegrove, S., Goesele,
M., Newcombe, R., et al.: Neural 3d video synthesis from multi-view video. In: CVPR (2022)
[39] Li, Z., Niklaus, S., Snavely, N., Wang, O.: Neural scene flow fields for space-time view synthesis of
dynamic scenes. In: CVPR (2021)
[40] Lin, C.H., Gao, J., Tang, L., Takikawa, T., Zeng, X., Huang, X., Kreis, K., Fidler, S., Liu, M.Y., Lin, T.Y.:
Magic3d: High-resolution text-to-3d content creation. In: CVPR (2023)
[41] Lindell, D.B., Martel, J.N., Wetzstein, G.: Autoint: Automatic integration for fast neural volume rendering. In: CVPR (2021)
[42]
Ling, H., Kim, S.W., Torralba, A., Fidler, S., Kreis, K.: Align your gaussians: Text-to-4d with dynamic 3d gaussians and composed diffusion models, arXiv preprint 13X26 (2023)
11

---

[43] Liu. I., Su, H. Wang, X.: Dynamic gaussians mesh: Consistent mesh reconstruction from monocular videos. arXiv preprint arXiv:2404. 12379 (2024)
[44] Liu, J.W., Cao, Y.P., Mao, W., Zhang, W., Zhang, DJ.., Keppo, J., Shan, Y. Qie, X., Shou, M.Z.: Devrf: Fast deformable voxel radiance fields for dynamic scenes, arXiv preprint arXiv:2205. 15723 (2022)
[45] Liu, L.. Gu, J., Zaw Lin, K., Chua, T.S., Theobalt. C.: Neural sparse voxel fields. In: NeurIPS (2020)
[46] Liu, Y., Guan, H. Luo, C., Fan, L.. Peng, J., Zhang. Z.: Citygaussian: Real-time high-quality large-scale
scene rendering with gaussians. arXiv preprint arXiv: 2404.01133 (2024)
[47] Liu, Y.L., Gao, C., Meuleman, A., Tseng, H. Y. Saraf, A., Kim, C., Chuang, Y.Y., Kopt, J., Huang, J.B.: Robust dynamic radiance fields. In: CVPR (2023)
[48] Lombardi, S., Simon, T., Saragih, J., Schwartz. G., Lehrmann, A., Sheikh, Y.: Neural volumes: Learning
dynamic renderable volumes from images. arviv preprint arXiv:1906.0775: (2019)
[49] Lombardi, S., Simon, T., Schwartz, G., Zollhoefer, M., Sheikh, Y., Saragih, J.: Mixture of volumetric primitives for efficient neural rendering. arXiv preprint arXiv:2103.012
(2021)
[50] Long, X., Guo, Y.C., Lin, C., Liu, Y., Dou, Z., Liu, L.,, Ma, Y.. Zhang, S.H., Habermann, M., Theobalt, C., al.: Wonder3d: Single image to 3d using cross-domain diffusion. arXiv preprint arXiv:2310.1500 (2023)
[51] Lu, Y., Zhang, J., Li, S., Fang. T., McKinnon, D., Tsin, Y., Quan, L., Cao, X., Yao, Y: Direct2. 5: Diverse text-to-3d generation via multi-view 2.5 d diffusion. arXiv preprint arXiv:2311 1.15980 (2023)
[52] Luiten, J., Kopanas, G., Leibe, B., Ramanan, D.: Dynamic gaussians: Tracking by persistent dynamic view synthesis. In: 3DV (2024)
[53] Ma, I.., Li, X., Liao, J., Zhang, Q., Wang, X., Wang. Sander, P.V.: Deblur-nerf: Neural radiance fields from blurry images. arXiv preprint arXiv:2111. (2021)
[54] Mildenhall, B., Srinivasan, P.P.. Ortiz-Cayon, R., Kalantari. N.K., Ramamoorthi, R., Ng. R., Kar, A.: Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Trans.
Graph. (2019)
[55] Mildenhall, B., Srinivasan, P.P.. Tancik, M., Barron, J.T.. Ramamoorthi, R., Ng. R.: Nerf: Representing scenes neural radiance fields for view synthesis. In: ECCV (2020)
[56] Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza,
D.,. Massa, F. El-Nouby. A., et al.: Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.077 193 (2023)
[57] Park, K., Sinha, U., Barron, J.T., Bouaziz, S., Goldman, D.B., Scitz, S.M., Martin-Brualla, R.: Nerfies: Deformable neural radiance fields. In: ICCV (2021)
[58] Park, K.. Sinha, U. Hedman, P., Barron, J.T., Bouaziz, S., Goldman, D.B., Martin Martin-Brualla, R., Seitz,
S.M.: Hypemerf: a higher-dimensional representation for topologically varying neural radiance fields. ACM Trans. Graph. (2021)
[59] Peng, S., Yan, Y., Shuai, Q. Bao, H., Zhou, X.: Representing volumetric videos as dynamic mip maps. In: CVPR (2023)
[60] Peng, S., Zhang, Y., Xu, Y., Wang, Q., Shuai, Q. Bao, H., Zhou, X.: Neural body: Implicit neural
representations with structured latent codes for novel view synthesis of dynamic humans. In: CVPR (2021)
[61] Penner, E., Zhang, L.: Soft 3d reconstruction for view synthesis. ACM Trans. Graph. (2017)
[62] Poole, B., Jain, A., Barron, J.T., Mildenhall, B.; Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 14988 (2022)
[63] Pumarola, A., Corona, F.,, Pons-Moll, G., Moreno-Noguer, F: D-nerf: Neural radiance fields for dynamic scenes. In: CVPR (2021)
[64] Rebain, D., Jiang, W., Yazdani, S., Li, K.. Yi, K.M., Tagliasacchi, A.: Derf: Decomposed radiance fields. In: CVPR (2021)
[65] Reiser, C., Peng. S., Liao, Y., Geiger, A.: Kilonerf: Speeding up neural radiance fields with thousands of tiny mips. In: CVPR (2021)
[66] Reiser, C., Szeliski, R., Verbin, D., Srinivasan, P.P., Mildenhall, B., Geiger. A., Barron, J.T., Hedman, P.: Merf: Memory-efficieee radiance fields for real-time view synthesis in unbounded scenes. arXiv preprint arXiv:2302.1 12249 (2023)
[67] Riegler, G., Koltun, Free view synthesis. In: ECCV (2020)
[68] Sara Fridovich- Krddovich-Keilee and Giacomo Meanti, Warburg, F.R., Recht, B., Kanazawa, A.: K-planes: Explicit radiance fields in space, time, and appearance. In: CVPR (2023)
12

---

[69] Shao, R.,. Zheng, Z. Tu, H. Liu, B.. Zhang, H. Liu. Y.: Tensor4d: Efficient neural 4d decomposition for high-fidelity dynamic reconstruction and rendering. In: CVPR (2023)
[70] Shuai, Q. Guo, H., Xu, Z., Lin, H., Peng. S., Bao, H., Zhou, X.: Real-time view synthesis for large scenes with millions of square meters (2024)
[71] Singer, U., Sheynin, S., Polyak, A., Ashual, O., Makarov, I., Kokkinos, F., Goyal, N., Vedaldi, A., Parikh,
D., Johnson, J., et al.: Text-to-4d dynamic scene generation. arXiv preprint arXiv:2301. 1280 (2023)
[72] Sitzmann, V., Thies, J.. Heide. F., Nießner, M., Wetzstein, G., Zollhofer, M.: Deepvoxels: Learning persistent 3d feature embeddings. In: CVPR (2019)
[73] Song, L., Chen, A., Li, Z.. Chen, Z., Chen, L., Yuan, J., Xu, Y.. Geiger, A.: Nerfplayer: A streamable dynamic scene representation with decomposed neural radiance fields. arXiv preprint arX1v:2210.15947 (2022)
[74] Srinivasan, P.P., Mildenhall, B., Tancik, M., Barron, J.T., Tucker, R., Snavely, N.: Lighthouse: Predicting
lighting volumes for spatially-coheren illumination. In: CVPR (2020)
[75] Srinivasan, P.P.. Tucker, R., Barron, J.T., Ramamoorthi, R., Ng. R., Snavely, N.: of view extrapolation with multiplane images. In: CVPR (2019)
N.: Pushing the boundaries
[76] Su, S.Y., Yu, F., Zollhöfer, M., Rhodin, H.: A-nerf: Articulated neural radiance fields for learning human shape, appearance, and pose. In: NcurIPS (2021)
[77] Tang, J., Chen, Z., Chen, X., Wang, T. Zeng. G., Liu, Z.: Lgm: Large multi view gaussian model for high-resolution 3d content creation. arviv preprint arXiv:2402.05004 (2024)
[78] Thies, J., Zollhöfer, M., Nießner, M.: Deferred neural rendering: Image synthesis using neural textures. ACM Trans. Graph. (2019)
[79] Tretschk, E., Tewari, A., Golyanik, V., Zollhöfer, M., Lassner, C., Theobalt, C.: Non-rigid neural radiance fields: Reconstruction and novel view synthesis of dynamic scene from monocular video. In: ICCV (2021)
[80] Tucker, R., Snavely, N.: Single-view view synthesis with multiplane images. In: CVPR (2020)
[81] Voleti, V., Yao, C.H., Boss, M., Letts, A., Pankratz, D., Tochilkin, D., Laforte, C., Rombach, R., Jampani, V.: Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion. arXiv preprint arXiv: 1403.12008 (2024)
[82] Waechter, M., Moehrle, N., Goesele, M.: Let there be color! large-scale texturing of 3d reconstructions. In: ECCV (2014)
[83] Wang, F., Chen, 7., Wang, G., Song, Y., Liu, H.: Masked space-time hash encoding for efficient dynamic scene reconstruction. In: NeurIPS (2023)
[84] Wang, F., Tan, S., Li, X.. Tian, Z., Liu, H.: Mixed neural voxels for fast multi-view video synthesis. arXiv preprint arXiv:2212.0011 (2022)
[85] Wang, H., Ren, J.. Huang. Z., Olszewski, K., Chai, M., Fu, Y. Tulyakov, S.: R21: Distilling neural radiance field to neural light field for efficient novel view synthesis. In: ECCV (2022)
[86] Wang, I.., Zhang, J., Liu, X., Zhao, F., Zhang, Y., Zhang, Y., Wu, M., Yu, J., Xu, L.: Fourier plenoctrees for dynamic radiance field rendering real-time. In: CVPR (2022)
[87] Wang. P., Liu, L., Liu, Y.. Theobalt, C., Komura. T., Wang, W.: Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. In: NeurIPS (2021)
[88] Wang, X., Wang, Y., Ye, J., Wang, Z., Sun, F., Liu, P., Wang, L., Sun, K., Wang, X., Hc, B.: Animatable- dreamer: Text-guided non-rigid 3d model generation and reconstruction with canonical score distillation. arXiv preprint arXiv:2312.03777 (2023)
[89] Wang, Y., Dong, Y., Sun, F,, Yang, X.: Root pose decompasition towards generic non-rigid 3d reconstruc- tion with monocular videos. In: ICCV (2023)
[90] Wang, Z., Lu, C., Wang, Y., Bao, F., Li, C.. Su, H.. Zhu, J.: Prolificdreamer: High-fideli and diverse text-to-3d generation with variational score distillation. In: NeurIPS (2023)
[91] Wang, Z., Wang, Y., Chen, Y., Xiang, C., Chen, S., Yu, D., Li, C., Su, H., Zhu, J.: Crm: Single image to 3d textured mesh with convolutional reconstruction model. arXiv preprint arXiv:2403.055 .05034 (2024)
[92] Wang, Z., Li, L.. Shen, Z., Shen, L., Bo, L.: 4k-nerf: High fidelity neural radiance fields at ultra high resolutions. arXiv preprint arXiv:2212.04701 (2022)
[93] Wood, D.N., Azuma, D.I., Aldinger, K., Curless, B., Duchamp, T., Salesin, D.H., Stuetzle, W.: Surface
light fields for 3d photography. In: Proceedings of the 27th annual conference on Computer graphics and interactive techniques (2000)
13

---

[94] Wu, G.. Yi, T., Fang. J. Xie, L., Zhang, X.. Wei, W. Liu, W. Tian, Q. Xinggang, W.: 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint arXiv:2310.08528 (2023)
[95] Xian, W., Huang, J.B., Kopf, J., Kim, C.: Space-time neural irradiance fields for free-viewpoint video. In: CVPR (2021)
[96] Y., Shi, Z.. Yifan, W., Peng. S., Yang. Shen, Y. Gordon. W.: Grm: Large gaussian reconstruction
model for efficient 3d reconstruction and generation. arXiv preprint arXiv: 2403.14621 (2024)
197] Yang, G., Sun, D., Jampani, V., Vlasic, D., Cole, F., Chang, H., Ramanan, D., Freeman, W.T., Liu, C.: LASR: learning articulated shape reconstruction from a monocular video. In: ICCV (2021)
[98] Yang, G., Vo, M., Neverova, N., Ramanan, D., Vedaldi, A., Joo, H.: Banmo: Building animatable 3d
neural models from many casual videos. In: CVPR (2022)
[99] Yang, G., Wang, C., Reddy, N.D., Ramanan, D.: Reconstructing animatable categorics from videos. In: CVPR (2023)
[100] Yang, Z., Gao, X., Zhou, W. Jiao, Zhang, Y., Jin, X.: Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. arXiv preprint arXiv:2309.133 13101 (2023)
[101] Yariv, I.., Kasten, Y., Moran, D., Galun, M., Atzmon, M., Basri, R., Lipman, Y.: Multiview neural surface
reconstruction by disentangling geometry and appearance. In: NeurIPS (2020)
[102] Yu, A., Li, R., Tancik, M., Li, H., Ng, R., Kanazawa, A.: Plenoctrees for real-time rendering of neural radiance fields. In: CVPR (2021)
[103] Zhang, R., Isola, P. Efros, A.A., Shechtman, E., Wang, O.: unreasonable effectiveness deep
features as a perceptual metric. In: CVPR (2018)
[104] Zhao, F., Yang. W. Zhang. J., Lin, P., Zhang, Y. Yu, J., Xu, L.: Humannerf: Efficiently generated human
radiance field from sparse inputs. In: CVPR (2022)
[105] Zhou, T., Tucker, R., Flynn, J., Fyffe, G., Snavely, N.: Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint arXiv: arXiv:1805.0981 805.09817 (2018)
[106] Zou, Z.X., Yu, Z., Guo, Y.C., Li, Y., Liang, D., Cao, Y.P., Zhang, S.H.: Triplane meets gaussian splatting:
Fast and generalizable single-view 3d reconstruction with transformers. arXiv preprint arXiv:23 12.02331 (2023)
14

---

A Appendix Supplemental Material
Table 2: A summary of important symbols in DGS.
Symbel
Defisition and Usage
4 e (RX) 440 RRX))
Principal langeniaal vectors in the static stale:.
4, F FR F.
Scaling factors the static state.
PI gixt
Conter point couninttec (world space) k-th Goussion surfel in
(u) gast
Coordinati (world space) the static
= (u. the local we coordinate system contered př.
R; = [62. 12 tj] 80(3)
Rotation mutris of the 1-th Gaussinn surfel in the static state.
Sİ dinglt (). 32x
Scaling mairis the k-ih Gaussian surfol in static state, diugonal matrix.
P R
Center point coordinate (world space)
Gaussian surfel warped state.
Pi(M) gaxt
Coordinate (world space) the warped given (a.e) the local coordinate system centered
e XXX), V. e exre 1999,A Al e R
Center, rotation matrix, and diagonal scaling matrix of the 6-th Caussian ellipsoid bone.
w' 12
Skimmig weight veciors.
J e SE(3)
A rigid transoomation that moves the b-th bone from its static state to the warped state time
[R] 880)
The warping lunction, weighted combitation of
Q.
The quatermion procees and the inverse quaternion procees.
w R:26
A leumable latent code for representing the body at time 1,
ny RAxI
The normal of the k-intersecied Goussian surfel that is oriented towands the camera.
RAKL
surface normal estimated by the nearhy depth point p' at warped state time
AR; 5O(3) AS; SO(3)
Leamable refinement term adjusting R Leurable refinement term for adjusting S
A.1 Details of Skinning Representation
As mentioned in the main paper, the warping process from the static state to the warped state is modelled as time-varying non-rigid warping function with B bones to be key points. In the static state, the b-th bone is represented by 3D Gaussian ellipsoids [97] with the center oj E R rotation matrix Vž E 13x3 and diagonal scaling matrix A₃ E 3x333 For a 3D point Px (u), the skinning weight vectors w E Bx1 at time / is calculated by the normalized Mahalanobis distance following [98]
mB = (P)(u) cg)) T (PE -bb); w' Jsofmmax (m my..... .., ,,m.)
(11)
where mi denotes the squared distance between (u( and the b-th bone; cj E 3xx1 is the center of the b-th bone at time t. and Qj Vi V is the precision matrix composed by the bone orientation matrix Vi E 3R333 at time / and AT- Specifically, there is (Vgle') JI with of Vž, and Ag being learnable parameters. asoftmax is the softmax function.
A.2 Ablation Studies of Field Initialization and Refinement
In dynamic videos captured in the wild, one of the primary challenges is the initialization of camera poses. In synthetic videos, preserving temporal consistency in texture and geometry is problematic, which significantly complicates the task of camera registration. To address this, we utilize an implicit field to both initialize the camera poses and establish the warping field. Initially, transformation for each frame, followed by the computation of coarse camera poses through an adopt the approach outlined in NeuS [87] for scene representation.
estimate the
iterative process. Subsequently, Feature extraction is performed using Dino V2 [56], facilitating unsupervised registration. To enhance this process, train an additional channel in NeuS specifically for rendering features, which then employed for registration purposes as described RÁC [99]. The camera poses without initialization and refined camera poses are depicted in Fig. 6. Without field initialization, the performance of DGS will degrade, as shown in Table 3. Also, please refer to the quantitative ablation of refinement in Table 3.
A.3 Additional Qualitative Comparison
In this section, we present detailed comparison of our results with previous works, as illustrated in Fig. 7-10. Our method consistently achieves high-quality texture details while maintaining smooth and realistic geometry.
15

---

Table 3:
Quantitative ablation studies of the initialization and refinement.
Cat
Cheetah
Dragon
PSNR 1
SSIM f
LPIPS
PSNR 1
SSIM f LPIPS
PSNR
SSIM f
LPIPS 4
Ours w.o.
20.15
0.7961
0.2393
20.96
0.8194
0.1940
25.33
0.9146
0.0938
Ours w.o. refinement
24.19
0.8196
0.1797
24.10
0.8582
0.1242
27.71
0.9128
0.0687
Ours full
24.63
0.8432
0.1559
25.68
0.8843
0.1117
28.58
0.9392
0.0618
(a) Camera poses without field initialization
(b) Refined camera poses
Figure 6: Coarse camera poses and refined camera poses.
A.4
Interpolation on Time and Views
We present results for interpolation on time and views, as illustrated in Fig. 11 and Fig. 12.
A.5 Broader Impact
Generative models used in video generation might pose risks, for example, the potential for creating deepfakes or other misleading content that could used for harmful purposes like misinformation, privacy invasion, defamation. To mitigate these risks, have chosen to release only the recon- struction code, deliberately avoiding the release of components that could facilitate the generation of content with ethical concerns. This decision ensures that contribution is focused advancing reconstruction techniques without enabling the creation of new, potentially harmful video content.
Besides, we have carefully considered the potential ethical risks associated with generative models, particularly video content creation. To address these concerns, our model includes robust safety mechanisms designed to screen and prevent any misuse. We believe these measures effectively mitigate potential ethical risks and align with the standards the community.
16

---

BANKS
D Delemaalle 6S
Delemaal GS W. Pases
SO
Figure 7: Additional qualitative comparison with more novel views.
17

---

BANKS
Detomacle'ts
DefomableCS GS w. Pases
SOGS hooe
Figure 8: Additional qualitative comparison with more novel views.
18

---

Detomaale 2S
Delomaace USM Puree
Figure 9: Additional qualitative comparison with more novel views.
19

---

BANKs
Detomacle C5
Defomable G5 w. Pose
SOGS Poses
Figure 10: Additional qualitative comparison with more novel views.
20

---

on Fig Figure 11: Interpolation on time and andde vic
views. .
21

---

Figure 12: Interpolation on time and views.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification: Our main claims in the abstract and introduction accurately reflect the contri- butions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
23

---

Justification:
We build our proof upon papers cited.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All relevant details are provided within the text of our paper or through the references we have cited.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer: [Yes]
Justification:
We will release it after acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https: petttp:://iis....e cc/
public/ geides/
des CodeSu'm ss Moooolioiyyy cy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nips. cc/ cec/pbblice c/ggi des / /Codeeeoubb ss onPol lcy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: All relevant details are provided within the text of our paper or through the references we have cited.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Please refer to our Tab. 1.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
25

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Sec. 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ete:://neurrt /neuriisss cc httpp//c/ppccccccccccccttataaaaa
Answer: [Yes]
Justification: The NeurIPS Code of Ethics is conformed.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss it in Sec. 5.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the assets properly.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papers' raw w thcode com/ data sets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
27

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: No new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA].
Justification: No human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
Justification: No human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "wLE2LhPQk6", "reviewer": "Reviewer_crn6", "summary": "The paper proposes a technique called Dynamic Gaussian Surfels to effectively reconstruct 4D reqpresentation from a single generated video. DGS optimizes time-varying warping functions to transform Gaussian surfels and the authors adopt Neural SDF for initialization and proposes a geometry regularization technique to preserve the geometry integrity. Extensive experiments on 30 objects proves the effectiveness of the proposed method.", "questions": "See weaknesses. I might adjust the score according to the response from the authors.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "a. The resutls are great, show promising application of the proposed DGS.\n\n  b. The paper is well writting and easy to follow.\n\n  c. The authors compare their method with various 4D representations, i.e., skinning and bones, NeRF, Gaussian, and achieve better performance.", "weaknesses": "a. The novelty is limited. This work seems to be a combination of LARS (bones and warping), Gaussian Surfels/2DGS (3D representation) and SC-GS/4DGS (refinement). The real novelty might be the geometric regularization and field initialization, although the former is also similar to that used in previous 3D works.\n\n  b. Lack of evaluation details: The authors evaluated the comparison methods on generated video for novel view synthesis in Table 1, however, there is no gt for generated video's novel view resutls. How did the author conduct the evaluation?\n\n  c. The experiments is not extensive enough: The authors claim that their methods are designed for generated video, however, I didn't see any special designs, e.g., sovle the potential multi-view inonconsistency in the input video. So, I think it is a general 4D reconstruction method, and it is recommanded to test the proposed methods on commanly used 4D reconstruction datasets (object and scene-level), using standard evaluation metrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your valuable comments and insightful suggestions. We address all your comments below.\n\n**1. Novelty:** To the best of our knowledge, our method is the first to generate 4D content using a text-to-video model. We primarily focus on addressing the spatial-temporal inconsistencies in geometry and texture in generated videos, which is a novel question. Previous works, like SV3D and V3D, focus on enhancing the consistency of video diffusion. This approach is challenging and may lose the knowledge learned from high-quality videos and images when fine-tuning on videos rendered by 3D assets. Instead, we improve the tolerance of the reconstruction method to deal with this inconsistency. Specifically, we absorb multiview inconsistencies into the global motion of bones and unstable textures into the local motion of surfels. We select DGS of a specific frame when we want to obtain statistical meshes or 3DGS. We believe this design is novel.\n\nOur dynamic Gassian surfel is novel. Compared with LARS, we design additional neural skinning weights and a registration technique using DINO features and an invertible warping function. Please refer to Sec. 3.2 and Fig. 6. Compared with Gaussian Surfels/2DGS, we design a warped-state normal regularization to enhance the surface across all frames. Also, we elevate Gaussian Surfels to 3D dimensions for better visual quality, as shown in Fig. 5(c). Compared with SC-GS/4DGS, we applied a relatively simple but plausible warping model containing only 25 anisotropy control bones. This simplification, in contrast to the 500 control points of SC-GS and the dense motion field of 4DGS, helps to regularize motion and prevent overfitting, especially when the camera poses are inaccurate and the object's multi-view consistency cannot be assured, as shown in Fig. B. This method of mitigating Gaussian overfitting is also robust against noise or floater occlusion, as illustrated in Fig. C, where the mask of the dragon covered by sand is missing a piece.\n\nWe also design a novel refinement to deal with flickering textures in generated videos. When Gaussian surfels are well reconstructed, its' normal is aligned with the surface normal, and this makes the gradient of density along the surface normal very large:\n$$\\mathbf{G}(\\mathbf{u})=\\text{exp}(-\\frac{u^2+v^2}{2}), \\frac{d}{dx} \\mathbf{G}(\\mathbf{u}) = -\\left(\\frac{1}{\\sin^2(\\theta)} + \\frac{1}{\\sin^2(\\gamma)}\\right) x \\exp\\left(-\\frac{\\left(\\frac{1}{\\sin^2(\\theta)} + \\frac{1}{\\sin^2(\\gamma)}\\right) x^2}{2}\\right)\n$$\nwhere $\\mathbf{G}(\\cdot)$ is the density of surfel, $x$ goes along the surface normal, $\\theta$ and $\\gamma$ are the angles between u, v, and the surface, respectively. Considering that $\\theta$ and $\\gamma$ are very small, the density is sensitive to the motion along the surface normal. Consequently, this direct of gradient leads the flickering of the texture to be modeled as the position of the surfels moving back and forth over time. When the front-back relationship between the rear surfels and the surface surfels changes, the texture flickers accordingly.\n\nTo alleviate this flickering, we introduce a refinement stage that elevates surfels to Gaussian ellipsoids. Compared to surfels, 3D ellipsoids have a smoother density field, and provide a more robust representation during warping, reducing the impact of flickering. In addition, $\\Delta \\textbf{R}^*_k$ and scaling $\\Delta \\textbf{S}^*_k$ defined in Eq.8, remove the constraint of aligning the shortest axis with the surface normal. This flexibility makes the 3DGS less likely to unintentionally introduce texture flickering during motion. This is illustrated in Fig. E. \n\n**2. Evaluation details:** We follow the standard evaluation protocol for 4D reconstruction methods such as Hyper-NeRF and SC-GS. Specifically, we build the dataset by using every 4th frame as a training frame and taking the middle frame between each pair of training frames as a validation frame. We will provide additional details in the revised manuscript.\n\n\n**3. Special designs for generated videos:** Please refer to Q2 in the common response for more details.\n\n**4. Experiments on 4D reconstruction datasets:** We follow your advice and add more experiments on the commonly used 4D reconstruction datasets. Please refer to Q2 in the common response for more details.\n\n\nOnce again, thank you for your constructive feedback and for considering our paper for acceptance. We'll revise our paper according to your suggestions. We kindly request that you consider raising the score accordingly if we addressed your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UnNPY4ILnG", "reviewer": "Reviewer_1Eme", "summary": "Video generation models have shown great power recently. Transforming generated videos into 3D/4D representations is important for building a world simulator. This paper proposes an improved 4D reconstruction method from single-generated videos. The key component is the dynamic Gaussian surfels (DGS) technique. Incorporating an initialization stage of a non-rigid warping field, the Vidu4D method produces impressive 4D results with the video generation model Vidu.", "questions": "1. The provided results are all object-level 4D ones. Will this method work well on scene-level samples? I know it will be hard to generate the surfels of the background. \n2. What about the mesh and depth of the generated 4D representations?\n3. One missing related work: Liu I, Su H, Wang X. Dynamic Gaussians Mesh: Consistent Mesh Reconstruction from Monocular Videos[J]. arXiv preprint arXiv:2404.12379, 2024.", "limitations": "The paper has clearly addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The topic is valuable and interesting to transforming single generated videos into 3D/4D representations. The built 3D/4D representation is more controllable and explicit than a single video. Thus it can be used for rendering more videos with elaborate and customized camera trajectories. Besides, this technique has the potential to be a key component for building a world simulator from video generation models.\n2. The provided 4D results show impressive rendering quality, reaching the SOTA performance of this/related field. Besides, the normal looks good, revealing the advantage of modeling geometry from the proposed representation.", "weaknesses": "1. The motivation/necessity of building surfels needs to be further strengthened. It is easy to understand building surfels will undoubtedly help reconstruct the surface and geometry. If just considering rendering videos from the built 4D, will a vanilla 4D representation (without improvement on surface reconstruction) be enough? Fig. 4 and Table 1 provide convincing results. However, it is suggested to make it clearer in introducing the motivation, e.g. why better geometry leads to better synthesis.\n2. The organization of the method could be improved. For better understanding, it is suggested to first introduce the overall framework of Vidu4D and then demonstrate the dynamic Gaussian Surfels technique.\n3. The main method is more like a basic representation/reconstruction approach. Will it still benefit reconstruction from monocular videos captured in real life, not generated videos? Yet, real-life videos have better consistency than generated ones.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your valuable comments and insightful suggestions. We address all your comments below. If our response has addressed the concerns and brings new insights to the reviewer, we will highly appreciate it if the reviewer considers raising the score.\n\n**1. Motivation/necessity of building surfels:** Surfels improve the quality of surfaces, which in turn allows for the extraction of high-quality meshes, as illustrated in Fig. F. Furthermore, detailed geometry can enhance downstream applications, such as Gaussian-guided frame continuation at a specific camera pose, as demonstrated in Fig. I of our rebuttal PDF file, where a good depth/normal largely improves the continuation performance.\n\n**2. Organization of the method:** Thank you for the constructive feedback. We will reorganize the methodology section to first introduce the overall framework of Vidu4D, followed by a detailed demonstration of the DGS technique. As discussed in Q2 of the common response, we will provide a more detailed description and analysis of Vidu4D. We hope this adjustment will enhance the clarity and flow of our presentation.\n\n**3. Real monocular videos:** We evaluate our method on realistic scene-level benchmarks (Neural 3D Video dataset and NeRF-DS dataset). Please refer to Tab. C, Tab. D, Fig. D, and Fig. G of the rebuttal PDF file for experiments on real monocular videos.\n\n**4. Meshes and depth:** Please refer to Fig. F for reconstructed meshes and depth.\n\n**5. Missing related work:** Thank you for pointing out the missing related work. We will include a discussion of Liu et al.'s (2024) \"Dynamic Gaussians Mesh: Consistent Mesh Reconstruction from Monocular Videos\" in our revised manuscript to ensure comprehensive coverage of related research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5toYdZPNDe", "reviewer": "Reviewer_7btN", "summary": "The paper presents Vidu4D, a  reconstruction model that excels in accurately reconstructing 4D (i.e., sequential 3D) representations from single generated videos, addressing challenges associated with non-rigidity and frame distortion. At the core of Vidu4D is a proposed Dynamic Gaussian Surfels (DGS) technique. DGS optimizes time-varying warping functions to transform Gaussian surfels (surface elements) from a static state to a dynamically warped state. This transformation enables a precise depiction of motion and deformation over time. To preserve the structural integrity of surfacealigned Gaussian surfels, the authors design the warped-state geometric regularization based on continuous warping fields for estimating normals. Additionally, the method learns refinements on rotation and scaling parameters of Gaussian surfels, which alleviates texture flickering during the warping process and enhances the capture of fine-grained appearance details.", "questions": "Besides above problems, I think the below questions are needed to be addressed:\n\nWhat kind of generated video is used as reference video (better to show the reference mono video). Since for 4d reconstruction, if the reference video doesn't show some parts that are occluded across all frames, it is impossible to reconstruct them. An alternative is to use diffusion prior for novel view supervisions to hallucinate these parts, so what is actually happening here????", "limitations": "I'm confident it will be horrendously challenging for average readers to fully grip the hole picture without a major revision.\nThe paper is written to focus on 4d generation task or lift monocular generated video to 4d, however, 80% of the sections are dedicated to representation. If the author want to present this paper as a sknning-based dynamic 2dgs representation paper, then showing 4d generation only is not enough, 4d reconstruction (w/ many widely used benchmarks) should be used as well.\n\nBeside, I hope the author can re-organize the paper no matter it is accepted or not. It is better to put some of the cluttered annotations into appendix, and leave some room for actual pipeline of Vidu4D. The Banmo like joint representation and skinning of 2DGS could be fairly straight forward for people in the field to understand, so no need to put all details in the main paper.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "1. The paper shows interesting visual results, although the anonymous link in the submission pdf seems not working correctly.\n\n2. The paper proposes a Banmo-based dynamic 2dgs formulation for 4d reconstruction.", "weaknesses": "1. The paper's annotation is cluttered and extremely hard to follow. Sometimes ignoring some symbols in formulation is desirable when too many of them are presented.\n\n2. The real framework section 3.3 is extremely short. The \"more details in our appendix\" seems to be a false promise? \n\n3. The model hasn't evaluated on realistic scenes, so that it can be compared with other 4dgs methods using their official results.\n\n4. The paper uses banmo like learnable joint representation to drive deformation, however hasn't mentioned the limitations of this kind of methods, i.e., the 4d scene needs to be object centric.\n\n5. No evaluations of latency, it seems the pipeline needs a dynamic neus/nerf, then init 2dgs at the extracted zero level set, which makes the methods very dependable of the stage one and very time consuming.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your valuable comments and insightful suggestions. We address all your comments below.\n\n**1. Anonymous link:**\nThank you for your feedback. We believe the issue with the anonymous link might be due to a temporary network problem and we appreciate it if you could try again. Other reviewers might have successfully accessed the videos through the link. If the link remains inaccessible, please refer to Fig. 11-12 in the supplementary and Fig. J for more results.\n\n**2. Annotation and presentation:**\nThank you for your valuable advice. We acknowledge the need to enhance the clarity of our manuscript. To make our work easier to follow, we will simplify our annotations in future revisions. Additionally, we plan to release our code to aid in the replication and understanding of our research. While reviewers crn6 and yVYk mentioned that our manuscript is well-written and easy to follow, we recognize that there is some room for improvement. We will improve the overall presentation and ensure that our findings are communicated as clearly as possible.\n\n**3. Framework details:**\nIn Supp. A, we provide more detail about Vidu4D. Here we also provide more details of initialization and refinement.\n\nIn initialization, we first segment the foreground using TrackAnything. Then we extract the unsupervised features with DINOV2, optical flow with VCN [Learning to segment rigid motions from two frames], and metric depth with ZoeDepth. Considering the consistency of the generated video is limited, we register pair-wise camera poses using mask, depth, and flow. Then, we register the root pose of objects using DensePose and initialize the SDF field and warping field with volume rendering. During this process, we also refine the camera poses and root poses using a combination of photometric loss. To enhance registration with unsupervised features, we train an additional channel in NeuS specifically for rendering DinoV2 features, which are then employed for registration purposes as described in RAC. Compared with rasterization, the sampling strategy and continuity of volume rendering make it more suitable for refining poses.\n\nFor NeuS rendering, we backward warp sampling points in camera space $\\mathbf{X}^t$ to canonical space $\\mathbf{X}^*$:\n$$\\mathbf{J}^{t, -1} = \\mathcal{R}\\Big(\\sum_{b=1}^{B} w_{b}^{t} \\mathcal{Q}(\\mathbf{J}^t_{b})^{-1}\\Big), $$\n$$\\mathbf{X}^t = \\mathbf{J}^{t, -1} \\mathbf{X}^{*},$$\n\nwhich is an inversion of Eq. 4 in the main paper. By querying the SDF with $\\mathbf{X}^*$, we render RGB and compute the photometric loss to optimize the SDF and the warping field defined in Eq. 6 of the main paper. However, there are two gaps between NeuS warping and DGS warping. First, the sampling points of NeuS are distributed in the frustum of the camera, while the DGS are distributed on the surface. Additionally, we train the inversion of $\\mathbf{J}^t$ during initialization, while we utilize the non-inverse ones in DGS. To resolve the distribution gap and ensure that $\\mathbf{J}^t$ faithfully models the forward warping, we add a cycle loss:\n$$\n\\mathcal{L}_{\\mathrm{cyc}} = \\|\\| \\mathbf{J}^t \\( \\mathbf{J}^{t, -1} \\( \\mathbf{X}^{t} \\) \\) - \\mathbf{X}^{t} \\|\\|^{2},\n$$\nwhere $\\mathbf{X}^{t}$ are a combination of mesh surface points and ray sampling points.\n\nAfter initialization, we extract the mesh in canonical space with the marching cube and initialize Gaussian surfels on the mesh. We set the spherical harmonic in 0-th order to the RGB value of the nearest vertices. Here we keep the warping field and the learned camera poses.\n\n**4. Evaluated on realistic scenes and benchmarks:**\nPlease refer to the **common response** for more benchmarks. Specifically, we provide quantitive and qualitative comparisons on realistic scene-level benchmarks (Neural 3D Video dataset and NeRF-DS dataset) and object-level benchmarks (D-NeRF dataset).\n\n**5. Limitation of joint representation:**\nOur method is not limited to reconstructing object-centric scenes. Please see Fig. D and Fig. G in our rebuttal PDF.\n\n**6. Latency:**\nOn an Nvidia A800 GPU, generating a 1080p video takes approximately 10 minutes. Preprocessing requires around 12 minutes, initialization takes another 10 minutes, and reconstruction takes 30 minutes. Rendering a 1080p image takes less than 0.1 seconds. We provide rendering latency in Tab. A on the Neural 3D Video benchmark, indicating the superiority of our rendering latency.\n\n**7. Reference video:**\nDue to the submission guidelines, we are unable to upload videos. Instead, we have provided some frames from our reference videos in Fig. K.\n\n**8. Occluded parts:**\nThank you for your insightful comment. The generative capabilities of Vidu4D are indeed derived from the video generation model. If certain parts of an object are not visible in the reference video, those parts cannot be reconstructed. However, we have found that using the rendered results to guide the video generation model allows for training-free view completion. This approach leverages the strengths of the video generation model to hallucinate and fill in the occluded parts, thereby enhancing the reconstruction process, as shown in Fig. I.\n\nOnce again, thank you for your constructive feedback. We'll revise our paper according to your suggestions, move the skinning representation to the supplementary material, and add more details about the Vidu4D pipeline. We will open-source our code to facilitate understanding for readers. We kindly request that you consider raising the score accordingly if we addressed your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dIYCBhf4z9", "reviewer": "Reviewer_yVYk", "summary": "The paper presents Vidu4D, a  reconstruction model that can accurately reconstruct 4D (sequential 3D) representations from single generated videos. This method addressing key challenges and enabling high-fidelity virtual content creation. The proposed techniques, such as Dynamic Gaussian Surfels (DGS) and the initialization state, are good contributions that can benefit the field of multi-modal generation and 4D reconstruction.https://openreview.net/", "questions": "The paper does not discuss the computational complexity or runtime performance of Vidu4D, which could be an important consideration for practical applications of the method.", "limitations": "The paper does not mention how the Vidu4D  can be extended or adapted to handle other common challenges in 4D reconstruction, such as occlusions, lighting changes, or complex scene dynamics.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written.\n2. The qualitative results outperform existing methods.\n3. The proposed Dynamic Gaussian Surfels (DGS) approach sounds good. It optimizes time-varying warping functions to transform Gaussian surfels from a static to a dynamically warped state, precisely depicting motion and deformation over time.", "weaknesses": "- What is your video foundation model? Is it Stable Video Diffusion, SORA, Open SORA, or Your Vidu? \n\nIf you are using an unreleased foundation video model, is the improvement in qualitative results more due to DGS, or is it caused by the video foundation model? If you utilize Vidu, I believe the author needs to provide the results using open source video foundation models like SVD or Open-SORA.\n\n- The quantitative evaluation in the paper is limited to a small set of generated videos. \n\n- The performance of Vidu4D on a more diverse and larger dataset of generated videos is not reported, which could limit the generalizability of the findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your valuable comments and insightful suggestions. We address all your comments below. If our response has addressed the concerns, we will highly appreciate it if the reviewer considers raising the score.\n\n**1. Foundation model:** Our foundation model is Vidu. We believe the qualitative results are largely attributed to DGS. We compare the results of other dynamic NeRF and 3DGS in Fig. 4, and our DGS demonstrates state-of-the-art performance.\n\n**2. Open source video foundation models:** For comparison with open-source video generators, we found that generating plausible videos using our specific prompts is challenging for these open-source tools. Instead, we use ToonCrafter to interpolate two frames from our generated video for a fair comparison. It can be observed that, although the generated video lacks consistency, our reconstruction remains relatively stable. Please refer to Fig. I.\n\n**3. More results:** Please refer to Fig. J for results with more styles and categories.\n\n**4. Runtime:** As discussed in Sec. 4.1, for each reconstruction, the overall training takes over 1 hour on an A800 GPU. Specifically, generating a text-guided video takes 4 minutes, preprocessing takes 10 minutes, initialization takes 15 minutes, and the DGS stage takes about 30 minutes.\n\n**5. Challenges in 4D reconstruction:** Please refer to the Motion regularization part in the common response for our discussion about occlusions. For lighting changes, it is promising to introduce a time-dependent spherical harmonic function for Gaussian surfels. To handle complex scene dynamics, such as fluids or flames, adding more control bones or incorporating a dense MLP motion field should be an effective approach.\n\n**6. Benchmark:** Please refer to Q1 in the common response for more quantitative benchmarks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers's efforts as well as very detailed and insightful suggestions. We find there are common concerns to our paper, and we'd like to clarify them here. 

We also add a **PDF file** with more experiment results and visualizations.

_**Q1: From the 4D reconstruction perspective, evaluations on common 4D reconstruction datasets including real scenes and objects? (from crn6, 1Eme, 7btN)**_

Per the reviewers' suggestion, we provide detailed results in our attached PDF file both quantitively and qualitatively, on realistic scene-level benchmarks (Neural 3D Video dataset and NeRF-DS dataset) and object-level benchmarks (D-NeRF dataset). Please See Table A, Table B, Table C, and Table D, and the visualizations in Figure A, Figure D, and Figure G of the PDF file. 

**Detailed settings:**
-  We use PSNR, DSSIM, and LPIPS as evaluation metrics and follow the standard setting of SOTA methods to perform training and evaluation. For object-level D-NeRF data and scene-level NeRF-DS data, we train the model for 80000 iterations and start to perform normal regularization on the 40000th iteration. Specifically, on D-NeRF, we perform a group of experiments when ground truth camera poses are unavailable. We train our field initialization stage (Figure 3 of our main paper) for 2000 iterations which takes 10 minutes. 

- For the scene-level Neural 3D Video data, we follow the standard setting to train at the resolution $1352\times1014$ with 300 frames per scene. We train the model 30000 iterations with the normal regularization adding from the 10000th iteration. 


**Results:**

- Object-level 4D benchmark: From the D-NeRF experimental results in Table A (without GT poses), we observe that our DGS surpasses the second-best method by a large margin (29.06 vs. 20.05 for PSNR).  When given GT poses, our method still outperforms SOTA methods. Our method is especially superior at dynamic normals as shown in Figure A.

- Scene-level (realistic) 4D benchmark: From the Neural 3D Video and NeRF-DS experimental results in Table C and Table D, our DGS achieves the best or the second-best performance in capturing color and shows great superiority in modeling dynamic normals according to the visualizations in Figure D and Figure G.

In summary, for the 4D reconstruction benchmarks, our proposed method demonstrates superior performance compared to existing approaches, particularly in terms of geometric quality and handling unposed scenes.

_**Q2: From the general framework (generated video to 4D) perspective, further strengthened motivation and novelty? (from crn6, 1Eme)**_

**A:** We believe our work has a strong motivation and novelty in terms of the general framework and also special designs for 4D reconstruction from generated videos.  

- The overall framework: To the best of our knowledge, our method is the first to generate 4D content using a text-to-video model. The framework of 4D reconstruction from a generated video to achieve 4D generation is novel since it has great potential in modeling high-fidelity 4D representations and many natural downstream applications (as also mentioned by 1Eme). We add a natural example in Figure I of the PDF file that with our framework, we can perform 4D/video customization (based on input text prompts and camera poses) by a training-free video continuation and then render Gaussian.

- Special designs for generated videos: Generated videos and real videos do not have strict boundaries, but usually generated videos are observed to have more unexpected large-scale movement (non-rigidity and distortion leading to complex object root poses) and small-scale anomalies (flickering and float-occlusion). To address both challenges, we design our overall reconstruction method as a coarse-to-fine pipeline.
  -  The coarse part contains time-varying warpings ($\tilde{\textbf{R}}^t$ and $\tilde{\textbf{T}}^t$) to model the basic movement and register the camera and root pose. Besides, we adopt motion regularization (Eq.6) and motion field initialization (Sec. 3.3), together to reconstruct large-scale movement with non-rigidity and distortion even under limited viewpoints. Compared with directly applying 3DGS with dense motion, our design largely alleviates overfitting w.r.t. motions (as shown in Figure B). We provide more details in the response to crn6.
  - The fine part is performed in the static stage using a time-invariant rotation $\Delta \textbf{R}^*_k$ and scaling $\Delta \textbf{S}^*_k$ (Eq.8). This process further alleviates overfitting w.r.t. flickering and float-occlusion (as shown in Figure E and Figure C). We provide more details in the response to crn6.

Meta-review: This paper presents Vidu4D, a reconstruction model that  reconstructs 4D (sequential 3D) representations from single generated videos. The paper received divergent ratings (1x Borderline Reject, 2x Borderline Accept, 1x Weak Accept).

Three reviewers are positive about the method's design and the strong results. However, Reviewer 7btN gave a Borderline Reject rating, citing issues with the writing and organization of the manuscript, as well as some evaluation concerns. The authors' response did not receive a reply. Overall, I agree with the authors that issues with writing clarity should not be grounds for rejection and can be addressed in the revision. I believe this is an interesting paper with strong results that makes a valuable contribution to the area of 4D reconstruction. I will recommend an acceptance.

The authors are requested to revise the paper, taking into account the reviewers' comments for the camera-ready version.
Decision: Accept (poster)