Vidu4D: Single Generated Video to High-Fidelity 4D Reconstruction with Dynamic Gaussian Surfels

Authors: yikai  wang
xinzhou  wang
zilong  chen
zhengyi  wang
fuchun  sun
jun  zhu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 06:38:14

Abstract: Video generative models are receiving particular attention given their ability to generate realistic and imaginative frames. Besides, these models are also observed to exhibit strong 3D consistency, significantly enhancing their potential to act as world simulators.  In this work, we present Vidu4D, a novel reconstruction model that excels in accurately reconstructing 4D (i.e., sequential 3D) representations from single generated videos, addressing challenges associated with non-rigidity and frame distortion. This capability is pivotal for creating high-fidelity virtual contents that maintain both spatial and temporal coherence. At the core of Vidu4D is our proposed Dynamic Gaussian Surfels (DGS) technique. DGS optimizes time-varying warping functions to transform Gaussian surfels (surface elements) from a static state to a dynamically warped state. This transformation enables a precise depiction of motion and deformation over time. To preserve the structural integrity of surface-aligned Gaussian surfels, we design the warped-state geometric regularization based on continuous warping fields for estimating normals. Additionally, we learn refinements on rotation and scaling parameters of Gaussian surfels, which greatly alleviates texture flickering during the warping process and enhances the capture of fine-grained appearance details. Vidu4D also contains a novel initialization state that provides a proper start for the warping fields in DGS. Equipping Vidu4D with an existing video generative model, the overall framework demonstrates high-fidelity text-to-4D generation in both appearance and geometry.

Main paper: 
 introduction: 
 The field of multimodal generation exhibits significant advancements and holds great promise for various applications. Recently, video generative models have garnered attention for their remarkable capability to craft immersive and lifelike frames [4, 8] . These models produce visually stunning content while also exhibiting strong 3D consistency [15, 81] , largely increasing their potential to simulate realistic environments.Parallel to these developments, high-quality 4D reconstruction has made great strides [19, 58, 63, 94, 100] . This technique involves capturing and rendering detailed spatial and temporal information. When integrated with generative video technologies, 4D reconstruction potentially enables the creation of models that capture static scenes and dynamic sequences over time. This synthesis provides a more holistic representation of reality, which is crucial for applications such as virtual reality, scientific visualization, and embodied artificial intelligence. (e) Prompt: A small, fluffy creature with an appearance reminiscent of a mythical being. The creature's fur texture is rendered in high detail. The monster's large eyes and open mouth express wonder and curiosity.(f) Prompt: An isolated coloured abstract sculpture with a dali shape.Figure 1 : Text-(to-video)-to-4D samples generated by equipping Vidu4D with a pretrained video diffusion model [4] . For each sample, we exhibit per-frame 3D rendering for novel-view color, normal, and surfel feature. We observe that Vidu4D can reconstruct precisely detailed and photo-realistic 4D representation. See our accompanying videos in our project page for better visual quality. and 4DGS [94] have been developed, utilizing neural deformation fields with multi-layer perception (MLP) and triplane, respectively. SCGS [29] and dynamic 3D Gaussians [52] also advance the field by modeling time-varying scenes. Building on these advances, our work introduces dynamic Gaussian surfels, a novel extension of Gaussian representations that enhances the quality of both appearance and surface reconstruction under dynamic scenarios. A concurrent work DGM [43] builds time-consistent meshes from a monocular video with 3D Gaussian Splatting. In the realm of 3D or 4D generation, our approach diverges from recent progress in optimization-based [2, 13, 14, 40, 42, 62, 71, 88, 90] , feed-forward [26, 91, 106] , and multi-view reconstruction methods [15, 50, 51] by leveraging a video generative model to achieve generation capabilities. Our primary focus is on preserving high-quality appearance and geometrical integrity from generated videos. This results in a generation process that not only captures the nuances of motion and deformation but also maintains the high standards of realism and detail that are essential for creating immersive and lifelike virtual 3D representations. 
method: 
 We start by introducing the problem definition for 4D reconstruction in Sec. 3.1. Following that, we introduce our Vidu4D which encompasses two novel stages. The first stage is designed to learn Dynamic Gaussian Surfels (DGS), ensuring precise representation of both visual appearance and geometric structure during the non-rigid reconstruction process, as detailed in Sec. 3.2. The second stage focuses on establishing the initial non-rigid warping fields of DGS, as detailed in Sec. 3.3.When given a single sequence of RGB video with T frames, the goal of 4D reconstruction is to determine a sequential 3D representation that could be rendered to fit each video frame as much as possible. Specifically, suppose the 3D representation for the t-th frame (termed as time t) is parameterized by θ t , where t = 1, • • • , T . Given a differentiable rendering mapping g, we could obtain the rendered color at the frame pixel xt ∈ R 2 . We choose volume rendering as commonly adopted in NeRF [55] , Gaussian Splatting [33] , and Gaussian Surfels [16, 28] . The optimization of 4D reconstruction can be implemented by minimizing the empirical loss asEQUATIONwhere x t i ∈ R 3 is the i-th 3D point sampled or intersected with Gaussian primitives along the ray that emanates from the frame pixel xt ; N is the number of sampled or intersected points per ray; c(x t ) and ĉ(x t ) are the rendered color and the observed color at xt , respectively.In the following, we detail the proposed Vidu4D, a reconstruction pipeline comprising two key stages as illustrated in Fig. 2 , including a field initialization stage and a DGS stage.By optimizing Eq. ( 1), essentially our goal is to build a sequential 3D representation that could deform to be consistent with each 2D frame. We first start by considering an ideal video exhibiting different views of the same static object without object deformation, movement, or video distortion. To model the 3D representation with high appearance fidelity and geometry accuracy, we follow the method of using differentiable 2D Gaussian primitives as proposed by recent Gaussian Surfels advances [16, 28] . L B Q h L L u U g W C B 1 B B j g L q k Q T q u w J q b u 9 q V K 8 9 g F Q 8 D M r Y j 8 D x a S f g H m c U N W p l C 0 2 E R 3 S 9 5 G Z w j x e N J n L R h u Q H 3 g 0 0 P Z q B 5 R F 0 W t m c l b f G M u e N P T U 5 M l W p l R 0 2 2 y G L f Q i Q C a p U w 7 Y i d B I q k T M B g 0 w z V h B R 1 q M d a G g b U B + U k 4 w v H J g H m r R N L 5 T 6 B W i O 6 e + J h P p K 9 X 1 X d / o U u 2 q 2 N o L / 1 R o x e g U n 4 U E U I w R s s s i L h Y m h O Y r L b H M J D E V f G 8 o k 1 3 8 1 W Z d K y l C H m t EO u V g X f 6 8 U K y v l b k h k V 0 g T X p l F k U = " > A A A B / 3 i c b V D J S g N B E O 1 x j X E b F b x 4 G Q y C e A g z 4 n Y M 6 s F j F L N A E k N P p y Z p 0 r P Q X S O G c Q 7 + i h c P i n j 1 N 7 z 5 N 3 a S O W j i g 4 L H e 1 V U 1 X M j w R X a 9 r c x M z s 3 v 7 C Y W 8 o v r 6 y u r Z s b m 1 U V x p J B h Y U i l H W X K h A 8 g A p y F F C P J F D f F V B z + x d D v 3 Y P U v E w u M V B B C 2 f d g P u c U Z R S 2 1 z u 3 k J A m k T 4 Q F d L 7 l J 2 / 2 7 5 C B t m w W 7 a I 9 g T R M n I w W S o d w 2 v 5 q d k M U + B M g E V a r h 2 B G 2 E i q R M w F p v h k r i C j r 0 y 4 0 N A 2 o D 6 q V j O 5 P r T 2 t d C w v l L o C t E b q 7 4 m E + k o N f F d 3 + h R 7 a t I b i v 9 5 j R i 9 s 1 b C g y h G C N h 4 k R c L C 0 N r G I b V 4 R I Y i o E m l E m u b 7 V Y j 0 r K U E e WV G 9 + r k g v W + g 0 = " > A A A C I H i c b V D J S g N B F O y J W 4 x b 1 K O X w S C I Q p g R N R 6 D X j x G y Q b Z 6 O m 8 S Z r 0 L H S / E c M w n + L F X / H i Q R G 9 6 d f Y W Q R N L G i o r q p H 9 y s n F F y h Z X 0 a q Y X F p e W V 9 G p m b X 1 j c y u 7 v V N V Q S Q Z V F g g A l l 3 q A L B f a g g R w H 1 U A L 1 H A E 1 Z 3 AQ p g R N R 6 D X j x G y Q b Z 6 O m 8 S Z r 0 L H S / E c M w n + L F X / H i Q R G 9 6 d f Y W Q R N L G i o r q p H 9 y s n F F y h Z X 0 a q Y X F p e W V 9 G p m b X 1 j c y u 7 v V N V Q S Q Z V F g g A l l 3 q A L B f a g g R w H 1 U A L 1 H A E 1 ZG V i k X i H t K Y u i E e C B Y w g k F L n l l R 3 v j h p A f 0 E f w g g 0 n + 8 s y q X b O n s B a J U 5 A q K t D w z K 9 e P y J J S A U Q j p X q O n Y M b o Y l M M L p p N x L F I 0 x G e E B 7 W o q c E i V m 0 2 X n 1 h H W u l b Q S R 1 C b C m 6 u + J D I d K p a G v O 0 M M Q z X vS Z H R u k R P 1 K m Q i R T 9 f d E S g O t R 4 F n O g O K A z 3 v T c T / v F a C / p W b i j B O k I d s t s h P J M G I T C I g P a E 4 Q z k y h D I l z K 2 E D a i i D E 1 Q B R O C M / / y I q m f l p 2 L 8 v n d W a l y n c W R h w M 4 h G N w 4 B I q c A t V q A E D B c / w C m / W o / V i v V s f s 9 a c l c 3 s w x 9 Y n z / F 0 5 K z < / l a t e x i t > p ⇤ k < l a t e x i t s h a 1 _ b a s e 6 4 = " z o V H X k T S l E R h k 7 b L p n i h H X 9 6 f j Q = " > A A A B / H i c b V D L S s N A F J 3 U V 6 2 v a J d u B o s g L k o i v p Z F N y 4 r 2 A e 0 M U y m k 3 b o 5 M H M j R h C / R U 3 L h R x 6 4 e 4 8 2 + c t F l o 6 4 E L Z 8 6 5 l 7 n 3 e L H g C i z r 2 y g t L a + s r p X X K x u b W 9 s 7 5 u 5 e W 0 W J p K x F I x H J r k c U E z x k L e A g W D e W j A S e Y B 1 v f J 3 7 n Q c m F Y / C O 0 h j 5 g R k G H K f U w J a c s 2 q c p P 7 4 z 6 w R / D 8 D C b 5 y z V r V t 2 a A i 8 S u y A 1 V K D p m l / 9 Q U S T g I V A B V G q Z 1 s x O B m R w K l g k 0 o / U S w m d E y G r K d p S A K m n G y 6 / A Q f a m W A / U j q C g F P 1 d 8 T G Q m U S g N P d w Y E R m r e y 8 X / v F 4 C / q W T 8 T B O g I V 0 9 p G f C A w R z p P A A y 4 Z B Z F q Q q j k e l d M R 0 Q S C j q v i g 7 B n j 9 5 k b R P 6 v Z 5 / e z 2 t N a 4 K u I o o 3 1 0 g I 6 Q j S 5 Q A 9 2 g J m o h i l L 0 j F 7 R m / F k v B j v x s e s t W Q U M 1 X 0 B 8 b n D + f p l P M = < / l a t e x i t > s ⇤ u t ⇤ u Text promptJoint training with shared centers and warping < l a t e x i t s h a 1 _ b a s e 6 4 = " 8 V g C 0 P u a T 1 F P N E Q Y e z U V F r u V e S w = " > A A A C E X i c b V D L S s N A F J 3 U V 6 2 v q k s 3 w S I U F y U R X 8 u i G 5 d V 7 A O a t k w m N + 3 Q y Y O Z G 7 G E / I I b f 8 W N C 0 X c u n P n 3 5 i 0 F b T 1 w M C Z c 8 5 l 5 h 4 7 F F y h Y X x p u Y X F p e W V / G p h b X 1 j c 6 u 4 v d N Q Q S Q Z 1 F k g A t m y q Q L B f a g j R w G t U A L 1 b A F N e 3 i Z + c 0 7 k I o H / i 2 O Q u h 4 t O 9 z l z O K q d Q r l l U v 6 h 5 a y I U D s Y V w j 7 Y b 3 y R J F 3 8 u m G S J Q q 9 Y M i r G G P o 8 M a e k R K a o 9 Y q f l h O w y A M f m a B K t U 0 j x E 5 M J X I m I C l Y k Y K Q s i H t Q z u l P v V A d e L x R o l + k C q O 7 g Y y P T 7 q Y / X 3 R E w 9 p U a e n S Y 9 i g M 1 6 2 X i f 1 4 7 Q v e 8 E 3 M / j B B 8 N n n I j Y S O g Z 7 V o z t c A k M x S g l l k q d / 1 d m A S s o w L T E r w Z x d e Z 4 0 j i r m a e X k + r h U v Z j W k S d 7 Z J + U i U n O S J V c k R q p E 0 Y e y B N 5 I a / a o / a s v W n v k 2 h O m 8 7 s k j / Q P r 4 B w Q C e O g = = < / l a t e x i t > s ⇤ u Rt t ⇤ u Video diffusion model Field init. DGS Method details of DGS < l a t e x i t s h a _ b a s e = " c + J b x + X c d m A s R n Q D U w e N n I v k j E = " > A A A B + n i c d V D L S g N B E J z G e M r a O X w S C o h U T W B b j G A S I Y l h d j K r Q Y f z P S q Y c n e P G g i F e / x J t / + Q h q G h B Q H V T X e X F u h g Z A P a Z b n h M b O U X V Z X V v P T c a O k o U U W y U h d e F R z K U J e B w G S X S K C T v O n T Z + Y r L a L w H A Y x w T K h S + Y B S M M l a + f u G / g d e H a D P e u Q K x K V K q V T E x C r l s s G J c U n E J d m w y R g F N U e v m t u i C U B D F J q n X L I T F U q p A M M m H X a i e U x Z n x l q E h D b j u p O P T h j H K D s R p U C H i s f p I a a D I P B M Z D h W v / R u J f X i s B v x J R R g n w E M W e Q n E k O E R z n g n l C c g R w Y Q p k S l b M r q m i D E x a W R P C f f I o o r H d F K r H z g y a A t t o k o C N U R a e o h u q I o V v g J Q s V v P V o v u u k d cA A B + n i c d V D L S g N B E J z 1 b X y t e v Q y G I R 4 W W Y T X c 1 N 9 O I x g j G B Z A 2 z k 9 k 4 Z P b B T K 8 a 1 n y K F w + K e P V L v P k 3 T m I E F S 1 o K K q 6 6 e 4 K U i k 0 E P J u T U 3 P z M 7 N L y w W l p Z X V t f s 9 Y 0 L n W S K 8 T p L Z K K a A d V c i p j X Q Y D k z V R x G g W S N 4 L + y c h v X H O l R R K f w y D l f k R 7 s Q g F o 2 C k j r 1 e 6 / Q v o d Q G f g t B m G f D 3 Y 5 d J E 6 1 X K 1 U y p g 4 F d f z y o e G E I 9 U P Y J d h 4 x R R B P U O v Z b u 5 u w L O I x M E m 1 b r k k B T + n C g S T f F h o Z 5 q n l P V p j 7 c M j W n E t Z + P T x / i H a N 0 c Z g o U z H g s f p 9 I q e R 1 o M o M J 0 R h S v 9 2 x u J f 3 m t D M J D P x d x m g G P 2 e e i M J M Y E j z K A X e F 4 g z k w B D K l D C 3 Y n Z F F W V g 0 i q Y E L 4 + x f + T i 7 L j e s 7 + 2 V 7 x 6 H g S x w L a Q t u o h F x 0 g I 7 Q K a q h O m L o B t 2 j R / R k 3 V k P 1 r P 1 8 t k 6 Z U 1 m N t E P W K 8 f k p 2 U P Q = = < / l a t e x i t > P t k (u)< l a t e x i t s h a 1 _ b a s e 6 4 = " w i G t W I e p E X P c E T i p W v v D N V K q R J 0 = " > A A A C A 3 i c d V B N SG C + M T 4 x E = " > A A A C N H i c b V D J S g N B E O 1 x j X G L e v T S G A Q X C D P i d g z q Q f C i Y l T I J E N P p y Y 2 6 V n o r h H D M B / l x Q / x I o I H R b z 6 D c 4 k E d c H D a 9 e 1 a O 6 n h t J o d E 0 H 4 2 h 4 Z H R s f H C R H F y a n p m t j Q 3 f 6 7 D W H G o 8 V C G 6 t J l G q Q I o I Y C J V x G C p j v S(s ⇤ v + s ⇤ v ) Rt R ⇤ k t ⇤ v < l a t e x i t s h a 1 _ b a s e 6 4 = " N + P 4 Q A 6 O h h + z F J 7 6 y e v V t S N t Y M E = " > A A A C N H i c b V D J S g N B E O 1 x j X G L e v T S G A Q X C D P i d h T 1 I H i J Y l T I J E N P p y Y 2 6 V n o r h H D M B / l x Q / x I o I H R b z 6 D c 4 k E d c H D a 9 e 1 a O 6 n h t J o d E 0 H 4 2 h 4 Z H R s f H C R H F y a n p m t j Q 3 f 6 7 D W H G o 8 V C G 6 t J l G q Q I o I Y C J V x G C p j v S(s ⇤ u + s ⇤ u ) Rt R ⇤ k t ⇤ u* u ∈ R 3×1 , t * v ∈ R 3×1 and scaling factors s * u ∈ R, s * v ∈ R.Here, we use the notation " * " to represent parameters in the static state. A Gaussian surfel is computed as a 2D Gaussian defined in a local tangent plane in the world space. Following [28] , for any point u = (u, v) located on the uv coordinate system centered at p * k , its coordinate in the world space, denoted as 3) denotes the rotation matrix, and the diagonal matrixEQUATION* k = [t * u , t * v , t * u × t * v ] ∈ SO(S * k = diag(s * u , s * v , 0) ∈ R 3×3denotes the scaling matrix. In this work, our focus is on 4D reconstruction from a single generated video, which may exhibit large non-rigidity, distortion, or illumination changes. We introduce Dynamic Gaussian Surfels (DGS), a method designed to achieve precise 4D reconstruction while accommodating non-rigidity and other time-varying effects.Motivated by recent advancements in non-rigid reconstruction methods [57, 88, 98] , we aim to ensure that the target object maintains a consistent static state across different frames, thereby mitigating non-rigidity and distortion effects. To achieve this, we employ warping techniques on each Gaussian surfel represented by P * k (u), transforming them into a corresponding Gaussian surfel P t k (u) at time t, which is centered at p t k ∈ R 3 with a rotation matrix R t k ∈ SO(3) and a scaling matrix S t k ∈ R 3×3 . Non-rigid warping for Gaussian surfels. We now build the warping process from the static state to the warped state. We leverage a non-rigid warping function with B bones as key points to ease the training of deformation. In the static state, the b-th bone is represented by 3D Gaussian ellipsoids [97] , with more details provided in the Appendix. We let J t b ∈ SE(3) represent a rigid transformation that moves the b-th bone from its static state to the warped state at time t. In effect, J t b is achieved by non-linear mappings using a multi-layer perception (MLP) with SE(3) guaranteed, as will be given later in Eq. ( 5). The non-rigid warping function can be written as the weighted combination of J t b , where we apply dual quaternion blend skinning (DQB) [32] to ensure valid SE(3) after combination,EQUATIONwhere w t b is the b-th element of the skinning weight vector w t ∈ R B×1 , as detailed in the Appendix; Q and R denote the quaternion process and the inverse quaternion process, respectively. In this case, there is J t ∈ SE(3).We therefore rewrite the warping as J t = [ Rt , Tt ] with the rotation Rt ∈ SO(3) and translation Tt ∈ R 3 , and apply the corresponding transformation to Eq. ( 2) byEQUATIONNote that Eq. ( 4) holds for any given point P * k (u) including the center point of the k-th Gaussian surfel (i.e., p * k ) when u = (0, 0). By deriving Eq. ( 4), we enable connection of the warping function w.r.t. to any point u = (u, v) on the local coordinate system centered at p * k , which is needed later in Eq. ( 8) where u is an intersection with Gaussian surfels and a ray that emanates from the frame pixel.Warped-state normal regularization. To accurately capture the geometric representation, we follow similar methods in Gaussian Surfels [16, 28] to add normal consistency regularization which encourages all Gaussian surfels to be locally aligned with the actual surfaces. Differently, unlike 3D reconstruction for static scenes, 4D reconstruction commonly faces non-rigidity and distortion. Thus simply performing regularization to promote surface-aligned Gaussian surfels like previous methods harms the structural integrity due to the non-rigid warping.We therefore design a warped-state normal regularization. As mentioned, each point P t k (u) in the warped state at time t is transformed from its corresponding static point P * k (u) based on the warping function in Eq. ( 4), namely, P t k (u) = J t P * k (u) with J t composed by J t b . To maintain the structural integrity to a large extent when regularizing normal, we design J t b as a continuous field that takes both the point P * k (u) (or equivalently, u in the local coordinate system) and the time t as conditions. By this setting, J t b is expected to change continuously with the change of u or t. We implement the continuous field by using a NeRF-style MLP which directly outputs a 6-dimensional dual quaternion, and rely on the inverse quaternion process R to guarantee SE(3), i.e.,EQUATIONwhere γ t b is a learnable latent code for encoding the b-th bone at time t; both u and t are sent to the MLP as conditions to obtain J t b . Thus J t is also expected to be continuous w.r.t. u and t. Based on the above design, the normal consistency loss at time t is obtained similar to [28] ,EQUATIONwhere k indexes over intersected surfels along the ray that emanates from the frame pixel x;ω k = α k G k (u(x)) k-1 j=1 (1 -α j G j (u(x))) denotes the blending weight of the intersection point; n k represents the normal of the surfel that is oriented towards the camera; N t , computed with finite differences, is the surface normal estimated by the nearby depth point p t at warped state time t.In summary, by learning a continuous warping field and aligning the surfel normal with the estimated surface normal in the warped state, we ensure that all Gaussian surfels locally approximate the actual object surface without being noticeably impaired by the non-rigid warping.Dual branch structure with refinement. To further achieve fine-grained appearance and reduce the texture flickering during warping, we propose to learn refinement terms for adjusting the rotation matrices R * k and scaling matrices S * k (defined in Eq. ( 2)) in the static state. We suppose the refinement terms are ∆R * k ∈ SO(3) and ∆S * k ∈ R 3×3 , respectively. Note that the third-axis of ∆S * k is no longer necessarily 0. During refinement, we remain the center points p * k and the warping J t (i.e., including both Rt and Tt ) to be unchanged. The new warped process is formulated as,EQUATIONDuring the training of DGS, we maintain two branches including one with refinement and one without.In the warped state, both branches are jointly trained with shared warping functions and centers of Gaussian primitives2 . Due to the involvement of ∆R * k and ∆S * k , both branches have different rotation and scaling matrices of Gaussian primitives.Rasterization. Given a frame pixel x and a camera ray that emanates from x, following the staticstate methods to calculate intersection coordinates with Gaussian primitives along the ray [28, 33] , we could obtain warped-state intersection coordinates based on Eq. ( 4) and Eq. (7) . We then perform the volume rendering process [28] that integrates alpha-weighted appearance along the ray byEQUATIONwhere k indexes over intersected Gaussian primitives along the ray that emanates from the frame pixel x; α k and c k denote the opacity and view-dependent appearance parameterized with spherical harmonics of the k-th Gaussian surfel, respectively; G k (u(x)) = expu 2 +v 2 2 corresponds to the k-th intersection point u(x) which could be directly calculated when given P t k (u) or P ′t k (u) and the corresponding local coordinate system. During implementation, G k (u(x))) is further applied a low-pass filter following [7, 28] .A detailed architecture of DGS is depicted in Fig. 3 . Important symbols are summarized in our Appendix.Given that the camera trajectory of generated videos is unknown, SfM methods like COLMAP struggle to converge due to rigidity violations. Additionally, since the background of generated videos appears to exhibit soft deformation or flickering colors, proper estimation of camera/body poses through background SfM is hindered. These challenges often result in very few successful registrations, as demonstrated in previous monocular 4D reconstruction tasks [98] .To address this, we design an implicit field before performing DGS to initialize the camera poses and establish the continuous warping field in Eq. ( 5). In this part, we propose the field initialization as another key component of our pipeline to initialize the continuous warping field of DGS for fast and stable convergence, as detailed below.Initially, we train a neural Signed Distance Function (SDF) model [87] , leveraging the same warping structure with bones as utilized in DGS. While DGS transforms Gaussian surfels from the static state to the warped state for rasterization, the neural SDF reverses this process, mapping points along camera rays from the warped state back to the static state. For the neural SDF component, we optimize the reverse warping process and deduce the forward warping as its inverse by minimizing a cycle loss, inspired by [10, 98] . Subsequently, we initialize MLP(•) in Eq. ( 5) to obtain warping functions J f b by the network weights learned by the neural SDF part. During the rendering of the neural SDF, we perform backward warping on the warped-state sampling points to the static state,EQUATIONwhich is an inversion of Eq. (3) . By querying the SDF with a sample point X * in the static state, we render RGB and compute the photometric loss to optimize the SDF and the warping field defined in Eq. ( 5).Nevertheless, there are two discrepancies between the neural SDF warping and DGS warping. Firstly, sampling points of the neural SDF are distributed in the frustum of the camera, while sampling points of DGS are distributed on the object surface. Additionally, we train the inversion of during initialization, while we utilize the non-inverse ones in DGS. To resolve the distribution gap and ensure that faithfully models the forward warping, we add a cycle loss,EQUATION)where X f could be either a mesh surface point or a sample point on the camera ray.After initialization, we extract the canonical space mesh using marching cubes and initialize Gaussian surfels on it. We set the spherical harmonic in 0-th order to the RGB value of the nearest vertices.The warping field and learned camera poses are retained.With the field initialization before DGS, our Vidu4D is capable of performing a text-(to-video)-to-4D generation task with the integration of existing video diffusion models.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes]Answer: [Yes] Justification: We will release it after acceptance.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiment: 
 In this section, we provide an extensive evaluation of our method DGS (Sec. 3.2) with the initialization in Sec. 3.3, comparing both appearance and geometry against previous state-of-the-art methods. Additionally, we analyze the contributions of each proposed component in detail. [98] and D-NeRF [63] ) and Gaussian splatting-based methods (Deformable-GS [100] and SCGS [29] ). We also provide our learned camera poses to baseline approaches for a fair comparison. These variants are denoted as "w. Poses". Best view in color and zoom in.For all qualitative and quantitative experiments, we follow the standard pipeline for dynamic reconstruction [58] , to construct our evaluation setup by selecting every fourth frame as a training frame and designating the middle frame between each pair of training frames as a validation frame.Our model configuration involves several key parameters to balance reconstruction and regularization losses. For the field initialization stage, we use a similar architecture with 8 layers for volume rendering as in NeRF [55] , and initialize MLP for predicting SDF as an approximate unit sphere [101] . We obtain a neural SDF, a warping field, and camera poses after this stage. For the DGS stage, we initialize centers of the Gaussian surfels with the sampled surface points extracted from the neural SDF, and initialize the warping field by the forward field from the first stage. The dimension of the latent code embedding γ t b is set as 128. Following BANMo [98] , we adopt 25 bones to optimize skinning weights. For each reconstruction, the overall training takes over 1 hour on an A800 GPU.In the qualitative evaluation, we visually compare the novel-view reconstructions produced by our DGS against those generated by other state-of-the-art models, as illustrated in Fig. 4 . Our evaluation focuses on several key aspects including detail preservation, texture quality, and geometric accuracy. Compared to methods based on implicit fields, the integration of Gaussian in our approach facilitates the rendering of highly detailed textures. Additionally, benefiting from a more geometry-aware representation, our method produces superior normal maps compared to those purely Gaussian-based methods. This also enhances the robustness of our method against artifacts of the generated videos like occlusions. For instance, in the third clip of the series, which features a dragon shrouded in fog, both SCGS and Deformable-GS methods tend to overfit and subsequently show a decline in performance. In contrast, our method consistently delivers superior results.We provide the quantitative evaluation comparing our method with state-of-the-art works in Table 1 .Metrics include Peak Signal-to-Noise Ratio (PSNR) to evaluate the fidelity of the reconstructed textures, Structural Similarity Index (SSIM) for the quality evaluation, and LPIPS [103] as a perceptual metric. Our method exhibits superiority over all baseline methods, even with our learned poses, e.g., ∼2.5 PSNR increase over SCGS with poses for the averaged results.To understand the contributions of each component in Vidu4D, especially DGS, we conduct ablation studies in this section. We remove or alter specific elements of our model and observe the resulting performance changes in both appearance and geometry reconstruction.Geometric regularization. We evaluate the impact of warped-state normal regularization by disabling it during training. From Fig. 5 (b)(d), we observe that when removing the regularization, there is an obvious degradation in the structural integrity of surface-aligned Gaussian surfels, leading to noticeable inconsistency in the reconstructed 4D models.Refinement strategy. We examine the effect of omitting refinements by keeping one branch (the concept of branches could be better visualized in Fig. 3 ) during training, shown in Fig. 5 (b)(c). The performance indicates that removing refinements increases the loss of fine-grained appearance details. Additionally, we also find that refinements are crucial for mitigating the texture flickering issue.Additional ablations. Please refer to the Appendix for additional ablation studies that detail the effectiveness of our refinement strategy and field initialization. 
conclusion: 
 We introduce Vidu4D as a novel reconstruction model to achieve high-fidelity 4D representations from single generated videos. Vidu4D is powerful with our proposed DGS which builds the non-rigid warping field to transform Gaussian surfels, ensuring precise capture of motion and deformation over time. DGS also introduces key innovations that greatly enhance the accuracy and fidelity of 4D reconstruction, including dual branch refinement and warped-state geometric regularization.Our experiments demonstrate that Vidu4D outperforms existing methods in both quantitative and qualitative evaluations, highlighting its superiority in generating realistic and immersive 4D content.Limitations and broader impact. While Vidu4D with DGS presents a significant performance in 4D reconstruction, currently there are still limitations such as the reliance on video quality, scalability challenges for large scenes, and computational difficulties in real-time applications. Additionally, when equipping Vidu4D with generative models, as with any generative technology, there is a risk of producing deceptive content which needs more caution.  
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: All relevant details are provided within the text of our paper or through the references we have cited.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: Please refer to our Tab. 1.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We discuss it in Sec. 5.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: No such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite the assets properly.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: We present results for interpolation on time and views, as illustrated in Fig. 11 and Fig. 12 .Generative models used in video generation might pose risks, for example, the potential for creating deepfakes or other misleading content that could be used for harmful purposes like misinformation, privacy invasion, or defamation. To mitigate these risks, we have chosen to release only the reconstruction code, deliberately avoiding the release of components that could facilitate the generation of content with ethical concerns. This decision ensures that our contribution is focused on advancing reconstruction techniques without enabling the creation of new, potentially harmful video content.Besides, we have carefully considered the potential ethical risks associated with generative models, particularly in video content creation. To address these concerns, our model includes robust safety mechanisms designed to screen and prevent any misuse. We believe these measures effectively mitigate potential ethical risks and align with the standards of the community. NeurIPS Paper Checklist Justification: We discuss it Sec. 5. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Here, since the third-axis of the refined scaling matrix is not necessarily 0, we adopt "Gaussian primitive" for commonly referring to both Gaussian surfel and the refined Gaussian. 


Decision: This paper presents Vidu4D, a reconstruction model that  reconstructs 4D (sequential 3D) representations from single generated videos. The paper received divergent ratings (1x Borderline Reject, 2x Borderline Accept, 1x Weak Accept).

Three reviewers are positive about the method's design and the strong results. However, Reviewer 7btN gave a Borderline Reject rating, citing issues with the writing and organization of the manuscript, as well as some evaluation concerns. The authors' response did not receive a reply. Overall, I agree with the authors that issues with writing clarity should not be grounds for rejection and can be addressed in the revision. I believe this is an interesting paper with strong results that makes a valuable contribution to the area of 4D reconstruction. I will recommend an acceptance.

The authors are requested to revise the paper, taking into account the reviewers' comments for the camera-ready version.

Reviews: review: paper_summary: paper_summary: The paper proposes a technique called Dynamic Gaussian Surfels to effectively reconstruct 4D reqpresentation from a single generated video. DGS optimizes time-varying warping functions to transform Gaussian surfels and the authors adopt Neural SDF for initialization and proposes a geometry regularization technique to preserve the geometry integrity. Extensive experiments on 30 objects proves the effectiveness of the proposed method.
strength_weakness: strength_weakness: a. The novelty is limited. This work seems to be a combination of LARS (bones and warping), Gaussian Surfels/2DGS (3D representation) and SC-GS/4DGS (refinement). The real novelty might be the geometric regularization and field initialization, although the former is also similar to that used in previous 3D works.

  b. Lack of evaluation details: The authors evaluated the comparison methods on generated video for novel view synthesis in Table 1, however, there is no gt for generated video's novel view resutls. How did the author conduct the evaluation?

  c. The experiments is not extensive enough: The authors claim that their methods are designed for generated video, however, I didn't see any special designs, e.g., sovle the potential multi-view inonconsistency in the input video. So, I think it is a general 4D reconstruction method, and it is recommanded to test the proposed methods on commanly used 4D reconstruction datasets (object and scene-level), using standard evaluation metrics.
questions: questions: See weaknesses. I might adjust the score according to the response from the authors.
limitations: limitations: Yes
paper_summary: paper_summary: Video generation models have shown great power recently. Transforming generated videos into 3D/4D representations is important for building a world simulator. This paper proposes an improved 4D reconstruction method from single-generated videos. The key component is the dynamic Gaussian surfels (DGS) technique. Incorporating an initialization stage of a non-rigid warping field, the Vidu4D method produces impressive 4D results with the video generation model Vidu.
strength_weakness: strength_weakness: 1. The motivation/necessity of building surfels needs to be further strengthened. It is easy to understand building surfels will undoubtedly help reconstruct the surface and geometry. If just considering rendering videos from the built 4D, will a vanilla 4D representation (without improvement on surface reconstruction) be enough? Fig. 4 and Table 1 provide convincing results. However, it is suggested to make it clearer in introducing the motivation, e.g. why better geometry leads to better synthesis.
2. The organization of the method could be improved. For better understanding, it is suggested to first introduce the overall framework of Vidu4D and then demonstrate the dynamic Gaussian Surfels technique.
3. The main method is more like a basic representation/reconstruction approach. Will it still benefit reconstruction from monocular videos captured in real life, not generated videos? Yet, real-life videos have better consistency than generated ones.
questions: questions: 1. The provided results are all object-level 4D ones. Will this method work well on scene-level samples? I know it will be hard to generate the surfels of the background. 
2. What about the mesh and depth of the generated 4D representations?
3. One missing related work: Liu I, Su H, Wang X. Dynamic Gaussians Mesh: Consistent Mesh Reconstruction from Monocular Videos[J]. arXiv preprint arXiv:2404.12379, 2024.
limitations: limitations: The paper has clearly addressed the limitations.
paper_summary: paper_summary: The paper presents Vidu4D, a  reconstruction model that excels in accurately reconstructing 4D (i.e., sequential 3D) representations from single generated videos, addressing challenges associated with non-rigidity and frame distortion. At the core of Vidu4D is a proposed Dynamic Gaussian Surfels (DGS) technique. DGS optimizes time-varying warping functions to transform Gaussian surfels (surface elements) from a static state to a dynamically warped state. This transformation enables a precise depiction of motion and deformation over time. To preserve the structural integrity of surfacealigned Gaussian surfels, the authors design the warped-state geometric regularization based on continuous warping fields for estimating normals. Additionally, the method learns refinements on rotation and scaling parameters of Gaussian surfels, which alleviates texture flickering during the warping process and enhances the capture of fine-grained appearance details.
strength_weakness: strength_weakness: 1. The paper's annotation is cluttered and extremely hard to follow. Sometimes ignoring some symbols in formulation is desirable when too many of them are presented.

2. The real framework section 3.3 is extremely short. The "more details in our appendix" seems to be a false promise? 

3. The model hasn't evaluated on realistic scenes, so that it can be compared with other 4dgs methods using their official results.

4. The paper uses banmo like learnable joint representation to drive deformation, however hasn't mentioned the limitations of this kind of methods, i.e., the 4d scene needs to be object centric.

5. No evaluations of latency, it seems the pipeline needs a dynamic neus/nerf, then init 2dgs at the extracted zero level set, which makes the methods very dependable of the stage one and very time consuming.
questions: questions: Besides above problems, I think the below questions are needed to be addressed:

What kind of generated video is used as reference video (better to show the reference mono video). Since for 4d reconstruction, if the reference video doesn't show some parts that are occluded across all frames, it is impossible to reconstruct them. An alternative is to use diffusion prior for novel view supervisions to hallucinate these parts, so what is actually happening here????
limitations: limitations: I'm confident it will be horrendously challenging for average readers to fully grip the hole picture without a major revision.
The paper is written to focus on 4d generation task or lift monocular generated video to 4d, however, 80% of the sections are dedicated to representation. If the author want to present this paper as a sknning-based dynamic 2dgs representation paper, then showing 4d generation only is not enough, 4d reconstruction (w/ many widely used benchmarks) should be used as well.

Beside, I hope the author can re-organize the paper no matter it is accepted or not. It is better to put some of the cluttered annotations into appendix, and leave some room for actual pipeline of Vidu4D. The Banmo like joint representation and skinning of 2DGS could be fairly straight forward for people in the field to understand, so no need to put all details in the main paper.
paper_summary: paper_summary: The paper presents Vidu4D, a  reconstruction model that can accurately reconstruct 4D (sequential 3D) representations from single generated videos. This method addressing key challenges and enabling high-fidelity virtual content creation. The proposed techniques, such as Dynamic Gaussian Surfels (DGS) and the initialization state, are good contributions that can benefit the field of multi-modal generation and 4D reconstruction.https://openreview.net/
strength_weakness: strength_weakness: - What is your video foundation model? Is it Stable Video Diffusion, SORA, Open SORA, or Your Vidu? 

If you are using an unreleased foundation video model, is the improvement in qualitative results more due to DGS, or is it caused by the video foundation model? If you utilize Vidu, I believe the author needs to provide the results using open source video foundation models like SVD or Open-SORA.

- The quantitative evaluation in the paper is limited to a small set of generated videos. 

- The performance of Vidu4D on a more diverse and larger dataset of generated videos is not reported, which could limit the generalizability of the findings.
questions: questions: The paper does not discuss the computational complexity or runtime performance of Vidu4D, which could be an important consideration for practical applications of the method.
limitations: limitations: The paper does not mention how the Vidu4D  can be extended or adapted to handle other common challenges in 4D reconstruction, such as occlusions, lighting changes, or complex scene dynamics.

Comments: title: Response for Ethics Review
Official Review of Submission3117 by Reviewer yVYk
Further response for Reviewer crn6's feedback
Response for Ethics Review
Feedback
comment: We appreciate the reviewer's emphasis on the importance of discussing potential ethical implications. In our work, we have carefully considered the potential risks associated with generative models, particularly in video content creation. Generative models used in video generation pose significant risks, such as the potential for creating deepfakes or other misleading content that could be used for harmful purposes like misinformation, privacy invasion, or defamation.

To mitigate these risks, we have chosen to release only the reconstruction code, deliberately avoiding the release of components that could facilitate the generation of content with ethical concerns. This decision ensures that our contribution is focused on advancing reconstruction techniques without enabling the creation of new, potentially harmful video content.

We will also add a discussion on broader impacts in the appendix to thoroughly address ethical considerations, such as the risks associated with video generation, and hope this addition will align our work with community standards.
Thanks for the efforts of the authors for solving some my concerns. I will keep my initial positive rating.
Thank you again for your time and effort in reviewing our work and providing the constructive comments. Please feel free to let us know if you have any further questions by August 13 AoE, we are more than happy to address them.
Thank you again for your time and effort in reviewing our work and providing the constructive comments. Please feel free to let us know if you have any further questions by August 13 AoE, we are more than happy to address them.
Thanks for the authors' rebuttal! My concerns have been addressed and I will keep my initial positive rating. The supplemented contents are highly recommended to be added to the final version.
Thanks for the author's response. This work might be a pioneering work of pose-free dynamic scene reconstruction. In this case, I would suggest to design a standard and rigor evaluation protocol using real-world multi-view video captured with camera poses. Currently, the evaluation camera pose is perhaps the same as training camera pose, lacking the changes in views, perhaps limited by generated video. I would take it as a benchmark for 4D reconstruction from generated video instead of a benchmark for general pose-free 4D reconstruction methods.

As for the novelty, despite the explanation of the authors, I think this project is without significant technical innovation.


But as the first attempt to reconstruct 4D content from generated video, this work is encouraging. Thus, I'll keep my positive score. The authors are suggested to test their work on various video generation models in the future.
Sincerely thank you for your feedback. We have some further responses w.r.t. (a) 4D reconstruction and (b) special designs for generated videos.

**(a) 4D reconstruction**

Since our focus is on the 4D reconstruction, to the best of our knowledge, there are no **pose-free** benchmarks for **dynamic scenes**. To address your concern, we build the benchmark by collecting openly available videos from the SORA official webpage. We then strictly compare our method against existing state-of-the-art 4D reconstruction methods. We provide the details and results below.

**Benchmark details:** We collect 35 sub-videos from the SORA [1] webpage, including Drone_Ancient_Rome (20 seconds in total, split into 4 sub-videos), Robot_Scene (20 seconds in total, split into 3 sub-videos), Seaside_Aerial_View (20 seconds in total, split into 3 sub-videos), Mountain_Horizontal_View (17 seconds in total, split into 3 sub-videos), Snow_Sakura (17 seconds in total, split into 4 sub-videos), Westworld (25 seconds in total, split into 4 sub-videos), Chrismas_Snowman (17 seconds in total, split into 3 sub-videos), Butterfly_Under_Sea (20 seconds in total, split into 3 sub-videos), Minecraft1 (20 seconds in total, split into 4 sub-videos), Minecraft2 (20 seconds in total, split into 4 sub-videos).

**Evaluation method:** We follow the standard pipeline for dynamic reconstruction (Hyper-NeRF, SC-GS, etc), to construct our evaluation setup by selecting every fourth frame as a training frame and designating the middle frame between each pair of training frames as a validation frame.

**Results:**

| Method | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ |
| :---: | :----: | :---: |  :---: |
| Deformable-GS | 12.72 | 0.5773 | 0.2861 |
| 4D-GS | 12.15| 0.5609 | 0.2926 |
| SC-GS | 14.81 | 0.5914 | 0.2420 |
| SpacetimeGaussians | 13.24 | 0.5836 | 0.2633 |
| Ours without field initialization (Sec. 3.3) | 15.42 | 0.6167 | 0.2268 |
| Ours without dual branch refinement (Line 187) | 18.57 | 0.6852 | 0.1945 |
| **Ours (full model)** | **19.05** | **0.7323** | **0.1839** |


Upon acceptance, we will open-source this benchmark and the corresponding codebase to ensure its reproduction. Besides, since currently there are no pose-free benchmarks for dynamic scenes, we believe this built benchmark is a contribution. 

Our full model achieves 4.24 PSNR improvement compared to the existing best method. The result proves the superiority of our method and each element we propose (field initialization, dual branch refinement) on the pose-free dynamic scene benchmark.

**(b) Special designs for generated videos**

As we summarized in the common response, properties of generated videos include both larger-scale aspects (unknown poses and unexpected movement) and small-scale aspects (flickering, floater occlusion).

- For larger-scale aspects, we propose the Field initialization stage which provides a proper start for our Dynamic Gaussian Surfels (DGS) regarding both the pose and the movement (please see warping transformation in Eq. 6 of our main paper). Here we'd like to highlight that the field initialization also benefits movement learning since the warping transformation is learned as a continuous field. This design is novel and especially beneficial for generated videos. We will provide more details of the field initialization during revision.

- For small-scale aspects, we have proposed the Dual Branch Refinement (Line 187) and provided ablation studies to prove its effectiveness in alleviating flickering.

Again we are grateful for your feedback.

[1] Video Generation Models as World Simulators.
In our work, we have carefully considered the potential ethical risks associated with generative models, particularly in video content creation. To address these concerns, our model includes robust safety mechanisms designed to screen and prevent any misuse. Furthermore, we have decided to release only the reconstruction code, ensuring that our contribution does not facilitate the generation of content that could raise ethical issues. We believe these measures effectively mitigate potential ethical risks and align with the standards of the community.
Thanks for the efforts of the authors for solving my concerns. My feedback is as follows:
1. Novelty: 

a) This paper is not the first work of generating 4D content using text-to-video models. Previously there are many 4D generation works using text-to-video models, such as 4Dfy (CVPR2024), AYG (CVPR2024), Dream-in-4D(CVPR2024), DG4D (arxiv2023), 4DGen(arxiv2023), aniamte124(arxiv2023), etc. Besides, those works generate 360-degree dynamic objects utilizing text-to-video models. In contrast, this work only generates part of the object, which means the invisible part in the input video is missing in generation results, and the novel view synthesis is limited to small camera movement range.

b) Based on the above point, I would suggest the authors to claim that they are the first work which deals with 4D reconstruction from generated videos, rather than the first 4D generation work using text-to-video models.


c) I reserve my judgment on the novelty of the 4D reconstruction techniques proposed by the author. I don't think the proposed techniques have special design to handle obvious multi-view inconsistency. (In fact, I didn't see the word "multi-view inconsistency" or similar words in the main paper)


3. Please refer to 1

4. The comparison with SC-GS/D-NeRF/ etc. on dataset w/o gt pose has little meaning. They are not specially designed for no gt scenes. It's suggested to compare with works specially designed for pose-free scenes. For comparison on dataset w/ gt pose, considering there are many regularizations integrated in the proposed pipeline, comparable or slightly better performance is expected.

So I decide to maintain the score.