4Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Models
Authors: Heng Yu
Chaoyang Wang
Peiye Zhuang
Willi Menapace
Aliaksandr Siarohin
Junli Cao
Laszlo Attila Jeni
Sergey Tulyakov
Hsin-Ying Lee
Venue: NeurIPS 2024 poster (2024)
Keywords: 4D generation;  novel view synthesis; gaussian splatting; text-4D; 4D reconstruction
Primary area: generative_models
TL;DR: Generate 4D Gaussian splatting for photorealistic scene rendering from text input

Abstract: Existing dynamic scene generation methods mostly rely on distilling knowledge from pre-trained 3D generative models, which are typically fine-tuned on synthetic object datasets.
As a result, the generated scenes are often object-centric and lack photorealism. 
To address these limitations, we introduce a novel pipeline designed for photorealistic text-to-4D scene generation, discarding the dependency on multi-view generative models and instead fully utilizing video generative models trained on diverse real-world datasets. 
Our method begins by generating a reference video using the video generation model.
We then learn the canonical 3D representation of the video using a freeze-time video, delicately generated from the reference video.
To handle inconsistencies in the freeze-time video, we jointly learn a per-frame deformation to model these imperfections.
We then learn the temporal deformation based on the canonical representation to capture dynamic interactions in the reference video. 
The pipeline facilitates the generation of dynamic scenes with enhanced photorealism and structural integrity, viewable from multiple perspectives, thereby setting a new standard in 4D scene generation.

4Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Models
Heng 12+f
Chaoyang Wang
Peiye Zhuang' Willi Menapace'
Aliaksandr Siarohin'
Junli Cao!
László A Jeni? Sergey Tulyakov! Hsin- Hsin-Ying Lee
Snap Inc. CCrneeie Mellon University
Project page: http://aapp.eeearcchiiu.bb...o.
"A baby eating ice-cream."
View #1
Time #1
View #2
Time #2
Canonical 3DGS
Viewpoint
Two reading books square.
"wo pandas playing cards.
Deformation
Time
"A blue monster reading book.
Figure 1: 4Real is 4D generation framework that generate near-photorealiste dynamic scenes from text prompts. We use deformable 3D Gaussian Splats (D-3DGS) to model the scene. After generation, We can view the generated dynamic scenes any timestep from different camera poses.
Abstract
Existing dynamic scene generation methods mostly rely on distillngg knowledge from pre-trained 3D generative models, which typically fine-tuned on synthetic object datasets. As a result, the generated scenes are often object-centric and lack photorealism. To address these limitations, we introduce novel pipeline designed for photorealistic text-to-4D scene generation, discarding the dependency on multi-view generative models and instead fully utilizing video generative models trained on diverse real-world datasets. Our method begins by generating reference video using the video generation model. We then learn the canonical 3D representation of the video using a freeze- video, delicately generated from the reference video. To handle inconsistencies in the freeze-time video, we jointly learn a per-frame deformation to model these imperfections. We then learn the temporal deformation based on the canonical representation to capture dynamic interactions in the reference video. The pipeline facilitates the generation dynamic scenes with enhanced photorealism and structural integrity, viewable from multiple perspectives, thereby setting new standard in 4D scene generation.
Equal contribution.
Work done during internship at Snap Inc.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
As industries ranging from film production to virtual reality seek increasingly immersive and interactive experiences, the ability to generate dynamic 3D scenes over eemeeestttallly. 4D environments- promisss to revolutionize how we interact with digital content. Recently, significant advances in image and video generation have been driven by large-scale text-image and text-video datasets, along with the development diffusion models. Furthermore, image diffusion models have been adapted into 3D-aware multi-view generative models through fine-tuning with limited 3D data, serving as foundational priors for 3D and 4D generation.
In this work, we focus on photorealistic text-to-4D scene generation, Existing 4D generation pipelines, due to the lack of 4D data, typically employ image, multi-view, and video generation models as priors to synthesize 4D samples. However, the multi-view models, which provide critical 3D information, are fine-tuned on static and synthetic 3D assets. As result, current generated 4D results are predominantly object-centric, lacking photorealism, and limited in their ability to capture complex interactions between objects and environments.
In response, we propose 4Real, a novel pipeline designed for photorealistic dynamic scenes with dynamic objects within the environment. We discard the reliance on multi-view generative models and exploit video generative models trained on large-scale real-world videos, covering more diverse and general appearance, shape. motion, and the interaction between objects and environments. Compared to existing methods purely relying on score distillation sampling, the proposed pipeline provides more flexible use cases, more diverse results, and requires fewer computations.
The proposed method adopts deformable 3D Gaussian Splats (D-3DGS) as the representation for dynamic scenes. The pipeline unfolds in three steps. First, we begin by creating reference video featuring a dynamic scene with a pre-trained text-to-video diffusion model. Next, we reconstruct canonical 3D representation from selected frame of the reference video. To achieve this, we generate a 'frezzettime video with camera motion and minimal object movement by applying dataset context embedding and prompt engineering to the video diffusion model. However, practically, the generated video still may contain object motion and is prone to be geometrically inconsistent despite looking temporally smooth. We propose to recognize the inconsistencies as per-frame deformation with respect to the canonical representation and learn these deformations simultaneously with the canonical representation. Finally, we reconstruct temporal deformation from the reference video with the learned canonical representation. We choose a video score distillation sampling (SDS) strategy that renders two types of videos: fixed- viewpoint video with a static camera, and a freeze- time video with fixed time step. The fixed-viewpoint video helps learn temporally consistent motions, while the freeze-time video learns multi-view geometrical consistency.
4Real achieves text-driven dynamic scene generation with a mear-photorealistic appearance and realistic 3D motions. The generated scenes are viewable from different viewing angles and over time, as shown in Figure 1. Our contributions are summarized as follows:
We propose 4Real, the first photorealistic text-to-4D scene generation pipeline. Without reliance on biased multi-view image generation models trained with specialized datasets, the proposed pipeline can generate more diverse and eear-photoreaalistcc results with dynamic objects within realistic environments.
We leverage text-to-video diffusion to generate target reference videos and auxiliary freeze- time videos, thereby transforming the generation problem into a reconstruction problem and reducing reliance on tine-consuming score distillation sampling steps. We learn from the freeze- time videos canonical 3DGS along with per-frame deformation modeling video inconsistency, and learn from the reference videos temporal deformation.
The proposed pipeline provides users flexibility in selecting and editing videos that they want to lift to 4D, and can generate high-quality samples in more reasonable computation budget, taking 1.5 hours on an A 100 GPU compared to 10+ hours with competing methods [2, 72].
2 Related Works
Text-to-video generation. Our pipeline leveraged the advancements in video generation models, which have been spurred by the recent success of text-to-image diffusion models. Some approaches

---

Reference Vidco
Text-
"Two pandas play cards...."
Image Conditioned Video Generation
Autoregressive Generation
Froeze-time Video
Autoregressive Generation
Prompt engineering
Context Embedding
"Camera circulating around
"play cards"
"time elapse"
Static-object Dataset
Figure Generate reference and freeze-time videos. Given the input text prompt, we first generate reference video using text-to-vides diffusion model. The reference video will be our target to perform 4D reconstruction. In order to obtain canonical Gaussian Splats, we generate freeze- time video by performing coune-coniiioned video generation with prompt engineering and context embedding. We further perform auto-regressive generation to expand view angle coverage.
extend these capabilities by directly fine-tuning pre-trained image diffusion models [3, 4] or by integrating motion modules while maintaining fixed image models [17, 34, 71]. Others utilize both image and video data using 3D-Uet [20] and cascaded temporal and spatial upsampler [19, 48). More recently, some efforts [5, 35] propose to replace the Unet architectures with transformer-based architectures [9, 41] for improved quality, scalability, and training efficiency.
Object-centric 3D and 4D generation. The challenges of generating 3D and 4D data mostly lie in the scarcity of available data, due to the costly nature of data capture. Conventional 3D generative models [1, 10-12, 22, 28, 38, 50, 57] that train limited 3D data in various representation fall short in quality and diversity, while 2D diffusion models have shown incredible generalizability using large-scale data. To bridge the gap, Score Distillation Sampling (SDS) [42] and its variants [8, 27, 56, 58, 73] have been developed to utilize pre-trained text-to- tmagg models as priors to generate 3D objects by optimizing parametric spaces for 3D representations. Recently, multiview images have emerged as a popular representation in 3D generation. Multiview image diffusion models [30-32, 47] fine-uned from image diffusion models using synthetic object dataset [13]. To generate 3D objects, the multiview image diffusion models are used as approximate 3D priors [44, 62], or providing intermediate medium followed by 3D reconstruction methods [21, 25, 32, 51, 59, 65, 74]. Meanwhile, video diffusion models have also been explored as priors for some early 4D generation attempts [49] or data generators to provide multiview supervision for 3D model [52]. More recently, image, multiview, and video diffusion models are jointly used priors for 4D generation [2, 29, 45, 68, 72]. However, they typically produce results biased toward object-centric and non- photoorreealii outputs as the 3D priors are obtained from synthetic object datasets.
4D Reconstruction. Neural Radiance Fields (NeRFs) [36] has revolutionized 3D scene reconstruction and novel view synthesis. The breakthroughs have been expanded to dynamic seenes. There are two major branches of methods for dynamic NeRFs: one views them a 4D representation by extending the radiance fields with a time dimension [14, 16, 26, 53, 55, 64], while the other represents dynamic scene by canonical space coupled with deformation field that maps local observation to this space [15, 39, 40, 43, 54, 66]. Recently, 3D Gaussian Splatting (3DGS) [23] has been introduced as a promising representation that facilitates fast neural rendering using explicit Gaussian particles. 3DGS has then been applied to model dynamic scenes with the similar idea of building deformation field [33, 63, 67, 69]. We adopt deformable 3D Gassian Splats (D-3DGS) as our dynamic scene representation.
3 Method
Given input text prompts, we aim to generate photorealistic dynamic scenes including animated objects and detailed backgrounds that exhibit plausible relative scales and realistic interactions with moving objects. We adopt deformable 3D Gaussian Splats (D-3DGS) representation. First,

---

utilize a text-to-video diffusion model to create reference video with dynamic scenes. Next, frame from this reference video serves as the conditional input for the video diffusion model produce freeze-time video featuring circular camera motion and minimal object movement. Subsequently, we reconstruct the canonical 3D representation from the freeze-time video. Finally, we reconstruct temporal deformations to align with the object motions in the reference video.
In the following sections, we will first discuss the 4D Gaussian splats representation for dynamic scenes in Section 3.1, the generation of freeze-time videos in Section 3.2, the reconstruction of canonical 3DGS Section 3.3 and the temporal reconstruction in Section 3.4.
3.1 Dynamic Scene Representation
We employ Deformable 3D Gaussian Splats (D-3DGS) to model dynamic scenes, a popular choice for dynamic novel view synthesis tasks due its low rendering computation cost, state-of-the-art rendering fidelity, and compatibility with existing graphics rendering pipelines [33, 63, 67, 69]. D-3DGS consists of 3D Gaussian splats that represent the static 3D scene at a canonical frame and deformation field that models dynamic motions.
Canonical 3D Gaussian splats. 3D Gaussian Splats [23] consist of set of 3D Gaussian points, each associated with an opacity value a and RGB color E R The position and shape of each 3D Gaussian point represented by Gaussian mean position x E R and covariance matrix, which factorized into orientation, represented by quaternion g E RA, and scaling, represented by E R3. 3DGS are rendered by projecting Gaussian points the image plane and aggregating pixel values using NeRF-like volumetric rendering equations.
Deformation field. To represent motion in 4D scenes, employ deformation field Wt-deform that represents the offset 3DGS from the canonical frame to the frame at time t:
(1) The position and orientation of 3DGS at time are simply obtained by x; x Ax, and q, + Aq: We implement w using a multi-layer perceptron (MLP), which serves as a general representation for arbitrary deformations. It is worth noting that this can be substituted by more specialized representations, such as linear blend skinning, which is more suitable for articulated motions. In this work, we prefer MLPs due to their simplicity and generality.
We deform (x, t) (Ax,, Aq.).
3.2 Generating Reference and Freeze-time Videos
Given text prompt, we generate a reference video using video diffusion models. First, we would like obtain the canonical 3D Gaussian splats for this reference video by creating a freeze-time video conditioned on a selected frame. This process requires a video generation model that can: 1) perform image-to-video generation, ensuring the output video frames remain consistent with the input image, and 2) generate videos with substantial camera motion while maintaining a relatively static scene.
We evaluated several recent video models accessible to the authors, including SVD [3], SV3D [52], VideoCrafter [7], and Snap Video Model [35]. SVD and VideoCrafter support image-to-ve to--oooviooo generation but have limited camera movement and introduce random motions in non-rigid objects. SV3D creates object-centre
360 videos with a white background, unsuitable for the scene-level
generation. The Snap Video Model aligns most closely with requirements, offering the required capabilities for our application.
Frames-conditioned pixel-space video diffusion. The Snap Video Model is text-to-video diffusion model that diffuses pixel values instead of latents, resulting in less motion blur and fewer gridding effects, which more common in latent diffusion models. It also tends to produce larger camera and object motions [35]. We use a variant trained to generate videos from both text prompts and an arbitrary set of video frame inputs. This feature enhances flexibility for autoregressive generation, particularly when creating freeze-time videos with extensive view coverage. Details of training and inference are in the Supplementary.
Context conditioning and prompt engineering. The Snap Video Model is trained on mixed video datasets, with unique context embedding for each dataset to guide the model in generating videos that follow the specific distribution. One dataset consists of statie real-world objects with circular camera trajectories, helping the model generate freeze-time videos with large camera motions and

---

Freeze-time Video
Reference Camera Pose
Reference Video
Canonical 3DGS
Camera Pose
Recon
Recon
fermframe Deform:
Temporal Deform: We-deform
Wmv-deform
Score Distillation Sampling (SDS)
(i) Multi-view SDS
Condition Frame
Sample Cam. Traj. Freeze
Noised Frames
Crop
(ii) Temporal SDS
DownSample
Vidco Diffusion Model
SDS Loss
Fix Camera
Samplet
Denoised Frames
Figure Reconstructing Deformable 3DGS. First, we reconstruct the canonical 3DGS using the freeze-time video. As the generated freeze-time video may still contain object motions and geometric inconsistencics, we propose to model these imperfections as per-frame deformations that learned jointly with the canonical 3DGS. Then, reconstruct the temporal deformation from the reference vidco with the learned canonical 3DGS. In addition to the reconstruction loss and motion regularization, propose video SDS strategy. The video SDS strategy includes multi-view SDS, using videos freeze time and sampled camera trajectory, and a temporal SDS, using videos of fixed camera and sampled time steps. & denotes learnable parameters.
minimal object movements when its context embedding is used. The success rate be further improved by adding phrases such as "Camera circulating around" and "time elapse" to the beginning and end of the input prompt, while omitting action verbs. Despite this, limited by the current capability of the Snap Video Model, multi-view inconsistency and moderate object motion may still occur, which our reconstruction steps in Section 3.3 address.
Extending view coverage. The raw output from the Snap Video Model is sequence of 16 frames, resulting limited view coverage. To extend coverage, perform iterative autoregressive generation. At each step, we condition on the 8 previously generated frames, guiding the model to generate new frames that follow the camera trajectory of the prior frames. We extend the view coverage in both forward and backward directions until reaches a sufficient amount, such as 180 Covering 360 is currently challenging due to background clutter and the model's limitation in generating loop videos.
3.3 Robust Reconstruction of 3DGS from Noisy Freeze-time Videos
Constrained by the capability of current video models, generated freeze-time videos may still contain multi-view inconsistencies, preventing direct reconstruction of high-quality static 3DGS. These inconsistencies can be: 1) temporally consistent but geometrically incorrect 2D video, and 2) videos with object motions. To address these imperfections, we propose the following treatments.
Representing multi-view inconsistency as deformation. We treat multi-view inconsistency as per-frame deformation with respect the canonical 3DGS. Specifically, we introduce another deformation field W'ultt-viow (x, k) (Ax,, Aqu) similarly defined as in Eq. (1), which outputs changes position and orientation of a Gaussian point the canonical position x with respect its new position and orientation at the k-th frame. We choose the canonical frame to be the one from the reference video that is used as the conditional input to generate the freeze-time video. This implies zero-vale valued outputs for the deformation field at the canonical time frame.
5

---

We jointly optimize the deformation field Wmultiview and the parameters for the 3DGS by minimizing a combination of the following losses.
Image reconstruction loss computes the difference between images Ios rendered by the deformed 3DGS and the frames Ig from the freeze-time video.
Lroon
Σ I
[ + Axx, + s, -Ix
(2)
where k denotes the frame index, Pa represents the camera projection matrix which is initialized using Colmap [46], then jointly finetuned with the 3DGS, and with some abuse of notations, (x Ax2: Q + Aqk: ax, c) denotes parameters for the deformed 3DGS.
Small motion loss. Minimizing Leesn alone is insufficient to obtain plausible 3DGS, since there can be infinitely many pairs of deformation fields and 3DGS that produce identical image renderings. To address this under-constrained problem, we introduce the assumption that the deformations should be as small as possible. Specifically:
small mo.
114xx111 + ||
(3)
Score distillation sampling (SDS) loss. Hand-crafted heuristics like Lsmall may be insufficient for plausible canonical 3DGS, especially with significant multi-view inconsistencies in the freeze-time videos. For improved regularization, in the later stage of optimization, we employ score distillation sampling. This method ensures that the rendering of canonical 3DGS on randomly sampled camera trajectories aligns with the distribution dictated by the video diffusion model.
During each optimization step, we sample camera trajectory by first randomly selecting camera pose Pa from the camera trajectories of the freeze-time video and then randomly perturbing it to create P. The trajectory [P [Px=1....16G] is uniformly interpolated between the canonical camera pose and P. We then render the canonical 3DGS using this sampled trajectory to generate video frames:
Ta = Ios q. s, a, c, Pk). Vk: I,..., 16.
Ik To perform score distillation sampling, we add Gaussian noise to the rendered images can-Uis and then use the video diffusion model to produce one-step denoised images. Since the sampled camera trajectory originates from the canonical camera pose, the first frame Z of the rendered videos is identical to the corresponding frame I. from the freeze-time video, allowing us to substitute To T GS with Ie,, which can then serve as the conditional frame for the diffusion model. More specifically:
(1z, La, [ = Jdenoise (Pean-o E2, Tean-Gs + ..., I Tea--Gs + ele|To: o),
where e's represent the noise applied to the video frames, a denotes the noise level for the current diffusion step, and 2, indicates the xo prediction for each frame. Using Ia as the image condition input is crucial because it allows the diffusion model to provide more precise supervisions, and sharper details that are aligned with the reference video.
Then SDS is implemented by minimizing the sum-of-squar difference between the rendered video frames I Gn and the denoised frames Ig as follows,
LSs = I - 112.
(6)
In this context, we use [-] to indicate that the variables are detached from the automatic differentiation computation graph. It is straightforward to show that the gradient of LSDS is proportional to the original gradient formulation by Poole et al. [42].
Sampling with low-res pixel-space diffusion model. The video diffusion model we employ is two-stage pixel-space model. It first generates video at low resolution of 36x 64, which is then upsampled to resolution of 288 512. For SDS, we employ only the first-stage low-resolution model, as it is significantly less computationally demanding and not straightforward to apply both stages simultaneously. Fortunately, utilizing just the low-resolution SDS is adequate to yield high-quality results. This effectiveness stems from our method relying more on reconstruction loss enhance fine details, rather than depending primarily SDS.

---

To compute the SDS loss, we first downsample the rendered video frames to low resolution. Before the downsampling, frames are randomly shifted and cropped mitigate aliasing effects.
Remarks on computational efficiency. Previous methods employing SDS are typically computa- tionally intensive, making them less practical for real-world applications. For instance, executing one step of SDS training using single precision on A100-80G GPUs with SVD can take #2 seconds, not including the time required for rendering. Moreover, GPU memory often becomes overloaded with SVD, which cannot compute gradients for more than four frames simultancously due to substantial memory demands for backpropagation through the image encoder.
In contrast, our method implements SDS using the low-resolution, pixel-space Snap Video Model. This model leverages compressed transformer architecture [9]. which is significantly faster, requiring 2x200 ms per step. It is also more memory-efficient, allowing for gradient computation across all 16 frames in a single batch. This efficiency makes our approach more viable for practical applications.
3.4 Reconstruct Temporal Deformation
After obtaining canonical 3DGS representation, we proceed to generate temporal motion by fitting the deformation field (as detailed in Eq. (I)) to align with the reference video.
Image alignment loss computes the similarity between the rendered frames Tōs and the reference video frames I, combining wixel-wise intensity loss and structural similarity index measure (SSIM) loss [60], i.e., Laiign = >. IT's Iell + ( Tz)-
Deformation regularization. To ensure that the deformation field learns plausible motions, incorporate losses that enforce both spatial and temporal smoothness in the motion trajectories of each Gaussian splat, following CoGS [69] and Dynamic 3DGS [33]. These as-rigid-as as ---igidd-a--possbble losses encourage minimal variation in both translational and rotational motions of Gaussian splats within their spatial and temporal neighborhoods. We refer readers to the respective papers for detailed treatment. Implementation details specific to our adaptation are included in the supplementary.
Joint temporal and multi-view SDS. Relying solely on hand-crafted regularization often leads to noticeable artifacts in novel view rendering. especially when the camera position deviates significantly from that in the reference video. Therefore, in the later stages of optimization, we employ SDS to ensure that the rendered video frames adhere to the distributions of real videos. For evaluating the SDS loss, we render two types of videos:
The camera remains static while we sample time steps to deform the 3D Gaussian Splats. To ensure that the first frame can be used as a conditional input for the video diffusion model, the sampled time steps always start from O. This initial frame is rendered using the canonical 3D Gaussian Splats without any deformation.
Fixed- Viewpoint Videos:
2. Freeze-Time Videos: With fixed time step, we sample continuous sequence of camera poses starting from the reference video's 's camera pose. This setup allows us to directly use the corresponding frame from the reference video the conditional input the video diffusion model.
Applying SDS loss to both fixed-viewpoint and freeze-time videos is crucial. The former regularizes D-3DGS for temporally consistent motions, and the latter ensures multi-view consistency, aligning D-3DGS motions with the reference video and avoiding multi-perspective. optical illusions.
Improving details by re-upsampling We optionally perform an additional step to enhance the rendering details further. We found that downsampling and then re-upsampling the videos rendered by Gaussian splatting results in fewer artifacts and higher fidelity. Therefore, we render video with continuously changing viewpoints and time, and then re-upsample using the upsampler of the video difffusion model. The resulting video serves reconstruction target to fine-tune the Gaussian splats, thereby improving the overall quality.
4 Experiment
We evaluated our method using a set of 30 text prompts sourced from related works [2, 35, 72]. Please refer to the included webpage for the complete results. As illustrated in Figure 4, our method successfully generates scenes featuring complex illumination and semi-transparent objects such as water. Additionally, it demonstrates flexibility in creating diverse multi-object scenes.
7

---

Complex Illumination
Diversity
Semi-Transparent Material
Multiple Objects
Figure 4: Generating challenging scenes, 4Real is able to generate dynamic scenes with complex illumination and (semi)- transparatn materials such as water. is flexible produce diverse content and generate multi-object scenes. Please refer to our supplementary webpage for the complete results.
Ours
4Dfy
Ours
Dream-in-4D
"A buby punda reuding book.
"A cut singing.
ARIE
building
eating
"A cute small cut sitting outing chicken wingy watching a movile
humanoid robot fipping codn.
Figure 5: Qualitative comparison state-of-the-arss object-centric 4D generation methods.
Implementation details. The optimization consists of two stages: first, reconstructing the canonical 3DGS, and then learning the temporal deformation. At each stage, we perform SDS only during the last 5k iterations, and anneal the diffusion time steps similarly the scheduler used in Hifa [73]. Throughout our experiments, we run 20k iterations per stage, totaling 1.5 hours on a single A100 GPU. This duration is significantly shorter than that required by recent state-of-the-art methods, which typically require over 10 hours [2, 72]. Details are included the Supplementary.
Compare to object-centric text-4D generation. To the best of our knowledge, there are currently no existing text-to-4D scene generation methods, so we choose evaluate our method against recent state-of-the-art object-centric text-to-4D generation methods. Specifically, we conduct user study against 4Dfy [2] and Dream-in-4D [72], which are based on NeRF representation. These methods are known to achieve some of the best visual quality compared to other methods [29, 45, 49] using the same representations.
The user study was conducted using Amazon Turk, involving 30 evaluators per video pair. In each session, evaluators were presented with two anonymized vidcos. Each video depicted dynamic object or scene, with the camera moving along circular trajectory and stopping at four fixed poses to highlight object motions. We obtained 16 videos for 4Dfy and 14 videos for pream-in-4D from their respective project web pages. Evaluators were tasked with selecting their preferences based on seven criteria: motion realism, foreground/buckgroun photo-realism, 3D shape realism, generul realism, significunce of motion, and video-texi ulignment. As shown in Figure 6, our method outperformed the competition in every category. Sample frames are displayed in Figure 5, and videos used in the study are available in the supplementary materials.

---

Ours
4Dfy
Ours
Dream-in-
02: Foreground Photo Realssn
26-7
TL2
25.5
03: Background Photo
Q4: Shape Beation
102
28.8
20
32
03: Roaism
(6: Which is More dysumii
69.4
30.8
E6.3
33.2
07: diggnnent
20
40
60
80
1000
20
40
80
100
Figure 6: Comparative user study with sf object-centric 4D generation methods.
X-CLIP
Visual Quality
Temporal Consistencyl
Dynamic Degree
T-V Aligument
Factual Consistency
4Dly [2]
20:03
1.43
1.49
2.26
1:30
Ours
24.23
2.43
2.17
3.15
2.91
2.49
Dream-In-4D) [72]
19:52
1.34
1.37
3,02
2.27
1.30
AYG [29]
19:87
2.49
2.09
3.15
2.80
2.17
Ours
23.09
2.44
2.16
3.16
2,00
2.50
Table 1: Quantitative comparison against baselines using X-CLIP [37] and VideoScore [18].
Bascline
Ours
Baseline
Ours
Baseline
Figure 7: Qualitative comparison against baseline of 3D scene generation 4D object-centric generation. Our result is more natural in terms of object placement, motion and lighting.
In addition to existing methods, we also implement and compare with straightforward baseline: combining 3D scene generation with 4D object-centric generation. Although straightforward. implementation with high quality is challenging due to: (1) Inserting 3D objects with realistic and physically correct placement not trivial. Common issues include floating objects, misaligned scales between objects and background, and unrealistic scene layout. (2) Generating object motions that align well with the backgrounds difficult, such as ensuring generated target subject sits on sofa rather than stands on it. (3) Complex procedures are needed to relight objects to match environment lighting. We tried our best to create few baseline results with heavy manual efforts, as shown in Figure 7. We create the 3D background by removing the object from our generated freeze-time video. We then employ 4Dfy [2] to generate the foreground objects. Finally, we manually insert the objects into the background with the most plausible position and scale we can find. Despite these efforts, we find the inserted object appears disconnected from the background, especially when compared to results.
On top of user study, we also provide quantitative results in Table using employ the X-CLIP score [37], minimal extension of the CLIP score for videos. We also run VideoScore [18), video quality evaluation model trained using human feedback, which provides five scores assessing visual quality, temporal consistency, text-video alignment, and factual consistency. Our method significantly outperformed other methods in these metrics. However, we remain conservative about the effectiveness of these metrics since they have not been thoroughly studied and evaluated yet.
Ablation studies. We diagnose our system by removing individual components from the pipeline.
Per-frame deformation is crucial for reconstructing high-quality canonical 3D representations from noisy freeze-time videos. As illustrated in Figure 8, the absence of per-frame deformation results in significant artifacts, such as on the faces of the panda and the cat, and in blurrier areas like the boundaries of doors in kitchen.
Small motion regularization employed our pipeline. While view-dependent deformation can improve underfitting, it can also lead to overfitting to the reference video, resulting in noisy reconstruction. We thus adopt a regularization loss of the magnitude of the deformation to mitigate the issues. As shown in Figure 9.

---

Per-frame Deform
Miew SDS
Freeze-time Video
Joint Temp. & MV SDS
with
with
w/o
with
with
Figure 8: Ablation study of the impact of removing each component from the proposed pipeline.
w/o deform w/o reg
full model
w/o deform w/o reg
full model
Figure 9: Ablation per-view deformation and small motion regularization. Removing per-view deformation results in underfitting the regions which are not geometrically consistent. Removing small motion regularization of the per-view deformation field causes overfitting the freeze-time field, leading to noisy results.
Multi-view SDS aids in noise reduction and enhances shape realism. For example, as illustrated in Figure 8, it improves the depiction of the human head in the background, the upper boundary of the sofa, and the limbs of the goat.
Freeze-time video: Removing the step of generating freeze-time videos and the corresponding reconstruction loss Lieenn will result in systematic failure. This partly because apply SDS only in the later stages of training, which is insufficient to correct significant reconstruction errors.
Joint nulti-view and temporal SDS helps improve the sharpness of the final result during the reconstruction of temporal deformation. also helps prevent erroncous artifacts, such as the floating spot the edge of the cat's ear and the dark shadows above the cat's back, shown in Figure 8.
We have included additional samples from the ablation study in the supplementary web page.
Discussion and Conclusion
We propose 4Real, which, to knowledge, is the first method capable of generating near- photorealistic 4D scenes from text inputs.
However, our method has limitations: 1) It inherits limitations from the underlying video generation model, such as video resolution, blurriness or artifacts during fast motion, particularly with thin or small objects, and occasional text-video misalignment. 2) Reconstructing from videos with dynamic content challenging, and our method may fail due inaccuracies in camera pose estimation, rapid movements, the sudden appearance and disappearance of objects, and abrupt lighting changes. 3) Our method does not produce high-quality geometry such as meshes, due to the limitations of using 3DGS. 4) It still takes over hour generate a 2-second 4D scene.
For future works, we envision the availability of stronger video generation models with more accurate camera pose and object motion control [6, 61], incorporating cross-frame attention [70] when generating freeze-time videos and utilizing feedforward 3D reconstruction [51] would help address these limitations.
10

---

References
[1] Panos Achlioptas, Olga Diamanti, Joannis Mitliagkas, and Leonidas Guibas. Learning representations and generative models for 3d point clouds. IMCL, 2018. 3
[2] Sherwin Bahmani, Ivan Skorokhodow, Victor Rong. Gordon Wetzstein, Leonidas Guihas, Peter Wonka, Sergey Tulyakov, Jeong Joon Park, Andrea Tagliasacchi, and David B Lindell. 4d-fy: Text-to-4d generation using hybrid score distillation sampling, In CVPR, 2023. 2. 3. 7,
[3] Andreas Blattmann, Tim Dockhor, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, al. Stable video diffusion: Scaling latent video diffusion models to large datasets, arXiv preprint arXi::231..11227. 2023. 3,4
[4] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis, Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, 2023. 3
[5] Tim Brooks, Bill Pechles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL ctcpar//ppaai..oo/rreeecca com/research/ video- www.retinneenneeeeeeee
as- bas-wold--a sorrldd-imllttor...
[6] Changgu Chen, Junwei Shu, Lianggangxu Chen, Gaoqi He. Changbo Wang, and Yang Li. Motion- zero: Zero-shot moving object control framework for diffusion-hased video generation. arXiv preprint arXiv:2401.10150, 2024. 10
[7] Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models, 2024. 4
[8] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. arXiv preprint arXiv:2303. 13873,, 2023. 3
[9] Ting Chen and Lala Fit: Far-reaching interleaved transformers. arXiv preprint arXiv:2305.12683 2023.
3,
[10] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. CVPR, 2019.
[11] Yen-Chi Cheng, Hsin- Ying Lec, Sergey Tulyakov, Alexander G Schwing, and Liang- Yan Gui. Sdfusion: Multimodal 3d shape completion, reconstruction, and generation. In CVPR, 2023.
[12] Zezhou Cheng. Menglei Chai, Jian Ren, Hsin- Ying Lee, Kyle Olszewski. Zeng Huang, Subhransu Maji, and Sergey Tulyakov. Cross-modal 3d shape generation and manipulation. In ECCV, 2022.
[13] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In CVPR, 2023. 3
[14] Yilun Du, Yinan Zhang, Hong-Xing Yu, Joshua B Tenenbaum, and Jiajun Wu. Neural radiance flow for 4d view synthesis and video processing. In ICCV, 2021. 3
[15] Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nicßner, and
Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIGGRAPH Asia, 2022. 3
[16] Chen Gao. Ayush Saraf. Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocular video. In ICCV, 2021. 3
[17] Yuwci Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. In ICLR, 2024. 3
[18] Xuan He, Dongfu Jiang. Ge Zhang, Max Ku, Achint Soni, Sherman Siu, Haonan Chen, Abhranil Chandra, Ziyan Jiang, Aaran Aruiraj, et al. Mantisscore: Building automatic metries to simulate fine-grained human feedback for vidco generation. arXiv preprint arXiv:2406. / 15252, 2024.
[19] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole. Mohammad Norouzi, David Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.0230: 2022. 3
[20] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. In NewrIPS, 2022. 3

---

[21] Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. In ICLR, 2024. 3
[22] Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint arXiv:2305.0244 0004300000004,, 2023. 3
[23] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM TOG, 2023. 3, 16, 17
[24] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
2014. 17
[25] Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model. ICIR, 2024.
[26] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis dynamic scenes. CVPR, 2021.
[27] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. In CVPR, 2023. 3
[28] Chieh Hubert Lin, Hsin- Hsin-Ying Lee, Willi Menapace, Menglei Chai, Aliaksandr Siarohin, Ming- Yang. and Sergey Tulyakov. Infinicity: Infinite-scale city synthesis. In ICCV, 2023. 3
[29] Huan Ling, Seung Wook Kim, Antonio Torralba, Sanja Fidler, and Karsten Kreis. Align your gaussians: Text-to-4d with dynamic 3d gaussians and composed diffusion models. arXiv preprint 2312:331,,,3366, 2023. 3, 8.9
[30] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. 7ero-1-to-3: Zero-shot one image to 3d object. In ICCV, 2023. 3
[31] Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Synedreamer: Generating multiview--onniitennt images from single-view image. In ICLR, 2024.
[32] Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjic Liu, Yuexin Ma, Song-Hai Zhang, Mare Habermann, Christian Thcobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. arXiv preprint arXiv:2310. 15008, 2023. 3
[33] Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. arXiv preprint arXiv:2308.09)) 09713, 2023. 3, 4, 7, 15
[34] Aniruddha Mahapatra, Aliaksandr Siarohin, Hsin- Ying Lec, Sergey Tulyakov, and Jun-Ya Zhu. Text- guided synthesis of culerian cinemagraphs. ACM TOG, 2023. 3
[35] Willi Menapace, Aliaksandr Siarohin, Ivan Skorokhodov, Ekaterina Deyneka, Tsai-Shien Chen, Anil Kag. Yuwei Fang, Aleksei Stoliar, Elisa Ricci, Jian Ren, al. Snap video: Sealed spatiotemporal transformers for text-to-video synthesis. In CVPR, 2024. 3, 4,7
[36] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 2021.
[37] Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling. Expanding sanguae--imag pretrained models for general video recognition. In ECCV, 2022. 9
[38] Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint arX1::221..0875) 2022. 3
[39] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Brrtin- Nerfies: Deformable neural radiance fields. In ICCV, 2021. 3
[40] Keunhong Park, Utkarsh Sinha, Peter Hcdman, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M. Seitz. Hypemerf: A higher-dimensional representation for topologically varying neural radiance fields. ACM Trans. Craph., 2021. 3
[41] William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023.
12

---

[42] Ben Poole. Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In ICLR, 2023. 3,6
[43] Albert Pumarola, Euric Corona, Gerard Pons-Moll, and Francese Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In CVPR, 2021. 3
[44] Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin- Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, al. Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors. In ICLR, 2024.
[45] Jiawei Ren, Liang Pan. Jiaxiang Tang. Chi Zhang, Ang Cao. Gang Zeng. and Ziwei Liu. DreamgaussianAd: Generative 4d gaussian splatting. arXiv preprint arXiv:23/2.17142, 2023. 3,
[46] Johannes Lutz Schönberger and Jan- Michael Frahm. Structure- wor---froomoooooo revisited. In CVPR, 2016. 6,
17
[47] Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. In ICLR, 2024. 3
[48]
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. In ICLR, 2023. 3
[49] Uriel Singer, Shelly Sheynin, Adam Polyak, Oron Ashual, Turii Makarov, Filippos Kokkinos, Naman Goyal, Andrea Vedaldi, Devi Parikh, Justin Johnson, et al. Text-to-4d dynamic scene generation. arXiv preprint arXiv:2011. 11:30011//2800 //280, 2023. 3,8
[50] Edward Smith and David Meger. Improved adversarial systems for 3d object generation and reconstruction. In CoRL, 2017.
[51] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. I.gm: I Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint 24X1v:2402.00054, 2024.
3. 10
[52] Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion. arXiv preprint arXiv:2403.12009 2024. 3,
[53] Chaoyang Wang, Ben Eckart, Simon Lucey, and Orazio Gallo. Neural trajectory fields for dynamic novel view synthesis, arXiv preprint arXiv:2105.05994 2021. 3
[54] Chaoyang Wang. Lachlan Ewen MacDonald, László A. Jeni, and Simon Lucey. Flow supervision for deformable nerf. In CVPR, 2023. 3
[55] Chaoyang Wang, Peiye Zhuang, Aliaksandr Siarohin, Junli Cao, Guocheng Qian, Hsin Ying Lee, and Sergey Tulyakov. Diffusion priors for dynamic view synthesis from monocular videos. arXiv preprint arXiv:2401.05583, 2024. 3
[56] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Ych, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. CVPR, 2023.
[57] Tengfei Wang, Bo Zhang, Ting Zhang, Shuyang Gu, Jianmin Bao, Tadas Baltrusaitis, Jingjing Shen, Dong Chen, Fang Wen, Qifeng Chen, et al. Rodin: A generative model for sculpting 3d digital avatars using diffusson, In CVPR, 2023.
[58] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. arXiv preprint arXiv:2305.162 16213, 2023. 3
[59] Zhengyi Wang, Yikai Wang, Yifei Chen, Chendong Xiang, Shuo Chen, Dajiang Yu, Chongxuan Li, Hang Su, and Jun Zhu. Crm: Single image to 3d textured mesh with convolutional reconstruction model. arXiv preprint ar33:2403.00034 2024. 3
[60] Zhou Wang. Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: From error visibility to structural similarity. IEEE TIP, 2004. 7
[61] Zhouxia Wang, Ziyang Yuan, Xintao Wang, Tianshui Chen, Menghan Xia, Ping Luo, and Yin Shan. Motionctrl: A unified and flexible motion controller for video generation. arXiv preprint 2rXiv:2312.03641
2023. 10
13

---

[62] Haohan Weng. Tianyu Yang. Jianan Wang. Yu Li. Tong Zhang, CL Chen, and Lei Zhang. Consistent Improve consistency for image 3d object synthesis. arXiv preprint arXiv:2310.0S0P2 2023.
[63] Guanjun Wu, Taoran Yi, Jiemin Fang. Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic rendering. arXiv preprint arXiv:2310008528. 2023. 3,4
[64] Wenqi Xian, Jia-Bin Huang. Johannes Kopf, and Changil Kim. Space-tima neural irradiance fields for free-viewpoint video. In CVPR, 2021. 3
[65] Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang, Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, al. Dmv3d: Denoising multi-view diffuusion using large reconstruction model. In ICLR, 2024. 3
[66] Gengshan Yang. Chaoyang Wang. N. Dinesh Reddy, and Deva Ramanan. Reconstructing animatable categories from videos. In CVPR, 2023. 3
[67] Ziyi Yang, Xinyu Gao, Zhou, Shaohui Jiao, Yuqing Zhang, Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. arXiv preprint arXiv:2309131011 2023. 3,4,17 17
[68] Yuyang Yin, Dejia Xu, Zhangyang Wang, Yao Zhao, and Yunchao Wei 4dgen: Grounded 4d content generation with temporlll consistency. arXiv preprint urXiv:2312.17225, 2023.
[69] Heng Yu, Joel Julin, Zoltán Á. Milacski, Koichiro Ninumaa, and László A. Jeni. gaussian splatting. CVPR, 2024. 4,7,
Cogs:
Controllable
[70] Yifei Zeng, Yanqin Jiang, Siyu Zhu, Yuanxun Lui, Youtian Lin, Hao Zhu, Weiming Hu, Xun Cao, and Yao Yao. Stag4d: Spatial-temporal anchored generative 4d gaussians. arXiv preprint XXV:22403.149.99 2024.
[71] Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Tian. Controlvideo: Training-free controllable text-to-video generation. In ICLR, 2024.
[72] Yufeng Zheng, Xucting Li, Koki Nagano, Sifei Liu, Otmar Hilliges, and Shalini De Mello. A unified approach for text- image-guided scene generation. In CVPR, 2024. 2, 7,
[73] Joseph Zhu and Peiye Zhuang. Hifa: High-fidelity text-to-3d with advanced diffusion guidance. In ICLR, 2023. 3,8 8
[74] Peiye Zhuang, Songtang Han. Chaoyang Wang, Aliaksandr Siarohin, Jiaxu Zou, Michael Vasilkovsky, Vladislav Shakhrai, Sergey Korolev, Sergey Tulyakov, Hsin- Ying Lee. Gtr: Improving large 3d reconstruction models through geometry and texture refinement, arXiv preprint
2024.
14

---

A Masked Snap Video Model Details
We employ masked variant of the Snap Video model that is trained generate videos using both text and a set of known frames as conditioning. Given a set of known frames at arbitrary positions in the input sequence, the masked Snap Video model produces positions match the known frames. This mechanism allows the model to perform various tasks such image animation, frame interpolation and temporal extension of videos by providing known frames in the corresponding arrangements.
output video whose frames those
Tha masked model variant is obtained by extending the original Snap Video model's 's input channels by 3 to accomodate the conditioning frames input, and by training the model for an additional 200k steps using masked video modeling objective, while keeping the other training parameters unchanged.
In particular, we adopt a set of masking strategies which define the set of known frames provided as conditioning during training, each of which is randomly applied to input batch element with probability m:
Unconditional generation m = 0.6. Mask the whole input video. This setting corresponds to the original Snap Video setup, where no known frame is present.
Bernoulli m = 0.3. Mask each frames of the input video with probability (1 - 1/16), leaving on average single frame unmasked.
Frame interpolation m = 0.075. Mask all video frames apart from a set of frames sampled at regular time intervals.
Video extension m = 0.075. Mask the last N video frames.
B Deformation Regularization Details
Our motion deformation network predicts position offsets (Ax), rotation offsets (Aq), and scale offsets (As) for each time step. To prevent unnecessary movements (e.g.. the background) and to ensure movement consistency, we apply L1 norm regularization as follows:
1
Laorm
E(1Ax)) + ||qqil| + ||Asill). N
We introduce difference loss, denoted as Cains to ensure that the movement or trajectory of each Gaussian is consistent and smooth over time. This loss penalizes abrupt changes in the trajectory, promoting smoother transitions. The difference loss is formulated as follows:
N
Lai
|
Axi.-
(8)
N
1=1
Here, Axi, represents the position offset of Gaussian at time f. We also borrow the local-rigidity loss Erigid and rotational loss Lra from [33] (For more comprehensive understanding, please refer to their detailed paper). For the local-rigidity loss, a weighting scheme is applied that utilizes an unnormalized Gaussian weighting factor. This factor assigns different weights to each point based on its proximity to the center of the Gaussian distribution. The unnormalized Gaussian weighting factor defined as follows:
Wiy exp (~>ue |x.00 x
Using this weighting scheme, a local-rigidity loss is employed, denoted as rigid defined as follows:
rigid = wi.,ll(X,,,-1 x1,1-1 R2, R (x - x2,1l22
(10)
Crigid
k
cij
rigid
(11)
iEG j jkknn
15

---

This loss ensures that for each Gaussian i, neighboring Gaussians move in manner consistent with the rigid body transformation of the coordinate system over time. Specifically. this means that the relative positions and orientations of neighboring Gaussians are maintained as the coordinate system evolves, ensuring coherent and physically plausible motion pattern.
Additionally, we incorporate a rotational loss _ to ensure consistent rotations among neighboring Gaussians across different time steps. This loss function helps maintain the relative orientations of nearby Gaussians as they evolve, preserving the overall rotational coherence of the system. The rotational loss expressed as:
Lot
1 k|G
Σ Σ iEG jEknaas
Wi Whild - -..1
(12)
Here, á represents the normalized quaternion rotation of each Gaussian. The k-nearest neighbors, identified in the same manner as the preceding losses, are utilized to ensure consistent rotational behavior among neighboring Gaussians.
C Stage-wise Training
Our method consists of two main components: canonical space reconstruction and motion fitting.
1. Canonical Space Reconstruction
(a) Warm-up Stage:
Train the network for 3,000 iterations without any per-frame deformations to establish a baseline for the canonical space. This stage follows the same method as described in [23].
(b) Learning Deformations and GS Optimization:
Learn deformations for each frame and optimize the Gaussians over the course of 20,000 iterations.
Implement GS growth (including splitting, cloning, and pruning) using the strategy described in [23] during the first 15,000 iterations.
(c) Multi-view SDS:
Enable multi-view SDS in this stage from 15,000 iterations to help learn a good canonical space.
2. Motion Fitting
(a) Initial Motion Fitting:
Fit the motion over the next 10,000 iterations using only reference views at different timesteps.
Note: Cease GS optimization to focus solely on learning motion deformations.
(b) GS Growth for Motion Compensation:
Resume GS growth using the same strategy for an additional 5,000 iterations to address missing points in the canonical space caused by motion.
Note: Fix the learned canonical space from the previous step, and perform GS growth by focusing solely on the newly added Gaussians.
(c) Simultaneous Fine- Tuning:nn::
Fine-tune both the GS and the motion deformation network simultaneously using all training frames consisting of different camera views and timesteps.
(d) Joint Temporal and Multi-view SDS:
Enable temporal and multi-view SDS jointly in this stage when doing fine-tuning from 35,000 iterations and last for 5,000 iterations help learn consistent motion.
16

---

D Implementation Details
D.I Multi-view and Temporal SDS
During training with SDS, we alternately perform multi-view and temporal SDS, For each iteration, we randomly sample either a camera pose or a timestep. This sampling determines the specific spatial (view) dimension the temporal (time) dimension to train. By alternating between these two types of SDS, we ensure comprehensive coverage of both spatial and temporal aspects. This strategy enhances the overall consistency and robustness of the model. For each SDS step, randomly sample M = 16 frame poses and perform rendering, either following the sequence of camera movement or based on the order of time. It is important note that we perform noisy virtual trajectory sampling during multi-view SDS. Specifically, we randomly sample a camera pose that is different from the reference pose used generate the temporal video. Subsequently, we sample virtual camera pose within a specified radius of a circle centered on the sampled pose. By interpolating between this virtual camera pose and the reference camera pose, we generate MM 16 frame poses for rendering. This approach ensures a diverse set of viewpoints, enhancing the robustness and accuracy of the multi-view SDS process. When inputting our high-resolution rendered images into the low-resolution model for SDS, we first apply one of two preprocessing steps with equal probability. The image is either randomly shifted by S 1 pixels (where S is the downsampling scale) or randomly cropped to half its size. Following this, we downsample the image and input it into the low lowrresolution model. This preprocessing ensures that the low-resolution model receives varied inputs, enhancing the robustness of the training process.
D.2 GS Growth for Motion Fitting
During the GS growth for motion fitting, we fix the attributes (position, rotation, scale, opacity and color) of the learned canonical Gaussians by stopping the gradient flow to them. Instead, focus on performing splitting. cloning, and pruning operations based on the gradients of the newly added Gaussians. These new Gaussians are introduced at the beginning of the growth stage, specifically according to the gradients derived from the canonical Gaussians. This method ensures that the canonical structure remains stable while allowing the newly added Gaussians to adapt and optimize the motion fitting process.
D.3 Deformation Network
For the training process, we adopted the differential Gaussian rasterization technique from 3D GS [23] and utilized the same deformation network structure as described in [67] for both per- frame deformation and motion deformation. Both deformation networks consist of an MLP Fo: (7(x), >()) + (Ax, Aq, As), which applies position embedding 7 on each coordinate of the 3D Gaussians and time (or per-frame deformation code) and maps them to their corresponding deviations in position, rotation, and scaling. The weights 0 of the MLP are optimized during this mapping process. Each MLP Fe processes the input through eight fully connected layers, each employing ReLU activations and containing 256-dimensional hidden units, resulting in 256-dimensions feature vector. This feature vector is then passed through three additional fully connected layers (without activation functions) to independently output the offsets for position, rotation, and scaling over time. is worth noting that, similar to NeRF, the feature vector and the input are concatenated at the fourth layer, enhancing the network's 's ability capture complex deformations.
D.4 Training Setup
The initialization of the Gaussians was based on Structure-from Mottoon (SfM) results obtained from COLMAP [46]. We adhered the training settings specified in [23] for the canonical GS training. The learning rate for the positions of the Gaussians was set to undergo exponential decay, starting from 1.6 x 10-4 and decreasing to 1.6 x 10-6, Additionally, the learning rate for both scale and rotation parameters was maintained at 1 x 10-3 throughout the training process. The learning rate for the deformation network was set to exponentially decay from x 10-3 to 1 x 10-5, The optimization across these processes was conducted using the Adam optimizer [24] with 3 parameters set to (0.9,0 0.999). All experiments were performed on single NVIDIA A A100 GPUs with 80GB of memory.
17

---

E
Key Hyperparameters
E.1 Loss Weighting
For the regularization terms mentioned above, we assign a weight of 0.01 to each within the overall loss function, ensuring they contribute appropriately to the optimization process. For the reconstruction loss, we use a weight of 1. Additionally, for the SDS loss, we differentiate the weights based on the type of SDS being applied: weight of 20 for temporal SDS and weight of 5 for multi-view SDS. These weights are chosen to effectively capture motion dynamics and stabilize the training process.
E.2
GS Growth Threshold
For the GS growth stage during the canonical space reconstruction, we set the opacity threshold to To = 5 10-1 and the gradient threshold Tgrad 10-4 These thresholds help control the addition of new Gaussians based on their opacity and gradient values. In contrast, for the GS growth stage during motion fitting, we use a relatively higher opacity threshold of Ta = 10-2 to avoid introducing redundant Gaussians. This higher threshold ensures that only significant Gaussians are added, which aids maintaining efficiency and relevance during the motion fitting process.
User Study Details
The user study was conducted through single survey comprising seven questions. Each question asked the evaluators to compare two videos: one rendered with our method and the other with a different method using the same text prompt as input. Evaluators indicated their preferred video based on various aspects, as illustrated in Fig. 10. The evaluators were given the following instructions for each metric.
Motion Realism: Take into consideration of magnitude, smoothness, consistency of the motion. Pay close attention to deformed limbs human and animals and unnatural deformations.
Foreground Photo-realism: Assess realism of the appearance of the foreground object, do not consider the motion factor. That is the main object that looks more like taken with real video camera, pay attention unnatural color styles of the objects.
Background Photo-realism: Assess realism of the appearance of the background. That is the background that looks more like taken with real video camera, pay attention unnatural color or styles of the objects or bluriness.
3D Shape Realism: That is the video in which main object has the most natural shape, again pay attention deformed limbs of human and animals and unnatural deformations.
General Realism: Please provide your subjective appraisal of which video, in your opinion, stands out as superior based on appearance quality, 3D structure quality, and motion quality.
Significance of Motion: The video that contain the most amount of motion. Please exclude from consideration any random limbs deformations.
Video-text Alignment: That is which video reflect all the aspects included in the text description.
18

---

You description
regarding quality Please pay close attention to instructions and answer as thoughtully you each task one question one which acceptance the tank be
judged Please exclude eize of the video from any conoideration and attention only to the video content. consideration
Which
roalittic
smoothness,
close attention
animats
Which video has the most photo realistic foreground? Assess realism the appearance of the foreground cbject, not consicer the motion factor That the main object that stylea
taken with video
3) Which video has the most photo-realietic background? Assess reallom of the appearance of the background That the background that looks more like taken with real video bluriness.
attention
Which video has object better more realist shape? to the video which main object has most natural shaps, aggain pay attention deformed limos human and arimaass and unnatural deformations.
general looks
Which video most dynamics of? The video that contain the most amount of motion: Please exclude from considerati ation random limbs deformations.
sopects
diecciption
Fext Descriptione humanod robot playing the visin
Video A
Video
Which video has more mallltic motion?
Option
Option
Which video has the most photo-realistic foreground?
Option
Option
Which video has the most photo-realitiic beckground??
Oplion A
Option
object more realist shape?
Option
Option
general video more
Option A
Option
Which dynamic?
Option A
Option
following description?
Option A
Option
Figure 10: Screenshot of the user study webpage.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim to propose a pipeline that facilitates the generation of dynamic scenes with enhanced photorealism and structural integrity, viewable from multiple perspectives. Such claim is empirically verified with our generated results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are thoroughly discussed in the discussion section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
20

---

Answer:
[NA]
Justification:
This is not theoretical paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We detailed each implementation steps in the main paper and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
21

---

Question: Does the paper provide open access to the data and code, with sufficient instructions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
The implementation involves models that have not been open-sourced.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (tttps://nips.oo
wttpauumiiiiiinoooooooooiioaaoie for more details. Bonllic/guides/Cddeeuuee
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ cttppscccppbbccccc/pepcepppccccc /Codeeuoeeeeeeeeeeeeeee /ttaai/g//deeeeeeeeebaaiiiiiii
onnolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We specified the training details in the supplementary, and how the user study is conducted.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We conduct user study and report win rates between our method and competitions. The preference of our method ranges between 66.9 to 77.1 percentage across seven different categories.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources Question:
For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: We stated that our method runs with single A100 GPU.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /ttaar/////eupss/eeeeeeeeee http://trrrpppcppplii/i/cc/ccc/t csGuieellee
Answer: [Yes]
Justification: There is no concern over code of ehtics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there is a special consideration due laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Included at the end of conclusion section.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
23

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not release new models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi wwwode.hco....c. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the datasct/codd/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: We included detailed descriptions, and screenshot of instructions in the appendix.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: The experiment is conducted with Amazon Turk.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
25


Reviews:
[{"review_id": "dBjWJDHJZV", "reviewer": "Reviewer_3AzP", "summary": "The paper presents a method for 3D scene generation using video diffusion models. The main contribution is to present a method that does not require a multi-view generative model trained on synthetic data. The paper identifies the use of multi-view models as a strong limitation limiting existing works to generate high-quality scenes. The method first generates a video from the text prompt, and then also generates a second video starting from a reference frame that corresponds to a time-freezed camera scan of the scene. The second video allows for a static 3D reconstruction of the reference frame, that is then used in reconstructing the motion in the original video. The proposed method is novel and achieves better results than the state of the art.", "questions": "Does Colmap work for all the dynamic videos generated, even if the motion is large? \n\nThe results are generally very impressive but it would be very helpful to the community to also show failure examples so that the remaining challenges are clear.", "limitations": "Yes.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper is very well-motivated and well-written. The key limitation in existing works is very clearly and explicitly identified. \nThe method is explained clearly. The paper does a good job explaining a complex pipeline and the supplemental provides further details. \nThe evaluations demonstrate very clear improvements over the state of the art.", "weaknesses": "The paper shows high-quality results that outperform existing work, and also present ablations. I was curious whether the identification of the use of synthetic data could also be demonstrated as an ablation, e.g., finetuning the snap video model on synthetic data and adding another SDS loss to demonstrate that it is indeed the key factor that limits high quality. \n\nThe paper in the intro mentions \"reliance on multi-view generative models\". While the video generative model used in the paper does not take camera poses as conditioning, it is in fact trained on static multi-view data. If so, I would still call such a model capable of generating multi-view images as a multi-view model. How would the overall approach perform if the snap video model was not trained on static multi-view scans? It would be good to provide more details on the training data. What percentage of the training data consists of such multi-view recordings?\n\nThe paper claims efficiency as a contribution; however, later in L225-228, it correctly points out that most of this improvement is due to the use of snap video model, and not due to any contributions of this paper. If true, the claims should be adjusted. \n\nFrom L161-163, it seems like the sufficient amount of pose variation is chosen manually? Is it not possible to automate this? If this step is manual, how is it accounted for in the reported processing times?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Ablation of finetuning with synthetic data.** This is a great suggestion to rigorously prove the benefit of using models trained with real world data. However, conducting such experiments requires significant GPU hours to reach a meaningful conclusion, which we are unable to fulfill for rebuttal. Instead, we refer to results from text-MV image models finetuned using synthetic data, e.g. MVDreamer. As shown in Fig 1 of the PDF, images generated by MVDreamer are cartoonish, even when “realistic style” is included in the input text prompt.\n\n**Importance of training with static multi-view data.** Training video models with static multi-view data is crucial not only because it supplements the training set with videos of static scenes but, more importantly, because it allows us to learn a context embedding (associated to the multi-view dataset) that serves as a tool to control the model to generate freeze-time-like video. Without this step, the model predominantly generates dynamic videos whenever nonrigid subjects are included in the text prompt, regardless how the text prompt is engineered. \n\n**Details of training data.** The full training set combines a dozen of datasets, each corresponding to different type/style of videos, and is associated with a context embedding. For the static multi-view dataset, it is mainly created from MVImageNet, and sampled based on a predefined ratio during training. The ratio is set according to the number of video hours in each dataset, which results in roughly 1% for the static multi-view dataset.\n\n**Efficiency contribution claim.** It is partially correct that one source of efficiency comes from the video model we used. However, the main source of efficiency, as stated in Ln. 58, is that we \"transform the generation problem into a reconstruction problem and reduce reliance on time-consuming score distillation sampling steps.\" Specifically, as described in Ln. 263, during each stage of the reconstruction procedure, we perform SDS only during the last 5,000 iterations out of the total 20,000 iterations. This choice significantly reduces runtime, as each SDS iteration takes over 20 times longer than simply minimizing reconstruction losses.\n\n**Is choosing amount of pose variation automatic?** This part is automated in our experiments. First, we perform auto-regressive view extension until the video reaches a maximum of 72 frames, which is often more than enough for 180+ degree coverage. Next we run Colmap with the generated video. If Colmap fails to converge, indicating the current video is significantly geometrically inconsistent, the video is automatically shortened until Colmap is able to reach a solution. We will include this in the supplementary.\n\n**Does Colmap work for all the dynamic videos generated?** We recognize the challenge of pose estimation for dynamic videos. Therefore, we purposely generate dynamic videos without camera motion. This is achieved by using text prompts “camera is static”. We will include this detail into the supplementary.\n\n**Failure case.** As stated in Ln. 303, our method may fail if there are “rapid movements, the sudden appearance and disappearance of objects, and abrupt lighting changes”. We will include failure samples in the new draft.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jhRZXJChWl", "reviewer": "Reviewer_yWZo", "summary": "This paper proposes a new pipeline called 4Real, aiming for more photorealistic text-to-4D generation than prior work. The method first generates a reference video, then learn a canonical 3D representation from a freeze-time video. Afterwards, per-frame and temporal deformation are learned to model the gap between the canonical representation and the targeted video. This paper compares with two baselines and includes both visualizations on 30 examples and user study. The method is also much faster than prior work.", "questions": "1. From the results, it does seem that the baselines generate much less realistic objects than the proposed method as the paper has claimed. However, existing text-to-3D models have proved their power in generating realistic 3D assets [20, 24, etc.], so could it because the 3D generation model used in baselines are a bit outdated or the prompts were not the best ones? Maybe the realism in the baseline results can be largely improved by a bit prompt engineering (e.g., adding keywords like \"realistic\" in the prompt) or updating their used 3D generation model?\n\n2. Does the deformation model also time-varying appearance or only time-varying geometry?\n\n3. Minor suggestions:\n\n(1) maybe adding the input prompt to Fig. 4 may help the readers better understand the output content and quality.\n\n(2) Some typos:\n\nL24: \", E\" -> \". E\"\n\nL292: missing a period.", "limitations": "The limitations seem to have been sufficiently discussed.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper has lifted the requirement of training a 3D generative model using limited synthetic data from prior work. This is important since there are much less available 3D assets than videos in the real world. This paper shows a potential way of fully unleashing the power of the abundant diverse video data in improving the text-to-4D models.\n\n2. The model takes 1.5 hour for testing, which is much faster than 10+ hours from previous methods. Although it is still very slow compared  to other text-to-X models, it's already a great improvement.\n\n3. The paper has a good presentation and is well-written. It also shows all the testing results in the supplementary.", "weaknesses": "This paper has clearly identified quite a few key limitations of prior work and aims to solve them, which is great. But I don't think all of these claims have been validated. For example:\n\n(1) this work captures the interaction between objects and environments\n\nFrom all the submitted video results, they do not seem to include much (or any) interaction between the object and the environment (e.g., relative global movement between them). For example, for the \"bear driving a car\" comparison between ours and 4Dfy, the car in \"our result\" is not moving on the lawn. The movement only comes from the bear inside the car. \n\n(2) prior work being object-centric while this work not.\n\nAlthough it's clear that the compared baselines only generate foreground object while the proposed method also creates the background environment. Since there's not much interaction between the background and foreground (as explained above), a better baseline would be generating an empty background/environment that suits the prompt/foreground and then putting the baseline-generated object properly in the environment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Interaction between objects and environments.** Our approach aims for a generalizable approach to generate realistic interaction between objects and environments, which includes natural multi-object placement, realistic environmental lighting effects, and relative motions between objects and environment. We acknowledge the ambiguity of the term “interaction” and will clarify these points. As per the reviewer’s request, we provide additional samples (see Fig. 4 in PDF) with moderate global motion between the object and the environment, such as “a storm trooper walking forward …” and “a dog running from left to right in front of a chair”. While our method currently faces challenges in reconstructing content with more rapid motion (as stated in Ln. 303), this limitation may be alleviated using concurrent advancement in 4D reconstruction method with long-term point tracking, such as MoSca [1]. \n\n**Baseline by 3D scene generation + 4D object-centric generation.** Although this is a straightforward baseline, implementing it with high quality is challenging. \n\n- First, inserting 3D objects with realistic and physically correct placement is not trivial. Common issues include floating objects, misaligned scales between objects and background, and unrealistic scene layout. \n\n- Second, generating object motions that align well with the backgrounds is difficult, such as ensuring a generated target subject sits on a sofa rather than stands on it. \n\n- Third, complex procedures are needed to relight objects to match environment lighting.\n\nThere exists several recent works either specialized in human-environment interaction [4] or generating simple 3D scene layouts [5]. Currently, we are not aware of methods able to achieve all above points automatically.  We tried our best to create a few baseline results with heavy manual efforts, as shown in Fig. 2 in the PDF. We create the 3D background by removing the object from our generated freeze-time video. We then employ 4Dfy to generate the foreground objects. Finally, we manually insert the objects into the background with the most plausible position and scale we can find. Despite these efforts, we find the inserted object appears disconnected from the background, especially when compared to 4real’s results. For example, in the first example, the fox stands on the sofa instead of lying on the sofa; in the 2nd example, the racoon appears floating above the DJ equipment, and the lighting is inconsistent with the purple/blue environment lighting; and in the last example, the racoon appears popping out of the newspaper instead of lying on it. \n\nOverall, we think that 3D scene generation + 4D object-centric generation could be a viable alternative. However, this approach requires significant and complex research and engineering efforts. At this point, directly lifting 3D from generated videos, as demonstrated in our pipeline, appears to be a more generalizable and straightforward method. This is because the video model inherently produces realistic layouts, motion, and lighting.\n\n**Could up-to-date text-3D models improve realism.** The  reason why 4Dfy and dream-in-4d is less realistic is because relying on text-MV model (i.e. MVdream) trained with synthetic data. As a result, the model is biased to generate synthetic-style objects, even when “realistic style” is added to the text prompt (see Fig.1). Some text-3D methods achieve a certain level of realism by using generative models trained with real data, either performing SDS with text-image model [2], or training image-MV model using real data [3]. In this regard, our approach can be loosely considered an updated version of these methods with an improved 3D prior.\nWe will include this discussion in the next draft.\n\n\n**Does the deformation model also time-varying appearance?** No, we only model geometry deformation. We find that incorporating time-varying appearance worsens reconstruction results by making the 4D reconstruction problem more ambiguous.\n\nWe thank the reviewer for the valuable comments, and will modify accordingly.\n\n\n**Reference:**\n\n[1] MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffolds.\n\n[2] HiFA: High-fidelity Text-to-3D Generation with Advanced Diffusion Guidance\n\n[3] CAT3D: Create Anything in 3D with Multi-View Diffusion Models\n\n[4] GenZI: Zero-Shot 3D Human-Scene Interaction Generation\n\n[5] GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splatting", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7jYqTtJzZT", "reviewer": "Reviewer_7sxF", "summary": "This paper proposes a straightforward method to distill a 4D dynamic scene from a video diffusion model. Given a reference video (only generated videos are shown in the paper, no real video), they first generate a freeze-time video as a multi-view to reconstruct a reference 3DGS. Because they don't have a good multiview image model to capture the general scene multiview prior, they can only use a video model, which might produce small deformations. Some engineering effort is paid here to accept such deformation. After the reference Gaussians is reconstructed, an SDS stage is applied to both the frozen time multiview images and the temporal deformed frames.", "questions": "The authors also have to highlight their novelty, since the method is straightforward.", "limitations": "Some limitation is discussed, but no failure cases are shown.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This is an early attempt to generate 4D scenes from video diffusion models\n- Although very straightforward, the system works for generating some 4D videos.", "weaknesses": "- Object-centric? One important constraint this paper assumes is that because they want to generate dynamic scenes, they don't use sophisticated object-centric models. However, most video examples this paper shows do not have significant background motion, so one strong baseline/alternative is to just distill a static background and put a dynamic subject (from a dynamic object-centric SDS) in the foreground. Maybe this would achieve much better results than the paper's pipeline.\n- How consistent? One question arises when you have a reference temporal video. Because there is a hallucination of the freeze-time view only considering a specific frame, however, the hallucinated freeze-time multiview images may not be consistent with what is observed in the temporal video, how does the system handle such in-consistency?\n- Failure case: when i open the static website files in the supply, there is a folder called \"good results\", which the reviewer guesses may be because of some case selection, it would be critical to also show failure case.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Baseline by 3D scene generation + 4D object-centric generation.**\nAlthough this is a straightforward baseline, implementing it with high quality is challenging. \n\n- First, inserting 3D objects with realistic and physically correct placement is not trivial. Common issues include floating objects, misaligned scales between objects and background, and unrealistic scene layout. \n\n- Second, generating object motions that align well with the backgrounds is difficult, such as ensuring a generated target subject sits on a sofa rather than stands on it. \n\n- Third, complex procedures are needed to relight objects to match environment lighting.\n\nThere exists several recent works either specialized in human-environment interaction [3] or generating simple 3D scene layout [4]. Currently, we are not aware of methods able to achieve all above points automatically.  We tried our best to create a few baseline results with heavy manual efforts, as shown in Fig. 2 in the PDF. We create the 3D background by removing the object from our generated freeze-time video. We then employ 4Dfy to generate the foreground objects. Finally, we manually insert the objects into the background with the most plausible position and scale we can find. Despite these efforts, we find the inserted object appears disconnected from the background, especially when compared to 4real’s results. For example, in the first example, the fox stands on the sofa instead of lying on the sofa; in the 2nd example, the racoon appears floating above the DJ equipment, and the lighting is inconsistent with the purple/blue environment lighting; and in the last example, the racoon appears popping out of the newspaper instead of lying on it.\n\nOverall, we think that 3D scene generation + 4D object-centric generation could be a viable alternative. However, this approach requires significant and complex research and engineering efforts. At this point, directly lifting 3D from generated videos, as demonstrated in our pipeline, appears to be a more generalizable and straightforward method. This is because the video model inherently produces realistic layouts, motion, and lighting. Furthermore, we envision that incorporating concurrent advancements in 4D reconstruction (e.g., MoSca [1]) and 4D object-centric generative priors (e.g., SV4D [2]) will further improve the quality of our pipeline.\n\n\n**Consistency between reference video and freeze-time-video.** To minimize inconsistency, we generate reference videos without camera motion. This approach reduces disoccluded regions caused by camera movement, which is a major source of potential inconsistency between the reference video and the freeze-time video. Given this setup, we find the low-res video model is able to generate mostly consistent pixels in the remaining disoccluded regions caused by object motion, partially due to any inconsistency is hard to notice in low-res video. To further improve consistency, we merge two videos into a single video when feeding it into the video upsampler, leveraging the temporal self-attention in the upsampler to improve consistency.\n\n**Failure cases.** As stated in Ln. 303, our method may fail under conditions such as rapid movements, the sudden appearance and disappearance of objects, and abrupt lighting changes. We will include failure samples in the new draft. To clarify what we mean by “good results,” we initially generated approximately one hundred results with various styles of reference videos without human filtering to understand the limits and ideal use cases of our method. After analyzing these results, we identified our limitations as mentioned above and retained the set of “good” videos—those with moderate movements, no sudden appearance or disappearance of objects, and no significant lighting changes—as example results for the intended use case of our method. The selection process can be entirely automatic via heuristic rules based on our observations. \n\n**Highlight of Novelty.** In the 3D generation field, the paradigm where a 3D reconstruction following multiview image generation is one of the mainstream approaches. Although the idea is straightforward, achieving convergence to this solution and making it work effectively is not trivial. Similarly, the proposed generation-via-reconstruction pipeline for 4D generation is non-trivial in practice. Naively combining existing video models and 4D reconstruction techniques does not yield desirable results. The major contributions and novelties of our approach are (1) a workflow to obtain plausible freeze-time videos from reference videos, given current video models of limited capability, (2) a deformation field to handle the inconsistencies in imperfect freeze-time videos, and (3) a joint temporal and multiview SDS to help reconstruct temporal deformation. With these proposed techniques along with other efforts to adopt various existing components and regularization effectively, we achieve the presented realistic 4D generation results. \n\n\n**Reference:**\n\n[1] MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffolds.\n\n[2] SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistency.\n\n[3] GenZI: Zero-Shot 3D Human-Scene Interaction Generation\n\n[4] GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splatting", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hqZPpc2vGt", "reviewer": "Reviewer_oa6s", "summary": "The paper introduces a pipeline designed for photorealistic text-to-4D scene generation, discarding the dependency on multi-view generative models and instead fully utilizing video generative models trained on diverse real-world datasets. The method begins by generating a reference video using the video generation model, then learns the canonical 3D representation of the video using a freeze-time video, delicately generated from the reference video. To handle inconsistencies in the freeze-time video, the method jointly learns a per-frame deformation to model these imperfections, then learns the temporal deformation based on the canonical representation to capture dynamic interactions in the reference video.", "questions": "1. The paper mentioned a freeze time view deformation field. Does every frame has it or it is only applied to the first frame.\n\n2. How important is the freeze time view deformation field. \n\n3.  The motion video diffusion needs to sample with the first frame, how does the video diffusion handle different sampled fps? Using it as a input? This setting potentially limits the extension of this method to longer video.", "limitations": "Multiview video can be a better prior, it is limited by training data, but will improve in the future. 4D generation might be benefited from this prior and reduces the requirement of multiple stages.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper delivers impressive visual results for 4D scene level generation. The quality is outstanding and scene level generation is more general than previous object level generation.\n\n2. The paper is well written with elaborated technical details.", "weaknesses": "1. The method seems extremely slow due to multiple steps being applied. The method hasn't provide enough latency comparisons.\n\n2. Besides user studies, I wonder if any qualitative results can be shown, for example, FVD. Or some video sequences from multiview data (in door or outdoor) and compute reconstruction loss, but applying the lifting for reference video (using one view video in the data set) and render the other view to compare with the gt novel view video.\n\n3. The model has a freeze time view deformation field, which indicates fitting static 3dgs scene from freeze-time video is unable to deliver good initial 3dgs field. How to balance the 3d consistency and plausibility from freeze time camera prior? More reasoning and analysis is desirable.\n\n4. The paper uses freeze time SDS some part and multi-view SDS somewhere else. Better to improve the consistency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Latency comparison.** In Ln. 64, we mentioned that our overall runtime is approximately 1.5 hours on a single A100 GPU, compared to over 10 hours for 4Dfy and Dream-in-4D. To provide additional latency analysis, we break down our runtime as follows: approximately 10 minutes for reference and freeze-time video generation, about 2 minutes for running COLMAP, around 20 minutes for reconstructing the canonical GS, and less than 1 hour for the remaining reconstruction steps with SDS.\n\n**Additional qualitative metrics.** We included additional metrics to quantitatively evaluate the rendered videos (see Table 1 of the PDF). We employ the XCLIP score [1], a minimal extension of the CLIP score for videos. We also run VideoScore [2], a video quality evaluation model trained using human feedback, which provides five scores assessing visual quality, temporal consistency, text-video alignment, and factual consistency. Our method significantly outperformed other methods in these metrics. However, we remain conservative about the effectiveness of these metrics since they have not been thoroughly studied for text-to-3D/4D generation.\n\n Method            | **X-CLIP $\\uparrow$** | **Visual Quality $\\uparrow$** | **Temporal Consistency$\\uparrow$** | **Dynamic Degree $\\uparrow$** | **T-V Alignment $\\uparrow$** | **Factual Consistency $\\uparrow$** \n-----------------|-----------------------|-------------------------------|------------------------------------|-------------------------------|------------------------------|------------------------------------\n **4Dfy**        | 20.03                 | 1.43                          | 1.49                               | 3.05                          | 2.26                         | 1.30                               \n **Ours**        | **24.23**       | **2.43**               | **2.17**                      | **3.15**                 | **2.91**               | **2.49**                      \n **Dream-in-4D** | 19.52                 | 1.34                          | 1.37                               | 3.02                          | 2.27                         | 1.20                               \n **Ours**        | **24.77**        | **2.41**                 | **2.15**                      | **3.14**                 | **2.89**                | **2.46**                      \n **AYG**         | 19.87                 | **2.49**                 | 2.09                               | 3.15                          | 2.80                         | 2.47                               \n **Ours**        | **23.09**        | 2.44                          | **2.16**                      | **3.16**                 | **2.90**               | **2.50**                      \n\n\nAdditionally, we excluded the FVD metric, as computing it with a limited number of rendered videos is statistically not meaningful. Evaluating with real-world multi-view videos is also not easily applicable to our pipeline. This is because we made specific design choices, such as requiring the reference video to have no camera motion to avoid inconsistency between the reference video and the freeze-time video. This assumption does not hold for casually captured dynamic videos.\n\n\n **Analysis of freeze time view deformation field.** The reason for introducing view-dependent deformation (referred as per-view deformation in the paper) is to treat inconsistency in the generated freeze-time videos as geometric deformation. Without the view-dependent deformation, the canonical GS tends to underfit the freeze-time video, leading to blurrier regions, particularly in the background, where the video model often produces geometrically incorrect results.\n\nHowever, while view-dependent deformation can improve underfitting, it can also lead to overfitting to the reference video, resulting in noisy reconstruction. To mitigate this, we employ a regularization loss of the magnitude of the deformation (as mentioned in lines 180-186)  to prevent overfitting. Through empirical testing with a few samples, a weighting of 0.01 was found to be effective and is used consistently throughout the experiments.\n\nAdditional visual comparisons are provided in Figure 5 of the PDF to support this analysis. These comparisons demonstrate the effects of removing either the per-view deformation or the small motion regularization.\n\n**Does freeze time view deformation field apply to other temporal frames?** No, it is only applied to fit the frames from the freeze-time video. It is not used for any other frames.\n\n\n**How does the video model handle different sampled fps?** The video model we used can take FPS as input. Additionally, as described in Ln. 142, it is a frame-conditioned model capable of performing either autoregressive generation or frame interpolation. Therefore, theoretically, our method can handle longer videos in the cost of longer processing time in the following reconstruction stage.   \n\n**Multiview video prior.**  Yes, we also strongly believe that developing a generalizable multiview video diffusion model is one of the most promising directions for 4D generation. We briefly mentioned potential implementations, such as cross-frame attention, in Ln. 308. In the new draft, we will include a discussion of more concurrent efforts in multiview video generation. At this point, however, our method serves as a pioneering effort in demonstrating the possibility of generating realistic 4D scenes.\n\n**References:**\n\n[1]  Expanding Language-Image Pretrained Models for General Video Recognition. \n\n[2] VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1VMgPwYbB3", "reviewer": "Reviewer_vJ8m", "summary": "4Real presents a novel framework for dynamic 3D scene generation. Given an input video, the framework selects a frame and prompts a text-to-video diffusion model to generate a \"freeze-time video\" that reconstructs the canonical 3D scene. Then, video-SDS-based optimization is used to optimize a dynamic 3DGS with reference to the input video. Compared to existing approaches that focus more on object-centric generation, the new method is able to generate more complex 4D scenes including multiple objects.", "questions": "Please kindly address the weaknesses above.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. 4Real generates photo-realistic 180-degree 4D scenes for the first time. In comparison, previous 4D generation works are mainly object-centric, restricted by the object-centric image models that they rely on. 4Real only relies on native text-to-video diffusion models, thus addressing the bias. The approach is more favourable than 4Dfy and Dream-in-4D as suggested by a user study.\n2. Although the work mainly follows the previous video-SDS-based 4D generation frameworks, it proposes several effective techniques to enable modeling more complex scenes that include multiple objects. A key design is to generate the canonical 3D through freeze-time video and video extension. An additional dynamic 3DGS fitting and a heuristic loss (small motion regularization) are proposed to offset the remained motions in the freeze-time videos, which I find reasonable and novel.\n3. The paper also provides empirical values to the video-SDS-based 4D generation frameworks. It reduces the video-sds computational cost dramatically by leveraging low-resolution pixel diffusion models.", "weaknesses": "1. The description of context conditioning (L149) is not clear to me. \"One dataset consists of static real-world objects with circular camera trajectories\", is this dataset similar to [CO3D](https://ai.meta.com/datasets/co3d-dataset/), does it contain background, and how critical is the context embedding? Including this information can be important to reproduce this work.\n2. The evaluation did not compare with AYG [28], whose results could also be easily accessed. AYG optimizes a dynamic 3DGS while the compared 4Dfy and Dream-in-4D optimize a dynamic NeRF, so I would say AYG is closer to this work. Although similar conclusions could be drawn for both AYG and 4Dfy/Dream-in-4D as they are all object-centric, the argument in L271 that claims \"These methods are known to achieve some of the best visual quality compared to AYG and ...\" is neither well-grounded nor convincing. L270 is also wrong to claim that Dream-in-4D is based on D-3DGS.\n3. Generating a freeze-time video for the static 3D generation makes a lot of sense, but I wonder if it is possible that the generated freeze-time video could conflict with the reference video. Moreover, the canonical 3D generation stage seems to be solving a standalone image/text-to-scene task, it would be interesting to know if recent 3D scene generation works like RealmDreamer[A] and CAT3D[B] could be a drop-in replacement.\n\nMinor:\n- Editing is mentioned in the contributions, but it has not been discussed in any other places in the paper.\n\n[A] Shriram et al., RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusion\n\n[B] Gao et al., CAT3D: Create Anything in 3D with Multi-View Diffusion Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Context embedding.** First, the dataset for “static real-world objects with circular camera trajectories”, consists of data similar to the MV-ImagNet, with real-world backgrounds. Second, the use of context embedding is critical, as it controls the video model to generate freeze-time-like video. Without this step, the model predominantly generates dynamic videos whenever nonrigid subjects are included in the text prompt, regardless how the text prompt is engineered. The usage of context embedding can definitely be abandoned in the future when the video models are equipped with better camera-control and better consistency to input prompts. \n\n\n**Compare with AYG and correct description of Dream-in-4D.** We thank the reviewer for pointing out our mistake in wrongly describing Dream-in-4D as a 3D GS-based method. We will correct this error accordingly. Additionally, we have included a comparison against AYG, as shown in Figure 3 and Table 1 in the PDF. We find that our overall assessment of the advantages of our method over prior object-centric methods remains valid.\n\n**Consistency between reference video and freeze-time-video.** To minimize inconsistency, we generate reference videos without camera motion. This approach reduces disoccluded regions caused by camera movement, which is a major source of potential inconsistency between the reference video and the freeze-time video. Given this setup, we find the low-res video model is able to generate mostly consistent pixels in the remaining disoccluded regions caused by object motion, partially due to any inconsistency is hard to notice in low-res video. To further improve consistency, we merge two videos into a single video when feeding it into the video upsampler, leveraging the temporal self-attention in the upsampler to improve consistency.\n\n**Applying recent 3D scene generation works.** Yes, we believe recent image-mv models trained for 3d scene generation can be a drop-in replacement for the part of generating freeze-time video. Unfortunately, none of these concurrent approaches (i.e., RealmDreamer and CAT3D) are open-sourced yet, so it is not clear how they perform compared to the video model we employed. We will include a discussion of these methods in the revised version.\n\n**Video editing.** Thanks for the reminder. We mentioned the flexibility in selecting and editing videos in our contributions based on our generation-via-reconstruction pipeline that allows users to generate, select, and edit the reference video they want. We will include our results of applying simple video editing techniques, such as face-swapping or attributes manipulation, in the revised draft of the supplementary material.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank the reviewers for the positive feedback and insightful comments. We appreciate all the suggestions and will revise the paper accordingly.  

To address major concerns from the reviewers, we include the following additional results in the PDF:

- **Additional baseline results from combining 3D scene generation and 4D object-centric generation (@yWZo, 7sxF)**: We found that this seemingly straightforward baseline is actually non-trivial to produce high-quality results with realistic object placement, object motion, and lighting effects, even with significant human intervention such as setting the placement position and object scale.
- **Additional results on text-mv models (@3AzP, yWZo)**: Text-MV model trained with synthetic data is biased to generate synthetic-style images, even when ”realistic style” is added to the text prompt.
- **Examples showing some global relative motion between the object and the background (@yWZo)**
- **Comparison against AYG (@vJ8m)**
- **Quantitative evaluation using XCLIP and VideoScore (@oa6s)**: XCLIP is a minimal extension of the CLIP score for videos. VideoScore is a video quality evaluation model trained using human feedback, which provides five scores assessing visual quality, temporal consistency, text-video alignment, and factual consistency.
- **Additional visual ablation of deformation (@oa6s)**: We add visual analysis of per-view deformation and the effect of using deformation regularization to balance between 3D consistency and plausibility from freeze time video prior.

We will elaborate on and address every question and suggestion in the following individual response.

Meta-review: The paper proposed a text-to-4D generation framework, which first generates a canonical 3DGS from a generated freeze-time video and further optimizes with a video SDS-based loss to incorporate dynamics. The proposed pipeline enables generating more complex 4D scenes over video-SDS baselines (e.g. 4Dfy), overcomes the limitation on limited 3D/4D data, and provides promising results on 4D generation. 

After the rebuttal, all the reviewers are positive about the paper. The AC agrees with the reviewers' suggestions on accepting this paper. As promised during the rebuttal, please add the additional comparisons with baselines, as well as additional results provided in the rebuttal  to support the claims mentioned in the paper.
Decision: Accept (poster)