4Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Models

Authors: heng  yu
chaoyang  wang
peiye  zhuang
willi  menapace
aliaksandr  siarohin
junli  cao
lászló a jeni
sergey  tulyakov
hsin-ying  lee
canonical 3dgs deformation
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 07:27:47

Abstract: Existing dynamic scene generation methods mostly rely on distilling knowledge from pre-trained 3D generative models, which are typically fine-tuned on synthetic object datasets.
As a result, the generated scenes are often object-centric and lack photorealism. 
To address these limitations, we introduce a novel pipeline designed for photorealistic text-to-4D scene generation, discarding the dependency on multi-view generative models and instead fully utilizing video generative models trained on diverse real-world datasets. 
Our method begins by generating a reference video using the video generation model.
We then learn the canonical 3D representation of the video using a freeze-time video, delicately generated from the reference video.
To handle inconsistencies in the freeze-time video, we jointly learn a per-frame deformation to model these imperfections.
We then learn the temporal deformation based on the canonical representation to capture dynamic interactions in the reference video. 
The pipeline facilitates the generation of dynamic scenes with enhanced photorealism and structural integrity, viewable from multiple perspectives, thereby setting a new standard in 4D scene generation.

Main paper: 
 introduction: 
 As industries ranging from film production to virtual reality seek increasingly immersive and interactive experiences, the ability to generate dynamic 3D scenes over time-essentially, 4D environments-promises to revolutionize how we interact with digital content. Recently, significant advances in image and video generation have been driven by large-scale text-image and text-video datasets, along with the development of diffusion models. Furthermore, image diffusion models have been adapted into 3D-aware multi-view generative models through fine-tuning with limited 3D data, serving as foundational priors for 3D and 4D generation.In this work, we focus on photorealistic text-to-4D scene generation, Existing 4D generation pipelines, due to the lack of 4D data, typically employ image, multi-view, and video generation models as priors to synthesize 4D samples. However, the multi-view models, which provide critical 3D information, are fine-tuned on static and synthetic 3D assets. As a result, current generated 4D results are predominantly object-centric, lacking photorealism, and limited in their ability to capture complex interactions between objects and environments.In response, we propose 4Real, a novel pipeline designed for photorealistic dynamic scenes with dynamic objects within the environment. We discard the reliance on multi-view generative models and exploit video generative models trained on large-scale real-world videos, covering more diverse and general appearance, shape, motion, and the interaction between objects and environments. Compared to existing methods purely relying on score distillation sampling, the proposed pipeline provides more flexible use cases, more diverse results, and requires fewer computations.The proposed method adopts deformable 3D Gaussian Splats (D-3DGS) as the representation for dynamic scenes. The pipeline unfolds in three steps. First, we begin by creating a reference video featuring a dynamic scene with a pre-trained text-to-video diffusion model. Next, we reconstruct a canonical 3D representation from a selected frame of the reference video. To achieve this, we generate a 'freeze-time' video with camera motion and minimal object movement by applying dataset context embedding and prompt engineering to the video diffusion model. However, practically, the generated video still may contain object motion and is prone to be geometrically inconsistent despite looking temporally smooth. We propose to recognize the inconsistencies as per-frame deformation with respect to the canonical representation and learn these deformations simultaneously with the canonical representation. Finally, we reconstruct temporal deformation from the reference video with the learned canonical representation. We choose a video score distillation sampling (SDS) strategy that renders two types of videos: a fixed-viewpoint video with a static camera, and a freeze-time video with a fixed time step. The fixed-viewpoint video helps learn temporally consistent motions, while the freeze-time video learns multi-view geometrical consistency.4Real achieves text-driven dynamic scene generation with a near-photorealistic appearance and realistic 3D motions. The generated scenes are viewable from different viewing angles and over time, as shown in Figure 1 . Our contributions are summarized as follows:• We propose 4Real, the first photorealistic text-to-4D scene generation pipeline. Without reliance on biased multi-view image generation models trained with specialized datasets, the proposed pipeline can generate more diverse and near-photorealistic results with dynamic objects within realistic environments.• We leverage text-to-video diffusion to generate target reference videos and auxiliary freezetime videos, thereby transforming the generation problem into a reconstruction problem and reducing reliance on time-consuming score distillation sampling steps. We learn from the freeze-time videos canonical 3DGS along with per-frame deformation modeling video inconsistency, and learn from the reference videos temporal deformation.• The proposed pipeline provides users flexibility in selecting and editing videos that they want to lift to 4D, and can generate high-quality samples in a more reasonable computation budget, taking 1.5 hours on an A100 GPU compared to 10+ hours with competing methods [2, 72] .(a) Warm-up Stage:• Train the network for 3,000 iterations without any per-frame deformations to establish a baseline for the canonical space. This stage follows the same method as described in [23] . (b) Learning Deformations and GS Optimization:• Learn deformations for each frame and optimize the Gaussians over the course of 20,000 iterations. • Implement GS growth (including splitting, cloning, and pruning) using the strategy described in [23] during the first 15,000 iterations. (c) Multi-view SDS:• Enable multi-view SDS in this stage from 15,000 iterations to help learn a good canonical space.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: We claim to propose a pipeline that facilitates the generation of dynamic scenes with enhanced photorealism and structural integrity, viewable from multiple perspectives. Such claim is empirically verified with our generated results.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related works: 
 Text-to-video generation. Our pipeline leveraged the advancements in video generation models, which have been spurred by the recent success of text-to-image diffusion models. Some approaches Given the input text prompt, we first generate a reference video using text-to-video diffusion model. The reference video will be our target to perform 4D reconstruction. In order to obtain canonical Gaussian Splats, we generate a freezetime video by performing frame-conditioned video generation with prompt engineering and context embedding. We further perform auto-regressive generation to expand view angle coverage.extend these capabilities by directly fine-tuning pre-trained image diffusion models [3, 4] or by integrating motion modules while maintaining fixed image models [17, 34, 71] . Others utilize both image and video data using 3D-Uet [20] and cascaded temporal and spatial upsampler [19, 48] .More recently, some efforts [5, 35] propose to replace the Unet architectures with transformer-based architectures [9, 41] for improved quality, scalability, and training efficiency.Object-centric 3D and 4D generation. The challenges of generating 3D and 4D data mostly lie in the scarcity of available data, due to the costly nature of data capture. Conventional 3D generative models [1, 10-12, 22, 28, 38, 50, 57] that train on limited 3D data in various representation fall short in quality and diversity, while 2D diffusion models have shown incredible generalizability using large-scale data. To bridge the gap, Score Distillation Sampling (SDS) [42] and its variants [8, 27, 56, 58, 73] have been developed to utilize pre-trained text-to-image models as priors to generate 3D objects by optimizing parametric spaces for 3D representations. Recently, multiview images have emerged as a popular representation in 3D generation. Multiview image diffusion models [30] [31] [32] 47] are fine-uned from image diffusion models using synthetic object dataset [13] . To generate 3D objects, the multiview image diffusion models are used as approximate 3D priors [44, 62] , or as providing intermediate medium followed by 3D reconstruction methods [21, 25, 32, 51, 59, 65, 74] . Meanwhile, video diffusion models have also been explored as priors for some early 4D generation attempts [49] or as data generators to provide multiview supervision for 3D model [52] . More recently, image, multiview, and video diffusion models are jointly used as priors for 4D generation [2, 29, 45, 68, 72] . However, they typically produce results biased toward object-centric and non-photorealistic outputs as the 3D priors are obtained from synthetic object datasets.4D Reconstruction. Neural Radiance Fields (NeRFs) [36] has revolutionized 3D scene reconstruction and novel view synthesis. The breakthroughs have been expanded to dynamic scenes. There are two major branches of methods for dynamic NeRFs: one views them as a 4D representation by extending the radiance fields with a time dimension [14, 16, 26, 53, 55, 64] , while the other represents a dynamic scene by a canonical space coupled with a deformation field that maps local observation to this space [15, 39, 40, 43, 54, 66] . Recently, 3D Gaussian Splatting (3DGS) [23] has been introduced as a promising representation that facilitates fast neural rendering using explicit Gaussian particles. 3DGS has then been applied to model dynamic scenes with the similar idea of building a deformation field [33, 63, 67, 69] . We adopt deformable 3D Gassian Splats (D-3DGS) as our dynamic scene representation.(a) Initial Motion Fitting:• Fit the motion over the next 10,000 iterations using only reference views at different timesteps. Note: Cease GS optimization to focus solely on learning motion deformations. (b) GS Growth for Motion Compensation:• Resume GS growth using the same strategy for an additional 5,000 iterations to address missing points in the canonical space caused by motion. Note: Fix the learned canonical space from the previous step, and perform GS growth by focusing solely on the newly added Gaussians. (c) Simultaneous Fine-Tuning:• Fine-tune both the GS and the motion deformation network simultaneously using all training frames consisting of different camera views and timesteps. (d) Joint Temporal and Multi-view SDS:• Enable temporal and multi-view SDS jointly in this stage when doing fine-tuning from 35,000 iterations and last for 5,000 iterations to help learn a consistent motion.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Limitations are thoroughly discussed in the discussion section.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community.Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 Given input text prompts, we aim to generate photorealistic dynamic scenes including animated objects and detailed backgrounds that exhibit plausible relative scales and realistic interactions with moving objects. We adopt deformable 3D Gaussian Splats (D-3DGS) as our representation. First, we utilize a text-to-video diffusion model to create a reference video with dynamic scenes. Next, a frame from this reference video serves as the conditional input for the video diffusion model to produce a freeze-time video featuring circular camera motion and minimal object movement. Subsequently, we reconstruct the canonical 3D representation from the freeze-time video. Finally, we reconstruct temporal deformations to align with the object motions in the reference video.In the following sections, we will first discuss the 4D Gaussian splats representation for dynamic scenes in Section 3.1, the generation of freeze-time videos in Section 3.2, the reconstruction of canonical 3DGS in Section 3.3 and the temporal reconstruction in Section 3.4.We employ Deformable 3D Gaussian Splats (D-3DGS) to model dynamic scenes, a popular choice for dynamic novel view synthesis tasks due to its low rendering computation cost, state-of-the-art rendering fidelity, and compatibility with existing graphics rendering pipelines [33, 63, 67, 69] . D-3DGS consists of 3D Gaussian splats that represent the static 3D scene at a canonical frame and a deformation field that models dynamic motions.Canonical 3D Gaussian splats. 3D Gaussian Splats [23] consist of a set of 3D Gaussian points, each associated with an opacity value α and RGB color c ∈ R 3 . The position and shape of each 3D Gaussian point are represented by a Gaussian mean position x ∈ R 3 and a covariance matrix, which is factorized into orientation, represented by a quaternion q ∈ R 4 , and scaling, represented by s ∈ R 3 . 3DGS are rendered by projecting Gaussian points onto the image plane and aggregating pixel values using NeRF-like volumetric rendering equations.Deformation field. To represent motion in 4D scenes, we employ a deformation field w t-deform that represents the offset of 3DGS from the canonical frame to the frame at time t:EQUATIONThe position and orientation of 3DGS at time t are simply obtained by x t = x + ∆x t and q t = q + ∆q t . We implement w using a multi-layer perceptron (MLP), which serves as a general representation for arbitrary deformations. It is worth noting that this can be substituted by more specialized representations, such as linear blend skinning, which is more suitable for articulated motions. In this work, we prefer MLPs due to their simplicity and generality.Given a text prompt, we generate a reference video using video diffusion models. First, we would like to obtain the canonical 3D Gaussian splats for this reference video by creating a freeze-time video conditioned on a selected frame. This process requires a video generation model that can: 1) perform image-to-video generation, ensuring the output video frames remain consistent with the input image, and 2) generate videos with substantial camera motion while maintaining a relatively static scene.We evaluated several recent video models accessible to the authors, including SVD [3] , SV3D [52] , VideoCrafter [7] , and Snap Video Model [35] . SVD and VideoCrafter support image-to-video generation but have limited camera movement and introduce random motions in non-rigid objects. SV3D creates object-centric 360°videos with a white background, unsuitable for the scene-level generation. The Snap Video Model aligns most closely with our requirements, offering the required capabilities for our application.Frames-conditioned pixel-space video diffusion. The Snap Video Model is a text-to-video diffusion model that diffuses pixel values instead of latents, resulting in less motion blur and fewer gridding effects, which are more common in latent diffusion models. It also tends to produce larger camera and object motions [35] . We use a variant trained to generate videos from both text prompts and an arbitrary set of video frame inputs. This feature enhances flexibility for autoregressive generation, particularly when creating freeze-time videos with extensive view coverage. Figure 3 : Reconstructing Deformable 3DGS. First, we reconstruct the canonical 3DGS using the freeze-time video. As the generated freeze-time video may still contain object motions and geometric inconsistencies, we propose to model these imperfections as per-frame deformations that are learned jointly with the canonical 3DGS. Then, we reconstruct the temporal deformation from the reference video with the learned canonical 3DGS. In addition to the reconstruction loss and motion regularization, we propose a video SDS strategy. The video SDS strategy includes a multi-view SDS, using videos of freeze time and sampled camera trajectory, and a temporal SDS, using videos of fixed camera and sampled time steps. denotes learnable parameters. minimal object movements when its context embedding is used. The success rate can be further improved by adding phrases such as "Camera circulating around" and "time elapse" to the beginning and end of the input prompt, while omitting action verbs. Despite this, limited by the current capability of the Snap Video Model, multi-view inconsistency and moderate object motion may still occur, which our reconstruction steps in Section 3.3 address.Extending view coverage. The raw output from the Snap Video Model is a sequence of 16 frames, resulting in limited view coverage. To extend coverage, we perform iterative autoregressive generation. At each step, we condition on the 8 previously generated frames, guiding the model to generate 8 new frames that follow the camera trajectory of the prior frames. We extend the view coverage in both forward and backward directions until it reaches a sufficient amount, such as 180 • . Covering 360 • is currently challenging due to background clutter and the model's limitation in generating loop videos.Constrained by the capability of current video models, generated freeze-time videos may still contain multi-view inconsistencies, preventing direct reconstruction of high-quality static 3DGS. These inconsistencies can be: 1) temporally consistent but geometrically incorrect 2D video, and 2) videos with object motions. To address these imperfections, we propose the following treatments.Representing multi-view inconsistency as deformation. We treat multi-view inconsistency as per-frame deformation with respect to the canonical 3DGS. Specifically, we introduce another deformation field w multi-view (x, k) = (∆x k , ∆q k ) similarly defined as in Eq. ( 1), which outputs changes in position and orientation of a Gaussian point at the canonical position x with respect to its new position and orientation at the k-th frame. We choose the canonical frame to be the one from the reference video that is used as the conditional input to generate the freeze-time video. This implies zero-valued outputs for the deformation field at the canonical time frame.We jointly optimize the deformation field w multi-view and the parameters for the 3DGS by minimizing a combination of the following losses.Image reconstruction loss computes the difference between images I GS rendered by the deformed 3DGS and the frames I k from the freeze-time video.EQUATIONwhere k denotes the frame index, P k represents the camera projection matrix which is initialized using Colmap [46] , then jointly finetuned with the 3DGS, and with some abuse of notations, (x + ∆x k , q + ∆q k , s, α, c) denotes parameters for the deformed 3DGS.Small motion loss. Minimizing L recon alone is insufficient to obtain plausible 3DGS, since there can be infinitely many pairs of deformation fields and 3DGS that produce identical image renderings. To address this under-constrained problem, we introduce the assumption that the deformations should be as small as possible. Specifically:EQUATIONScore distillation sampling (SDS) loss. Hand-crafted heuristics like L small mo. may be insufficient for plausible canonical 3DGS, especially with significant multi-view inconsistencies in the freeze-time videos. For improved regularization, in the later stage of optimization, we employ score distillation sampling. This method ensures that the rendering of canonical 3DGS on randomly sampled camera trajectories aligns with the distribution dictated by the video diffusion model.During each optimization step, we sample a camera trajectory by first randomly selecting a camera pose P k from the camera trajectories of the freeze-time video and then randomly perturbing it to create P. The trajectory { Pk=1,...,16 } is uniformly interpolated between the canonical camera pose and P. We then render the canonical 3DGS using this sampled trajectory to generate video frames:EQUATIONTo perform score distillation sampling, we add Gaussian noise to the rendered images I k can-GS and then use the video diffusion model to produce one-step denoised images. Since the sampled camera trajectory originates from the canonical camera pose, the first frame I 0 can-GS of the rendered videos is identical to the corresponding frame I c from the freeze-time video, allowing us to substitute I 0 can-GS with I c , which can then serve as the conditional frame for the diffusion model. More specifically:EQUATIONwhere ϵ's represent the noise applied to the video frames, σ denotes the noise level for the current diffusion step, and Îk indicates the x 0 prediction for each frame. Using I c as the image condition input is crucial because it allows the diffusion model to provide more precise supervisions, and sharper details that are aligned with the reference video.Then SDS is implemented by minimizing the sum-of-square difference between the rendered video frames I k can-GS and the denoised frames Îk , as follows,EQUATION)In this context, we use ⌈•⌉ to indicate that the variables are detached from the automatic differentiation computation graph. It is straightforward to show that the gradient of L SDS is proportional to the original gradient formulation by Poole et al. [42] .Sampling with low-res pixel-space diffusion model. The video diffusion model we employ is a two-stage pixel-space model. It first generates a video at a low resolution of 36×64, which is then upsampled to a resolution of 288×512. For SDS, we employ only the first-stage low-resolution model, as it is significantly less computationally demanding and not straightforward to apply both stages simultaneously. Fortunately, utilizing just the low-resolution SDS is adequate to yield high-quality results. This effectiveness stems from our method relying more on reconstruction loss to enhance fine details, rather than depending primarily on SDS.To compute the SDS loss, we first downsample the rendered video frames to low resolution. Before the downsampling, frames are randomly shifted and cropped to mitigate aliasing effects.Remarks on computational efficiency. Previous methods employing SDS are typically computationally intensive, making them less practical for real-world applications. For instance, executing one step of SDS training using single precision on A100-80G GPUs with SVD can take ≈2 seconds, not including the time required for rendering. Moreover, GPU memory often becomes overloaded with SVD, which cannot compute gradients for more than four frames simultaneously due to substantial memory demands for backpropagation through the image encoder.In contrast, our method implements SDS using the low-resolution, pixel-space Snap Video Model. This model leverages a compressed transformer architecture [9] , which is significantly faster, requiring ≈200 ms per step. It is also more memory-efficient, allowing for gradient computation across all 16 frames in a single batch. This efficiency makes our approach more viable for practical applications.After obtaining canonical 3DGS representation, we proceed to generate temporal motion by fitting the deformation field (as detailed in Eq. ( 1)) to align with the reference video.Image alignment loss computes the similarity between the rendered frames I t GS and the reference video frames I t , combining pixel-wise intensity loss and structural similarity index measure (SSIM) loss [60] , i.e., L align = t ∥I t GS -I t ∥ + SSIM(I t GS , I t ). Deformation regularization. To ensure that the deformation field learns plausible motions, we incorporate losses that enforce both spatial and temporal smoothness in the motion trajectories of each Gaussian splat, following CoGS [69] and Dynamic 3DGS [33] . These as-rigid-as-possible losses encourage minimal variation in both translational and rotational motions of Gaussian splats within their spatial and temporal neighborhoods. We refer readers to the respective papers for detailed treatment. Implementation details specific to our adaptation are included in the supplementary.Joint temporal and multi-view SDS. Relying solely on hand-crafted regularization often leads to noticeable artifacts in novel view rendering, especially when the camera position deviates significantly from that in the reference video. Therefore, in the later stages of optimization, we employ SDS to ensure that the rendered video frames adhere to the distributions of real videos. For evaluating the SDS loss, we render two types of videos:1. Fixed-Viewpoint Videos: The camera remains static while we sample time steps to deform the 3D Gaussian Splats. To ensure that the first frame can be used as a conditional input for the video diffusion model, the sampled time steps always start from 0. This initial frame is rendered using the canonical 3D Gaussian Splats without any deformation.2. Freeze-Time Videos: With a fixed time step, we sample a continuous sequence of camera poses starting from the reference video's camera pose. This setup allows us to directly use the corresponding frame from the reference video as the conditional input to the video diffusion model.Applying SDS loss to both fixed-viewpoint and freeze-time videos is crucial. The former regularizes D-3DGS for temporally consistent motions, and the latter ensures multi-view consistency, aligning D-3DGS motions with the reference video and avoiding multi-perspective optical illusions.Improving details by re-upsampling We optionally perform an additional step to enhance the rendering details further. We found that downsampling and then re-upsampling the videos rendered by Gaussian splatting results in fewer artifacts and higher fidelity. Therefore, we render a video with continuously changing viewpoints and time, and then re-upsample it using the upsampler of the video diffusion model. The resulting video serves as a reconstruction target to fine-tune the Gaussian splats, thereby improving the overall quality.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: The implementation involves models that have not been open-sourced.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiment: 
 We evaluated our method using a set of 30 text prompts sourced from related works [2, 35, 72] . Please refer to the included webpage for the complete results. As illustrated in Figure 4 , our method successfully generates scenes featuring complex illumination and semi-transparent objects such as water. Additionally, it demonstrates flexibility in creating diverse multi-object scenes. Implementation details. The optimization consists of two stages: first, reconstructing the canonical 3DGS, and then learning the temporal deformation. At each stage, we perform SDS only during the last 5k iterations, and we anneal the diffusion time steps similarly to the scheduler used in Hifa [73] .Throughout our experiments, we run 20k iterations per stage, totaling 1.5 hours on a single A100 GPU. This duration is significantly shorter than that required by recent state-of-the-art methods, which typically require over 10 hours [2, 72] . Details are included in the Supplementary.Compare to object-centric text-4D generation. To the best of our knowledge, there are currently no existing text-to-4D scene generation methods, so we choose to evaluate our method against recent state-of-the-art object-centric text-to-4D generation methods. Specifically, we conduct a user study against 4Dfy [2] and Dream-in-4D [72] , which are based on NeRF representation. These methods are known to achieve some of the best visual quality compared to other methods [29, 45, 49] using the same representations.The user study was conducted using Amazon Turk, involving 30 evaluators per video pair. In each session, evaluators were presented with two anonymized videos. Each video depicted a dynamic object or scene, with the camera moving along a circular trajectory and stopping at four fixed poses to highlight object motions. We obtained 16 videos for 4Dfy and 14 videos for Dream-in-4D from their respective project web pages. Evaluators were tasked with selecting their preferences based on seven criteria: motion realism, foreground/background photo-realism, 3D shape realism, general realism, significance of motion, and video-text alignment. As shown in Figure 6 , our method outperformed the competition in every category. Sample frames are displayed in Figure 5 , and all videos used in the study are available in the supplementary materials. In addition to existing methods, we also implement and compare with a straightforward baseline: combining 3D scene generation with 4D object-centric generation. Although straightforward, implementation with high quality is challenging due to: (1) Inserting 3D objects with realistic and physically correct placement is not trivial. Common issues include floating objects, misaligned scales between objects and background, and unrealistic scene layout. (2) Generating object motions that align well with the backgrounds is difficult, such as ensuring a generated target subject sits on a sofa rather than stands on it. (3) Complex procedures are needed to relight objects to match environment lighting. We tried our best to create a few baseline results with heavy manual efforts, as shown in Figure 7 . We create the 3D background by removing the object from our generated freeze-time video. We then employ 4Dfy [2] to generate the foreground objects. Finally, we manually insert the objects into the background with the most plausible position and scale we can find. Despite these efforts, we find the inserted object appears disconnected from the background, especially when compared to our results.On top of user study, we also provide quantitative results in Table 1 using employ the X-CLIP score [37] , a minimal extension of the CLIP score for videos. We also run VideoScore [18] , a video quality evaluation model trained using human feedback, which provides five scores assessing visual quality, temporal consistency, text-video alignment, and factual consistency. Our method significantly outperformed other methods in these metrics. However, we remain conservative about the effectiveness of these metrics since they have not been thoroughly studied and evaluated yet.Ablation studies. We diagnose our system by removing individual components from the pipeline.Per-frame deformation is crucial for reconstructing high-quality canonical 3D representations from noisy freeze-time videos. As illustrated in Figure 8 , the absence of per-frame deformation results in significant artifacts, such as on the faces of the panda and the cat, and in blurrier areas like the boundaries of doors in a kitchen.Small motion regularization is employed in our pipeline. While view-dependent deformation can improve underfitting, it can also lead to overfitting to the reference video, resulting in noisy reconstruction. We thus adopt a regularization loss of the magnitude of the deformation to mitigate the issues. As shown in Figure 9 . Multi-view SDS aids in noise reduction and enhances shape realism. For example, as illustrated in Figure 8 , it improves the depiction of the human head in the background, the upper boundary of the sofa, and the limbs of the goat.Freeze-time video: Removing the step of generating freeze-time videos and the corresponding reconstruction loss L recon will result in systematic failure. This is partly because we apply SDS only in the later stages of training, which is insufficient to correct significant reconstruction errors.Joint multi-view and temporal SDS helps improve the sharpness of the final result during the reconstruction of temporal deformation. It also helps prevent erroneous artifacts, such as the floating spot on the edge of the cat's ear and the dark shadows above the cat's back, as shown in Figure 8 .We have included additional samples from the ablation study in the supplementary web page. 
discussion and conclusion: 
 We propose 4Real, which, to our knowledge, is the first method capable of generating nearphotorealistic 4D scenes from text inputs. However, our method has limitations: 1) It inherits limitations from the underlying video generation model, such as video resolution, blurriness or artifacts during fast motion, particularly with thin or small objects, and occasional text-video misalignment. 2) Reconstructing from videos with dynamic content is challenging, and our method may fail due to inaccuracies in camera pose estimation, rapid movements, the sudden appearance and disappearance of objects, and abrupt lighting changes. 3) Our method does not produce high-quality geometry such as meshes, due to the limitations of using 3DGS. 4) It still takes over an hour to generate a 2-second 4D scene.For future works, we envision the availability of stronger video generation models with more accurate camera pose and object motion control [6, 61] , incorporating cross-frame attention [70] when generating freeze-time videos and utilizing feedforward 3D reconstruction [51] would help address these limitations. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We specified the training details in the supplementary, and how the user study is conducted.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We conduct user study and report win rates between our method and competitions. The preference of our method ranges between 66.9 to 77.1 percentage across seven different categories.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: Included at the end of conclusion section. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper does not release new models. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [NA] Justification: The paper does not use existing assets. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: We employ a masked variant of the Snap Video model that is trained to generate videos using both text and a set of known frames as conditioning. Given a set of known frames at arbitrary positions in the input sequence, the masked Snap Video model produces an output video whose frames at those positions match the known frames. This mechanism allows the model to perform various tasks such as image animation, frame interpolation and temporal extension of videos by providing known frames in the corresponding arrangements.Tha masked model variant is obtained by extending the original Snap Video model's input channels by 3 to accomodate the conditioning frames input, and by training the model for an additional 200k steps using a masked video modeling objective, while keeping the other training parameters unchanged.In particular, we adopt a set of masking strategies which define the set of known frames provided as conditioning during training, each of which is randomly applied to an input batch element with probability m:• Unconditional generation m = 0.6. Mask the whole input video. This setting corresponds to the original Snap Video setup, where no known frame is present. • Bernoulli m = 0.3. Mask each frames of the input video with probability (1 -1/16), leaving on average a single frame unmasked. • Frame interpolation m = 0.075. Mask all video frames apart from a set of frames sampled at regular time intervals. • Video extension m = 0.075. Mask the last N video frames.Our motion deformation network predicts position offsets (∆x), rotation offsets (∆q), and scale offsets (∆s) for each time step. To prevent unnecessary movements (e.g., in the background) and to ensure movement consistency, we apply an L 1 norm regularization as follows:EQUATIONWe introduce a difference loss, denoted as L diff , to ensure that the movement or trajectory of each Gaussian is consistent and smooth over time. This loss penalizes abrupt changes in the trajectory, promoting smoother transitions. The difference loss is formulated as follows:EQUATIONHere, ∆x i,t represents the position offset of Gaussian i at time t. We also borrow the local-rigidity loss L rigid and rotational loss L rot from [33] (For a more comprehensive understanding, please refer to their detailed paper). For the local-rigidity loss, a weighting scheme is applied that utilizes an unnormalized Gaussian weighting factor. This factor assigns different weights to each point based on its proximity to the center of the Gaussian distribution. The unnormalized Gaussian weighting factor is defined as follows:EQUATIONUsing this weighting scheme, a local-rigidity loss is employed, denoted as L rigid , defined as follows:EQUATIONEQUATION)This loss ensures that for each Gaussian i, neighboring Gaussians j move in a manner consistent with the rigid body transformation of the coordinate system over time. Specifically, this means that the relative positions and orientations of neighboring Gaussians are maintained as the coordinate system evolves, ensuring a coherent and physically plausible motion pattern.Additionally, we incorporate a rotational loss L rot to ensure consistent rotations among neighboring Gaussians across different time steps. This loss function helps maintain the relative orientations of nearby Gaussians as they evolve, preserving the overall rotational coherence of the system. The rotational loss is expressed as:EQUATIONHere, q represents the normalized quaternion rotation of each Gaussian. The k-nearest neighbors, identified in the same manner as in the preceding losses, are utilized to ensure consistent rotational behavior among neighboring Gaussians.Our method consists of two main components: canonical space reconstruction and motion fitting.During training with SDS, we alternately perform multi-view and temporal SDS. For each iteration, we randomly sample either a camera pose or a timestep. This sampling determines the specific spatial (view) dimension or the temporal (time) dimension to train. By alternating between these two types of SDS, we ensure comprehensive coverage of both spatial and temporal aspects. This strategy enhances the overall consistency and robustness of the model. For each SDS step, we randomly sample M = 16 frame poses and perform rendering, either following the sequence of camera movement or based on the order of time. It is important to note that we perform noisy virtual trajectory sampling during multi-view SDS. Specifically, we randomly sample a camera pose that is different from the reference pose used to generate the temporal video. Subsequently, we sample a virtual camera pose within a specified radius of a circle centered on the sampled pose. By interpolating between this virtual camera pose and the reference camera pose, we generate M = 16 frame poses for rendering. This approach ensures a diverse set of viewpoints, enhancing the robustness and accuracy of the multi-view SDS process. When inputting our high-resolution rendered images into the low-resolution model for SDS, we first apply one of two preprocessing steps with equal probability. The image is either randomly shifted by S -1 pixels (where S is the downsampling scale) or randomly cropped to half its size. Following this, we downsample the image and input it into the low-resolution model. This preprocessing ensures that the low-resolution model receives varied inputs, enhancing the robustness of the training process.During the GS growth for motion fitting, we fix the attributes (position, rotation, scale, opacity and color) of the learned canonical Gaussians by stopping the gradient flow to them. Instead, we focus on performing splitting, cloning, and pruning operations based on the gradients of the newly added Gaussians. These new Gaussians are introduced at the beginning of the growth stage, specifically according to the gradients derived from the canonical Gaussians. This method ensures that the canonical structure remains stable while allowing the newly added Gaussians to adapt and optimize the motion fitting process.For the training process, we adopted the differential Gaussian rasterization technique from 3D GS [23] and utilized the same deformation network structure as described in [67] for both perframe deformation and motion deformation. Both deformation networks consist of an MLP F θ : (γ(x), γ(t)) → (∆x, ∆q, ∆s), which applies position embedding γ on each coordinate of the 3D Gaussians and time (or per-frame deformation code) and maps them to their corresponding deviations in position, rotation, and scaling. The weights θ of the MLP are optimized during this mapping process. Each MLP F θ processes the input through eight fully connected layers, each employing ReLU activations and containing 256-dimensional hidden units, resulting in a 256-dimensional feature vector. This feature vector is then passed through three additional fully connected layers (without activation functions) to independently output the offsets for position, rotation, and scaling over time. It is worth noting that, similar to NeRF, the feature vector and the input are concatenated at the fourth layer, enhancing the network's ability to capture complex deformations.The initialization of the Gaussians was based on Structure-from-Motion (SfM) results obtained from COLMAP [46] . We adhered to the training settings specified in [23] for the canonical GS training. The learning rate for the positions of the Gaussians was set to undergo exponential decay, starting from 1.6 × 10 -4 and decreasing to 1.6 × 10 -6 . Additionally, the learning rate for both scale and rotation parameters was maintained at 1 × 10 -3 throughout the training process. The learning rate for the deformation network was set to exponentially decay from 1 × 10 -3 to 1 × 10 -5 . The optimization across these processes was conducted using the Adam optimizer [24] with β parameters set to (0.9, 0.999). All experiments were performed on single NVIDIA A100 GPUs with 80GB of memory.For the regularization terms mentioned above, we assign a weight of 0.01 to each within the overall loss function, ensuring they contribute appropriately to the optimization process. For the reconstruction loss, we use a weight of 1. Additionally, for the SDS loss, we differentiate the weights based on the type of SDS being applied: a weight of 20 for temporal SDS and a weight of 5 for multi-view SDS. These weights are chosen to effectively capture motion dynamics and stabilize the training process.For the GS growth stage during the canonical space reconstruction, we set the opacity threshold to τ α = 5 × 10 -3 and the gradient threshold to τ grad = 2 × 10 -4 . These thresholds help control the addition of new Gaussians based on their opacity and gradient values. In contrast, for the GS growth stage during motion fitting, we use a relatively higher opacity threshold of τ α = 1 × 10 -2 to avoid introducing redundant Gaussians. This higher threshold ensures that only significant Gaussians are added, which aids in maintaining efficiency and relevance during the motion fitting process.The user study was conducted through a single survey comprising seven questions. Each question asked the evaluators to compare two videos: one rendered with our method and the other with a different method using the same text prompt as input. Evaluators indicated their preferred video based on various aspects, as illustrated in Fig. 10 . The evaluators were given the following instructions for each metric.• Motion Realism: Take into consideration of magnitude, smoothness, consistency of the motion. Pay close attention to deformed limbs of human and animals and unnatural deformations. • Foreground Photo-realism: Assess realism of the appearance of the foreground object, do not consider the motion factor. That is the main object that looks more like taken with real video camera, pay attention to unnatural color or styles of the objects. • Background Photo-realism: Assess realism of the appearance of the background. That is the background that looks more like taken with real video camera, pay attention to unnatural color or styles of the objects or bluriness. • 3D Shape Realism: That is the video in which main object has the most natural shape, again pay attention to deformed limbs of human and animals and unnatural deformations. • General Realism: Please provide your subjective appraisal of which video, in your opinion, stands out as superior based on appearance quality, 3D structure quality, and motion quality. • Significance of Motion: The video that contain the most amount of motion. Please exclude from consideration any random limbs deformations. • Video-text Alignment: That is which video reflect all the aspects included in the text description.Figure 10 : Screenshot of the user study webpage.NeurIPS Paper Checklist 


Decision: The paper proposed a text-to-4D generation framework, which first generates a canonical 3DGS from a generated freeze-time video and further optimizes with a video SDS-based loss to incorporate dynamics. The proposed pipeline enables generating more complex 4D scenes over video-SDS baselines (e.g. 4Dfy), overcomes the limitation on limited 3D/4D data, and provides promising results on 4D generation. 

After the rebuttal, all the reviewers are positive about the paper. The AC agrees with the reviewers' suggestions on accepting this paper. As promised during the rebuttal, please add the additional comparisons with baselines, as well as additional results provided in the rebuttal  to support the claims mentioned in the paper.

Reviews: review: paper_summary: paper_summary: The paper presents a method for 3D scene generation using video diffusion models. The main contribution is to present a method that does not require a multi-view generative model trained on synthetic data. The paper identifies the use of multi-view models as a strong limitation limiting existing works to generate high-quality scenes. The method first generates a video from the text prompt, and then also generates a second video starting from a reference frame that corresponds to a time-freezed camera scan of the scene. The second video allows for a static 3D reconstruction of the reference frame, that is then used in reconstructing the motion in the original video. The proposed method is novel and achieves better results than the state of the art.
strength_weakness: strength_weakness: The paper shows high-quality results that outperform existing work, and also present ablations. I was curious whether the identification of the use of synthetic data could also be demonstrated as an ablation, e.g., finetuning the snap video model on synthetic data and adding another SDS loss to demonstrate that it is indeed the key factor that limits high quality. 

The paper in the intro mentions "reliance on multi-view generative models". While the video generative model used in the paper does not take camera poses as conditioning, it is in fact trained on static multi-view data. If so, I would still call such a model capable of generating multi-view images as a multi-view model. How would the overall approach perform if the snap video model was not trained on static multi-view scans? It would be good to provide more details on the training data. What percentage of the training data consists of such multi-view recordings?

The paper claims efficiency as a contribution; however, later in L225-228, it correctly points out that most of this improvement is due to the use of snap video model, and not due to any contributions of this paper. If true, the claims should be adjusted. 

From L161-163, it seems like the sufficient amount of pose variation is chosen manually? Is it not possible to automate this? If this step is manual, how is it accounted for in the reported processing times?
questions: questions: Does Colmap work for all the dynamic videos generated, even if the motion is large? 

The results are generally very impressive but it would be very helpful to the community to also show failure examples so that the remaining challenges are clear.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper proposes a new pipeline called 4Real, aiming for more photorealistic text-to-4D generation than prior work. The method first generates a reference video, then learn a canonical 3D representation from a freeze-time video. Afterwards, per-frame and temporal deformation are learned to model the gap between the canonical representation and the targeted video. This paper compares with two baselines and includes both visualizations on 30 examples and user study. The method is also much faster than prior work.
strength_weakness: strength_weakness: This paper has clearly identified quite a few key limitations of prior work and aims to solve them, which is great. But I don't think all of these claims have been validated. For example:

(1) this work captures the interaction between objects and environments

From all the submitted video results, they do not seem to include much (or any) interaction between the object and the environment (e.g., relative global movement between them). For example, for the "bear driving a car" comparison between ours and 4Dfy, the car in "our result" is not moving on the lawn. The movement only comes from the bear inside the car. 

(2) prior work being object-centric while this work not.

Although it's clear that the compared baselines only generate foreground object while the proposed method also creates the background environment. Since there's not much interaction between the background and foreground (as explained above), a better baseline would be generating an empty background/environment that suits the prompt/foreground and then putting the baseline-generated object properly in the environment.
questions: questions: 1. From the results, it does seem that the baselines generate much less realistic objects than the proposed method as the paper has claimed. However, existing text-to-3D models have proved their power in generating realistic 3D assets [20, 24, etc.], so could it because the 3D generation model used in baselines are a bit outdated or the prompts were not the best ones? Maybe the realism in the baseline results can be largely improved by a bit prompt engineering (e.g., adding keywords like "realistic" in the prompt) or updating their used 3D generation model?

2. Does the deformation model also time-varying appearance or only time-varying geometry?

3. Minor suggestions:

(1) maybe adding the input prompt to Fig. 4 may help the readers better understand the output content and quality.

(2) Some typos:

L24: ", E" -> ". E"

L292: missing a period.
limitations: limitations: The limitations seem to have been sufficiently discussed.
paper_summary: paper_summary: This paper proposes a straightforward method to distill a 4D dynamic scene from a video diffusion model. Given a reference video (only generated videos are shown in the paper, no real video), they first generate a freeze-time video as a multi-view to reconstruct a reference 3DGS. Because they don't have a good multiview image model to capture the general scene multiview prior, they can only use a video model, which might produce small deformations. Some engineering effort is paid here to accept such deformation. After the reference Gaussians is reconstructed, an SDS stage is applied to both the frozen time multiview images and the temporal deformed frames.
strength_weakness: strength_weakness: - Object-centric? One important constraint this paper assumes is that because they want to generate dynamic scenes, they don't use sophisticated object-centric models. However, most video examples this paper shows do not have significant background motion, so one strong baseline/alternative is to just distill a static background and put a dynamic subject (from a dynamic object-centric SDS) in the foreground. Maybe this would achieve much better results than the paper's pipeline.
- How consistent? One question arises when you have a reference temporal video. Because there is a hallucination of the freeze-time view only considering a specific frame, however, the hallucinated freeze-time multiview images may not be consistent with what is observed in the temporal video, how does the system handle such in-consistency?
- Failure case: when i open the static website files in the supply, there is a folder called "good results", which the reviewer guesses may be because of some case selection, it would be critical to also show failure case.
questions: questions: The authors also have to highlight their novelty, since the method is straightforward.
limitations: limitations: Some limitation is discussed, but no failure cases are shown.
paper_summary: paper_summary: The paper introduces a pipeline designed for photorealistic text-to-4D scene generation, discarding the dependency on multi-view generative models and instead fully utilizing video generative models trained on diverse real-world datasets. The method begins by generating a reference video using the video generation model, then learns the canonical 3D representation of the video using a freeze-time video, delicately generated from the reference video. To handle inconsistencies in the freeze-time video, the method jointly learns a per-frame deformation to model these imperfections, then learns the temporal deformation based on the canonical representation to capture dynamic interactions in the reference video.
strength_weakness: strength_weakness: 1. The method seems extremely slow due to multiple steps being applied. The method hasn't provide enough latency comparisons.

2. Besides user studies, I wonder if any qualitative results can be shown, for example, FVD. Or some video sequences from multiview data (in door or outdoor) and compute reconstruction loss, but applying the lifting for reference video (using one view video in the data set) and render the other view to compare with the gt novel view video.

3. The model has a freeze time view deformation field, which indicates fitting static 3dgs scene from freeze-time video is unable to deliver good initial 3dgs field. How to balance the 3d consistency and plausibility from freeze time camera prior? More reasoning and analysis is desirable.

4. The paper uses freeze time SDS some part and multi-view SDS somewhere else. Better to improve the consistency.
questions: questions: 1. The paper mentioned a freeze time view deformation field. Does every frame has it or it is only applied to the first frame.

2. How important is the freeze time view deformation field. 

3.  The motion video diffusion needs to sample with the first frame, how does the video diffusion handle different sampled fps? Using it as a input? This setting potentially limits the extension of this method to longer video.
limitations: limitations: Multiview video can be a better prior, it is limited by training data, but will improve in the future. 4D generation might be benefited from this prior and reduces the requirement of multiple stages.
paper_summary: paper_summary: 4Real presents a novel framework for dynamic 3D scene generation. Given an input video, the framework selects a frame and prompts a text-to-video diffusion model to generate a "freeze-time video" that reconstructs the canonical 3D scene. Then, video-SDS-based optimization is used to optimize a dynamic 3DGS with reference to the input video. Compared to existing approaches that focus more on object-centric generation, the new method is able to generate more complex 4D scenes including multiple objects.
strength_weakness: strength_weakness: 1. The description of context conditioning (L149) is not clear to me. "One dataset consists of static real-world objects with circular camera trajectories", is this dataset similar to [CO3D](https://ai.meta.com/datasets/co3d-dataset/), does it contain background, and how critical is the context embedding? Including this information can be important to reproduce this work.
2. The evaluation did not compare with AYG [28], whose results could also be easily accessed. AYG optimizes a dynamic 3DGS while the compared 4Dfy and Dream-in-4D optimize a dynamic NeRF, so I would say AYG is closer to this work. Although similar conclusions could be drawn for both AYG and 4Dfy/Dream-in-4D as they are all object-centric, the argument in L271 that claims "These methods are known to achieve some of the best visual quality compared to AYG and ..." is neither well-grounded nor convincing. L270 is also wrong to claim that Dream-in-4D is based on D-3DGS.
3. Generating a freeze-time video for the static 3D generation makes a lot of sense, but I wonder if it is possible that the generated freeze-time video could conflict with the reference video. Moreover, the canonical 3D generation stage seems to be solving a standalone image/text-to-scene task, it would be interesting to know if recent 3D scene generation works like RealmDreamer[A] and CAT3D[B] could be a drop-in replacement.

Minor:
- Editing is mentioned in the contributions, but it has not been discussed in any other places in the paper.

[A] Shriram et al., RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusion

[B] Gao et al., CAT3D: Create Anything in 3D with Multi-View Diffusion Models
questions: questions: Please kindly address the weaknesses above.
limitations: limitations: Yes.

Comments: title: Keep my positive recommendation
comment: Thanks for the great suggestions! We will add more rigorous analysis with synthetic data, and include thorough discussion of related concurrent works.
I have read through all reviewers' comments and the rebuttals by authors. My concerns have been addressed. Therefore, I would like to raise my rating to weak accept. Please add the relavant discussions in the rebuttal to the final version of the paper if accepted.
Thanks for the response. I would still argue for acceptance. I would urge to add more rigorous experiments to study the role of synthetic data in the final version if accepted. For efficiency, papers such as Wonder3D (and more recently Cat3D) demonstrate 3D results without score distillation. It would be good to tone down the arguments there and mention both the role of the snap video model and the existence of other non-SDS papers.
Thanks for the rebuttal. It clarifies most of my concerns. The numerical comparison with AYG looks good to me. Figure 2,3,4 in the rebuttal would be more helpful if more frames could be visualized. 

On 3D-video inconsistency, a static camera in reference video does not avoid the inconsistency as the objects can move as well, like turning around. "Merge two videos into a single video when upsampling" sounds like a reasonable trick, and should be discussed more in detail in the paper. It would also be better if more principled solutions could be discussed for future works.

I will keep my score.
After reading the reviews and rebuttals, I still think this paper provide valuable contribution to the community. The authors should include the above references [1-5] and discussions in the revision to better clarify the potential concerns, also, if possible, report the failure ratio that how many bad or good results will be generated from the 100 videos. I keep my positive recommendation.